When trying to use BertClassifier in a model I get an error

[eric]

Describe the bug
I have tried to incorporate BertClassifier into a model and ran into a strange error:

ValueError: Cannot get result() since the metric has not yet been built.

To Reproduce
https://colab.research.google.com/drive/105441YtXhvcNj4ZnlpS0KpAZILv_CwTb?usp=sharing

Expected behavior
I expect it to run without error

Would you like to help us fix it?
I've tried to understand how to work around this but so far have been unable to.

[ghost]

I have faced a similar problem.
Enabling TF_USE_LEGACY_KERAS solves it.
When you enable this, you need to match the version of tensorflow and tf_keras.
I think this is probably because bert_tiny_en_uncased was created with keras2.

!pip install keras-nlp

Below the line above, add the following three lines:

!pip install tf_keras==2.17.0
import os
os.environ["TF_USE_LEGACY_KERAS"]="1"

I hope this solves your problem.

!pip install keras-nlp
!pip install tf_keras==2.17.0
import os
os.environ["TF_USE_LEGACY_KERAS"]="1"
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from keras_nlp.models import BertClassifier

# Adjusted set of metrics for binary classification
METRICS = [
    tf.keras.metrics.BinaryAccuracy(name='accuracy'),
    tf.keras.metrics.BinaryCrossentropy(name='cross_entropy')
]

# Simplified model using BertClassifier
def build_simple_bert_model():
    max_text_length = 512
    learning_rate = 0.001

    # Text input processing
    text_input_ids = Input(shape=(max_text_length,), dtype=tf.int32, name="token_ids")
    text_attention_mask = Input(shape=(max_text_length,), dtype=tf.int32, name="padding_mask")
    text_segment_ids = Input(shape=(max_text_length,), dtype=tf.int32, name="segment_ids")

    # Initialize BERT Classifier
    bert_classifier = BertClassifier.from_preset(
        "bert_tiny_en_uncased",
        num_classes=1,  # Binary classification
        preprocessor=None,
        activation="sigmoid"
    )
    bert_classifier.trainable = True  # Set to True if you want to fine-tune BERT

    # BERT output
    bert_output = bert_classifier({
        'token_ids': text_input_ids,
        'padding_mask': text_attention_mask,
        'segment_ids': text_segment_ids
    })

    model = Model(inputs=[text_input_ids, text_attention_mask, text_segment_ids], outputs=bert_output)
    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
                  loss='binary_crossentropy',
                  metrics=METRICS)

    return model

# Build and compile the simple BERT model
model = build_simple_bert_model()

# Print model summary
model.summary()

# Create dummy data to simulate input
batch_size = 32  # Use a smaller batch size for debugging
sequence_length = 512

dummy_token_ids = np.random.randint(0, 30522, size=(batch_size, sequence_length)).astype(np.int32)
dummy_attention_mask = np.ones((batch_size, sequence_length), dtype=np.int32)
dummy_segment_ids = np.zeros((batch_size, sequence_length), dtype=np.int32)
dummy_labels = np.random.randint(0, 2, size=(batch_size, 1)).astype(np.float32)

# Train the simple BERT model
history = model.fit(
    [dummy_token_ids, dummy_attention_mask, dummy_segment_ids],
    dummy_labels,
    epochs=5,
    validation_split=0.2
)

P.S.
At the moment, installing keras-nlp==3.4.1 installed tensorflow==2.17.0.

[SamanehSaadat]

Hi @eric!

Could you explain a little bit more about what you're trying to do? Are you trying to use Bert without preprocessing and compiling it with specific loss and metric? Does this colab do what you're trying to do?

[eric]

I'm trying to use the classifier with the preprocessor, I'm just trying to use it as part of a bigger model that does other things.

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.