I'm trying to build a Keras model using a DenseFeatures layer as the input; the input comes as a dict of Tensors. TF is insisting that I use model.build() to build the model before optimization, but I can't build it due to DenseFeatures not having an input shape. I get the error
AttributeError: 'DenseFeatures' object has no attribute 'shape'
How can I resolve this? Here's my code:
input_layer = tf.keras.layers.DenseFeatures(params.columns)
predictions = Dense(1, input_dim=len(params.columns), activation='softmax')(input_layer)
model = Sequential([input_layer, predictions])
model.build()
ETA: some further information for insight: I'm not actually fitting the model with this code; rather, I'm creating an EstimatorSpec to use with a Sagemaker model (thus it seems like this will probably require a bit of weird footwork between two different paradigms.)
Related
I am applying quantization to a SSD model. The gist is attached. There is a custom object called "AnchorBoxes" which is added while loading the model. This works fine when I don't do quantization. But when I apply quantization, this custom object is not recognized.
I tried a work around.
def apply_quantization_to_conv2D(layer):
#print(layer)
if isinstance(layer, tf.keras.layers.Conv2D):
return tfmot.quantization.keras.quantize_annotate_layer(layer)
return layer
# Use `tf.keras.models.clone_model` to apply `apply_quantization_to_dense`
# to the layers of the model.
annotated_model = tf.keras.models.clone_model(
model,
clone_function=apply_quantization_to_conv2D,
)
#annotated_model.save('quantize_ready_model_20_01_Conv2D.h5', include_optimizer=True)
annotated_model.summary()
# Now that the Dense layers are annotated,
# `quantize_apply` actually makes the model quantization aware.
#quant_aware_model = tfmot.quantization.keras.quantize_apply(annotated_model)
I commented this line quant_aware_model = tfmot.quantization.keras.quantize_apply(annotated_model) in the above code as it was throwing the error ValueError: Unknown layer: AnchorBoxes
Instead I saved the model after applying quantization to the Conv2D layers as below
def apply_quantization_to_conv2D(layer):
#print(layer)
if isinstance(layer, tf.keras.layers.Conv2D):
return tfmot.quantization.keras.quantize_annotate_layer(layer)
return layer
# Use `tf.keras.models.clone_model` to apply `apply_quantization_to_dense`
# to the layers of the model.
annotated_model = tf.keras.models.clone_model(
model,
clone_function=apply_quantization_to_conv2D,
)
annotated_model.summary()
annotated_model.save('quantize_ready_model_20_01_Conv2D_1.h5', include_optimizer=True)
# Now that the Dense layers are annotated,
# `quantize_apply` actually makes the model quantization aware.
#quant_aware_model = tfmot.quantization.keras.quantize_apply(annotated_model)
#quant_aware_model.compile(optimizer=adam, loss=ssd_loss.compute_loss)
#quant_aware_model.summary()
Then I loaded the model hoping that the loaded quantized model as below will have the custom_objects attached to it.
with tfmot.quantization.keras.quantize_scope():
loaded_model = tf.keras.models.load_model('./quantize_ready_model_20_01_Conv2D_1.h5', custom_objects={'AnchorBoxes': AnchorBoxes})
Finally I applied the quantize_apply to the new loaded_model which has quantized layers.
quant_aware_model = tfmot.quantization.keras.quantize_apply(loaded_model)
which again resulted in the same error
ValueError: Unknown layer: AnchorBoxes
System information
TensorFlow version (installed from source or binary): TF 2.0.0
TensorFlow Model Optimization version (installed from source or binary): 0.5.0
Describe the expected behavior
When I run quantize_apply(model), the model should become quantization aware
Describe the current behavior
Throwing an error on the custom objects
Code to reproduce the issue
gist
The issue was fixed after passing the custom layer like this AnchorBoxes': AnchorBoxes in the below code.
with quantize_scope(
{'DefaultDenseQuantizeConfig': DefaultDenseQuantizeConfig,
'AnchorBoxes': AnchorBoxes}):
# Use `quantize_apply` to actually make the model quantization aware.
quant_aware_model = tfmot.quantization.keras.quantize_apply(annotated_model)
Hi I am having some serious problems saving and loading a tensorflow model which is combination of hugging face transformers + some custom layers to do classfication. I am using the latest Huggingface transformers tensorflow keras version. The idea is to extract features using distilbert and then run the features through CNN to do classification and extraction. I have got everything to work as far as getting the correct classifications.
The problem is in saving the model once trained and then loading the model again.
I am using tensorflow keras and tensorflow version 2.2
Following is the code to design the model, train it, evaluate it and then save and load it
bert_config = DistilBertConfig(dropout=0.2, attention_dropout=0.2, output_hidden_states=False)
bert_config.output_hidden_states = False
transformer_model = TFDistilBertModel.from_pretrained(DISTIL_BERT, config=bert_config)
input_ids_in = tf.keras.layers.Input(shape=(BERT_LENGTH,), name='input_token', dtype='int32')
input_masks_in = tf.keras.layers.Input(shape=(BERT_LENGTH,), name='masked_token', dtype='int32')
embedding_layer = transformer_model(input_ids_in, attention_mask=input_masks_in)[0]
x = tf.keras.layers.Bidirectional(
tf.keras.layers.LSTM(50, return_sequences=True, dropout=0.1,
recurrent_dropout=0, recurrent_activation="sigmoid",
unroll=False, use_bias=True, activation="tanh"))(embedding_layer)
x = tf.keras.layers.GlobalMaxPool1D()(x)
outputs = []
# lots of code here to define the dense layers to generate the outputs
# .....
# .....
model = Model(inputs=[input_ids_in, input_masks_in], outputs=outputs)
for model_layer in model.layers[:3]:
logger.info(f"Setting layer {model_layer.name} to not trainable")
model_layer.trainable = False
rms_optimizer = RMSprop(learning_rate=0.001)
model.compile(loss=SigmoidFocalCrossEntropy(), optimizer=rms_optimizer)
# the code to fit the model (which works)
# then code to evaluate the model (which also works)
# finally saving the model. This too works.
tf.keras.models.save_model(model, save_url, overwrite=True, include_optimizer=True, save_format="tf")
However, when I try to load the saved model using the following
tf.keras.models.load_model(
path, custom_objects={"Addons>SigmoidFocalCrossEntropy": SigmoidFocalCrossEntropy})
I get the following load error
ValueError: The two structures don't have the same nested structure.
First structure: type=TensorSpec str=TensorSpec(shape=(None, 128), dtype=tf.int32, name='inputs')
Second structure: type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='inputs/input_ids')}
More specifically: Substructure "type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='inputs/input_ids')}" is a sequence, while substructure "type=TensorSpec str=TensorSpec(shape=(None, 128), dtype=tf.int32, name='inputs')" is not
Entire first structure:
.
Entire second structure:
{'input_ids': .}
I believe the issue is because TFDistilBertModel layer can be called using a dictionary input from DistilBertTokenizer.encode() and that happens to be the first layer. So the model compiler on load expects that to be the input signature to the call model. However, the inputs defined to the model are two tensors of shape (None, 128)
So how do I tell the load function or the save function to assume the correct signatures?
I solved the issue.
The issue was the object transformer_model in the above code is itself not a layer. So if we want to embed it inside another keras layer we should use the internal keras layer that is wrapped in the model
So changing the line
embedding_layer = transformer_model(input_ids_in, attention_mask=input_masks_in[0]
to
embedding_layer = transformer_model.distilbert([input_ids_in, input_masks_in])[0]
makes everything work. Hope this helps someone else. Took a long time to debug through tf.keras code to figure this one out although in hindsight it is obvious. :)
I suffered the same problem, casually, yesterday. My solution is very similar to yours, I supposed that the problem was due to how tensorflow keras processes custom models so, the idea was to use the layers of the custom model inside my model. This has the advantage of not calling explicitly the layer by its name (in my case, it is useful for easy building more generic models using different pretrained encoders):
sent_encoder = getattr(transformers, self.model_name).from_pretrained(self.shortcut_weights).layers[0]
I don't explored all the models of HuggingFace, but a few that I tested seem to be a custom model with only one custom layer.
Your solution also works like a charm, in fact, both solutions are the same if "distilbert" references to ".layers[0]".
I am running Tensorflow 2.1 with keras API. I am following the following coding style:
model = tf.keras.Sequential()
...
model.fit(..., callbacks=callbacks)
Now, I would like to save some intermediate layer tensor value as image summary (as a sample what is happening at n-th training step). In order to do this, I've implemented my own callback class. I've also learned how keras.callbacks.TensorBoard is implemented, since it can save layer weights as image summaries.
I do the following in my on_epoch_end:
tensor = self.model.get_layer(layer_name).output
with context.eager_mode():
with ops.init_scope():
tensor = tf.keras.backend.get_value(tensor)
tf.summary.image(layer_name, tensor, step=step, max_outputs=1)
Unfortunately, I am still getting issue related to eager/graph modes:
tensor = tf.keras.backend.get_value(tensor)
File "/home/matwey/lab/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3241, in get_value
return x.numpy()
AttributeError: 'Tensor' object has no attribute 'numpy'
Unfortunately, there is a little to no documentation on how to correctly combine keras callbacks and tf.summary.image. How could I overcome this issue?
upd: tf_nightly-2.2.0.dev20200427 has the same behaviour.
I have keras pretrained model(model.h5). And I want to prune that model with tensorflow Magnitude-based weight pruning with Keras. One curious things is that my pretrained model is built with original keras model > I mean that is not from tensorflow.keras. Inside tensorflow Magnitude-based weight pruning with Keras example, they show how to do with tensorflow.keras model. I want to ask is that can I use their tool to prune my original keras pretrained model?
inside their weight pruning toolkit ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
inside their weight pruning toolkit ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
For my original pretrained model, I load the weight from model.h5 and can call model.summary() after I apply prune_low_magnitude() none of the method from model cannot call including model.summary() method. And show the error like AttributeError: 'NoneType' object has no attribute 'summary'
model = get_training_model(weight_decay)
model.load_weights('model/keras/model.h5')
model.summary()
epochs = 1
end_step = np.ceil(1.0 * 100 / 2).astype(np.int32) * epochs
print(end_step)
new_pruning_params = {
'pruning_schedule': tfm.sparsity.keras.PolynomialDecay(initial_sparsity=0.1,
final_sparsity=0.90,
begin_step=40,
end_step=end_step,
frequency=30)
}
new_pruned_model = tfm.sparsity.keras.prune_low_magnitude(model, **new_pruning_params)
print(new_pruned_model.summary())
Inside their weight pruning toolkit
enter link description here ,there is two way. one is pruned the model layer by layer while training and second is pruned the whole model. I tried the second way to prune the whole pretrained model. below is my code.
For my original pretrained model, I load the weight from model.h5 and can call model.summary() after I apply prune_low_magnitude() none of the method from model cannot call including model.summary() method. And show the error like
AttributeError: 'NoneType' object has no attribute 'summary'
I hope this answer still helps, but I recently had the same issue that prune_low_magnitude() returns an object of type 'None'. Also new_pruned_model.compile() would not work.
The model I had been using was a pretrained model that could be imported from tensorflow.python.keras.applications.
For me this worked:
(0) Import the libraries:
from tensorflow_model_optimization.python.core.api.sparsity import keras as sparsity
from tensorflow.python.keras.applications.<network_type> import <network_type>
(1) Define the pretrained model architecture
# define model architecture
loaded_model = <model_type>()
loaded_model.summary()
(2) Compile the model architecture and load the pretrained weights
# compile model
opt = SGD(lr=learn_rate, momentum=momentum)
loaded_model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
loaded_model.load_weights('weight_file.h5')
(3) set pruning parameters and assign pruning schedule
# set pruning parameters
pruning_params = {
'pruning_schedule': sparsity.PolynomialDecay(...)
}
# assign pruning schedule
model_pruned = sparsity.prune_low_magnitude(loaded_model, **pruning_params)
(4) compile model and show summary
# compile model
model_pruned.compile(
loss=tf.keras.losses.categorical_crossentropy,
optimizer='SGD',
metrics=['accuracy'])
model_pruned.summary()
It was important to import the libraries specifically from tensorflow.python.keras and use this keras model from the TensorFlow library.
Also, it was important to use the TensorFlow Beta Release (pip install tensorflow==2.0.0b1), otherwise still an object with type 'None' would be returned by prune_low_magnitude.
I am using PyCharm 2019.1.3 (x64) as IDE. Here is the link that led me to this solution: https://github.com/tensorflow/model-optimization/issues/12#issuecomment-526338458
I am trying to create a tensorflow model using keras.Sequential() as follows:
nn_1 = tf.keras.Sequential() model
nn_1 = tf.keras.Sequential() model
model = tf.keras.Sequential()
model.add(tf.keras.layers.concatenate([nn_1, nn_2]))
When I try this, I get the following error:
ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs
I don't understand why this is happening, since I am passing a list of 2 inputs to concatenate() . Am I missing something basic?
P.S. The same thing happened with dot() (which is what i was initially trying)