In TF 1.14, below code will raise exception.
optimizer = tf.keras.optimizers.SGD(learning_rate=params['lr'])
train_op = optimizer.minimize(loss, global_step=tf.train.get_or_create_global_step())
The exception is
TypeError: minimize() got an unexpected keyword argument 'global_step'
It seems that tf.keras.optimizers in unavailable in TF 1.14. However, why the global_step is gone in tf.keras.optimizers.SGD? It is supposed to be there in tf.train.Optimizer of TF 1.14
tf.keras.optimizers.Optimizer and tf.train.Optimizer are two distinct classes. There was never global_step in tf.keras.optimizers.Optimizer
Related
I'm using the following code to load an imagenet pre-trained VGG19 model and fit to my custom dataset.
from keras.applications.vgg19 import VGG19
optim = tf.keras.optimizers.RMSprop(momentum=0.9)
vgg19 = VGG19(include_top=False, weights='imagenet', input_tensor=tf.keras.layers.Input(shape=(224, 224, 3)))
vgg19.trainable = False
# x = keras.layers.GlobalAveragePooling2D()(model_vgg19_pt.output)
x = keras.layers.Flatten()(vgg19.output)
output = keras.layers.Dense(n_classes, activation='softmax')(x)
model_vgg19_pt = keras.models.Model(inputs=[vgg19.input], outputs=[output])
model_vgg19_pt.compile(optimizer=optim,
loss='categorical_crossentropy', metrics=['categorical_accuracy'])
callback = tf.keras.callbacks.LearningRateScheduler(scheduler)
model_vgg19_pt.fit(x_train, y_train, batch_size=20,
epochs=50, callbacks=[callback]
)
on model.fit() line, I get the following error
KeyError: 'The optimizer cannot recognize variable dense_1/kernel:0. This usually means you are trying to call the optimizer to update different parts of the model separately. Please call optimizer.build(variables) with the full list of trainable variables before the training loop or use legacy optimizer `tf.keras.optimizers.legacy.{self.class.name}.'
What does it mean and how can I fix it?
I get the same errors for
keras.applications.inception_v3
too, when using the same implementation method.
Additionally, this was working with jupyter notebook file on tensorflow cpu, but when running on a remote machine with tensorflow-gpu installed, I'm getting these errors.
This works fine with optimizer SGD, but not with RMSprop. why?
Additional
Using this:
model_vgg19_pt.compile(optimizer=tf.keras.optimizers.RMSprop(momentum=0.9),
loss='categorical_crossentropy', metrics=['categorical_accuracy'])
instead as used above works. But can somebody explain why....
Which version of Tensorflow GPU have you installed? TensorFlow 2.10 was the last TensorFlow release that supported GPU on native-Windows. Please check the link to install TensorFlow by following all the Hardware/Software requirements for the GPU support.
The LearningRateScheduler arguments in callback is not defined which you are passing while model compilation.
I was able to train the model after removing the callback from model.fit(). (Attaching the gist here for your reference)
For some reason the start_from_epoch argument in the EarlyStopping callback is not recognised.
import tensorflow as tf
cbk = [tf.keras.callbacks.EarlyStopping(monitor='val_loss',
min_delta=0,
patience=3,
restore_best_weights = True,
start_from_epoch=10
)]
TypeError: __init__() got an unexpected keyword argument 'start_from_epoch'
even though it's clearly listed as an argument in the documentation
I have recent versions of TensorFlow installed, so anyone know why this is the case?
I think you are using older version of Tensorflow Try this
!pip install -q tensorflow==2.11.0
monitor='val_loss',
min_delta=0,
patience=0,
verbose=0,
mode='auto',
baseline=None,
restore_best_weights=False,
start_from_epoch=0
)
apply_gradients might have been removed in future versions of optimizer in tensorflow or keras. DO not know why but I am getting this:
AttributeError: 'Adam' object has no attribute 'apply_gradients'
Any other way to achieve the same thing?
apply_gradients is something that is only possible in tensorflow.keras, because you can make manual training loops with eager execution on.
Pure keras must use symbolic graph and can only apply gradients with fit or train_on_batch.
I had the same problem. In the end, this initializer worked:
optimizer = tf.keras.optimizers.Adam()
But these lead to the error:
optimizer = keras.optimizers.Adam()
optimizer = tf.python.keras.optimizers.Adam()
I have a classifier that I trained using keras that is working really well. It uses keras.applications.MobileNetV2.
This classifier is well trained on around 200 categories, and has a high accuracy.
However, I would like to use the feature extraction layers from this classifier as part of an object detection model.
I have been using the Tensorflow Object Detection API, and looking into the SSDLite+MobileNetV2 model. I can start to run training, but the training is very slow and the bulk of the loss comes from the classification stage.
What I would like to do is assign the weights from my keras .h5 model to the Feature Extraction layer of MobileNetV2 in Tensorflow, but I'm not sure of the best way to do that.
I can load the h5 file easily, and get a list of layer names:
import keras
keras_model = keras.models.load_model("my_classifier.h5")
keras_names = [l.name for l in keras_model.layers]
print(keras_names)
I can also restore the tensorflow checkpoint from the object detection API and export the layers with weights:
tf.reset_default_graph()
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph('models/model.ckpt.meta')
what = new_saver.restore(sess, 'models/model.ckpt')
tf_names = []
for op in sess.graph.get_operations():
if "MobilenetV2" in op.name and "Assign" in op.name:
tf_names.append(op.name)
print(tf_names)
I cannot seem to get a good match-up between layer names from keras and from tensorflow. Even if I could I'm not sure of the next steps.
If anyone could give me some advice about the best way to approach this I would be very grateful.
Update:
I followed Sharky's suggestion below, with a slight modification:
new_saver = tf.train.import_meta_graph(os.path.join(keras_checkpoint_dir, 'keras_model.ckpt.meta'))
new_saver.restore(sess, os.path.join(keras_checkpoint_dir, tf.train.latest_checkpoint(keras_checkpoint_dir)))
However unfortunately I now get this error:
NotFoundError (see above for traceback): Restoring from checkpoint
failed. This is most likely due to a Variable name or other graph key
that is missing from the checkpoint. Please ensure that you have not
altered the graph expected based on the checkpoint. Original error:
Key
FeatureExtractor/MobilenetV2/expanded_conv_6/project/BatchNorm/gamma
not found in checkpoint [[node save/RestoreV2_295 (defined at
:7) = RestoreV2[dtypes=[DT_FLOAT],
_device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0,
save/RestoreV2_295/tensor_names,
save/RestoreV2_295/shape_and_slices)]] [[{{node
save/RestoreV2_196/_393}} = _Recvclient_terminated=false,
recv_device="/job:localhost/replica:0/task:0/device:GPU:0",
send_device="/job:localhost/replica:0/task:0/device:CPU:0",
send_device_incarnation=1, tensor_name="edge_789_save/RestoreV2_196",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/device:GPU:0"]]
Any ideas on how to get rid of this error?
You can use tf.keras.estimator.model_to_estimator
estimator = tf.keras.estimator.model_to_estimator(keras_model=model, model_dir=path)
saver = tf.train.Saver()
with tf.Session() as sess:
saver.restore(sess, os.path.join(path/keras, tf.train.latest_checkpoint(path/keras)))
print(tf.global_variables())
This should do the job. Note that it will create a subdirectory inside originally specified path.
Previously tf.variable_scope threw an error if variables were defined with the same name and the reuse parameter was not set to True. This no longer seems to be true in TF 1.0.0. The following minimum example shows this:
import tensorflow as tf
from tensorflow.contrib import layers
print(tf.__version__)
def net(in_data, reuse=None):
with tf.variable_scope("foo", reuse=reuse):
return layers.conv2d(in_data, 3, 3)
in_data = tf.random_uniform([10, 40, 30, 3])
out_data1 = net(in_data)
out_data2 = net(in_data, reuse=None)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
rval = sess.run([out_data1, out_data2])
TF ver 0.12.0 gives the following error:
ValueError: Variable foo/Conv/weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
TF ver 1.0.0 executes without error. Is this an intended change in behavior? If so, how should this type of error be raised in ver 1.0?
Note that this effect seems to be associated with tf.contrib.layers and tf.contrib.slim. tf.layers doesn't show the same issue. Why???