I'm training some models using tf.keras and want to save the trained model. There are two recommended ways of doing this, tf SavedModel and keras .h5 file. However things get really confusing with SavedModel.
Here are some short scripts to reproduce the issue:
import numpy as np
import tensorflow as tf
def my_network():
backbone_model = tf.keras.applications.InceptionResNetV2(
include_top=False, input_shape=(224, 224, 3), weights="imagenet", pooling="avg"
)
inputs = tf.keras.layers.Input(shape=(224, 224, 3), name="images")
backbone_features = backbone_model(inputs)
pre_embeddings = tf.keras.layers.Dense(
512,
activation=None,
name="pre_embeddings",
kernel_regularizer=tf.keras.regularizers.l2(),
)(backbone_features)
embeddings = tf.keras.layers.Lambda(
lambda x: tf.math.l2_normalize(x, 1, 1e-10), name="embeddings"
)(pre_embeddings)
probs = tf.keras.layers.Dense(
1000,
activation="softmax",
name="predictions",
kernel_regularizer=tf.keras.regularizers.l2(),
bias_regularizer=tf.keras.regularizers.l2(),
)(pre_embeddings)
return tf.keras.Model(inputs, [embeddings, probs], name="my_network")
img_arr = np.random.rand(1, 224, 224, 3)
resnet_model = my_network()
emb_1, _ = resnet_model.predict(img_arr)
resnet_model.save("./resnet_model.h5")
new_model = tf.keras.models.load_model('./resnet_model.h5')
emb_2, _ = new_model.predict(img_arr)
np.testing.assert_array_almost_equal(emb_1, emb_2)
The above script will work without errors. However it fails when I tried to save in the SavedModel format (by removing .h5 from model path). The model saved successfully but throws an error when loading and the error message is:
NotImplementedError: When subclassing the `Model` class, you should implement a `call` method.
I am confused because I did not use any subclassed model. As shown in the script, my network is built with functional API only.
Related
When extracting a model layer output as in the Tensorflow sequential model document example below, does the input x in the code go through the my_first_layer as well before going into my_intermediate_layer layer? Or does it directly go into the my_intermediate_layer layer without going through the my_first_layer layer?
If it directly goes into the my_intermediate_layer, the input to the my_intermediate_layer does not have the transformation done by my_first_layer Conv2D. However, it seems not right to me because the input should go through all the preceding layers.
Please help understand what layers does x go through?
Feature extraction with a Sequential model
initial_model = keras.Sequential(
[
keras.Input(shape=(250, 250, 3)),
layers.Conv2D(32, 5, strides=2, activation="relu", name="my_first_layer"),
layers.Conv2D(32, 3, activation="relu", name="my_intermediate_layer"),
layers.Conv2D(32, 3, activation="relu"),
]
)
# The model goes through the training.
...
# Feature extractor
feature_extractor = keras.Model(
inputs=initial_model.inputs,
outputs=initial_model.get_layer(name="my_intermediate_layer").output,
)
# Call feature extractor on test input.
x = tf.ones((1, 250, 250, 3))
features = feature_extractor(x)
Keras offers higher level of API, which runs on top of the TensorFlow machine learning platform. Keras offers two types of class to define the neural network model, namely 'Sequential Class' and 'Model Class.'
Sequential Class:
It groups a linear stack of layers to form a model, such that each layer has one input and one output tensor. One can add required layers to the defined model (schema-1) as shown below to execute sequentially as name suggests Keras Sequential Class,
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(8, input_shape=(16,)))
model.add(tf.keras.layers.Dense(4))
The schema for defining a sequential model Keras-Sequential Class Definition has shown below (schema-2),
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential(
[
layers.Dense(2, activation="relu", name="layer1"),
layers.Dense(3, activation="relu", name="layer2"),
layers.Dense(4, name="layer3"),
]
)
# Call model on a test input
x = tf.ones((3, 3))
y = model(x)
Model Class
It allows the user to build a custom model along with many layers as shown below,
import tensorflow as tf
inputs = tf.keras.Input(shape=(3,))
x = tf.keras.layers.Dense(4, activation=tf.nn.relu)(inputs)
outputs = tf.keras.layers.Dense(5, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
It allows one to create a new functional API model with additional layers Keras - Model Class as follows,
inputs = keras.Input(shape=(None, None, 3))
processed = keras.layers.RandomCrop(width=32, height=32)(inputs)
conv = keras.layers.Conv2D(filters=2, kernel_size=3)(processed)
pooling = keras.layers.GlobalAveragePooling2D()(conv)
feature = keras.layers.Dense(10)(pooling)
Note: The input tensors supports only dicts, lists or tuples but not lists of list, or dicts of dict.
I hope that this helps.
I am following the official Keras transfer learning and fine-tuning tutorial. It consists of loading the Xception model with include_top=False, and adding a new classifier part on top.
I am then saving the model with model.save() and loading with load_model().
So this is what I see when I do model.summary()
My problem is that I would like to iterate through the layers, while now Xception layers are somehow folded (on the picture: xception(Functional)). Is there a way to somehow unfold it, to see all the layers (including those that are creating Xception)?
For model. summary(), you can unfold that as follows:
from tensorflow import keras
base_model = keras.applications.Xception(
weights='imagenet', # Load weights pre-trained on ImageNet.
input_shape=(150, 150, 3),
include_top=False) # Do not include the ImageNet classifier at the top.
inputs = keras.Input(shape=(150, 150, 3))
x = base_model(inputs, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
outputs = keras.layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.summary() # full model
model.layers[1].summary() # only xception model
Note, you cal also see the layer using the plot_model utility.
keras.utils.plot_model(model, expand_nested=True)
I am trying to convert a trained model in tensorflow to Open VINO Intermediate Representation.
I have a model of the form given below
class Conv3DModel(tf.keras.Model):
def __init__(self):
super(Conv3DModel, self).__init__()
# Convolutions
self.conv1 = tf.compat.v2.keras.layers.Conv3D(32, (3, 3, 3), activation='relu', name="conv1", data_format='channels_last')
self.pool1 = tf.keras.layers.MaxPool3D(pool_size=(2, 2, 2), data_format='channels_last')
self.conv2 = tf.compat.v2.keras.layers.Conv3D(64, (3, 3, 3), activation='relu', name="conv1", data_format='channels_last')
self.pool2 = tf.keras.layers.MaxPool3D(pool_size=(2, 2,2), data_format='channels_last')
# LSTM & Flatten
self.convLSTM =tf.keras.layers.ConvLSTM2D(40, (3, 3))
self.flatten = tf.keras.layers.Flatten(name="flatten")
# Dense layers
self.d1 = tf.keras.layers.Dense(128, activation='relu', name="d1")
self.out = tf.keras.layers.Dense(6, activation='softmax', name="output")
def call(self, x):
x = self.conv1(x)
x = self.pool1(x)
x = self.conv2(x)
x = self.pool2(x)
x = self.convLSTM(x)
x = self.flatten(x)
x = self.d1(x)
return self.out(x)
I tried to convert the model into IR. The model is here .
I have trained this model in tensorflow 1.15. Tensorflow 2.0 is currently not supported.
Now I tried to run the command
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --saved_model_dir jester_trained_models/3dcnn-basic/ --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/
Now i got the following error
Model Optimizer arguments:
Common parameters:
Path to the Input Model: None
Path for generated IR: /home/deepanshu/open_vino/udacity_project_custom_model/
IR output name: saved_model
Log level: ERROR
Batch: Not specified, inherited from the model
Input layers: Not specified, inherited from the model
Output layers: Not specified, inherited from the model
Input shapes: Not specified, inherited from the model
Mean values: Not specified
Scale values: Not specified
Scale factor: Not specified
Precision of IR: FP32
Enable fusing: True
Enable grouped convolutions fusing: True
Move mean values to preprocess section: False
Reverse input channels: False
TensorFlow specific parameters:
Input model in text protobuf format: False
Path to model dump for TensorBoard: None
List of shared libraries with TensorFlow custom layers implementation: None
Update the configuration file with input/output node names: None
Use configuration file used to generate the model with Object Detection API: None
Operations to offload: None
Patterns to offload: None
Use the config file: None
Model Optimizer version: 2020.1.0-61-gd349c3ba4a
[ ERROR ] Unexpected exception happened during extracting attributes for node conv3d_model/conv_lst_m2d/bias/Read/ReadVariableOp. Original exception message: 'ascii' codec can't decode byte 0xc9 in position 1: ordinal not in range(128)
As far as I can see it is the tf.keras.layers.ConvLSTM2D(40, (3, 3)) causing problems . I am kind of stuck here . Can anyone tell me where can I proceed further ?
Thanks
Edit to the question
Now I rejected the above tensorflow implementation and used keras . My h5 model developed was converted into .pb format using this post.
Now I ran the model optimizer on this .pb file. Using the command
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --input_model /home/deepanshu/ml_playground/jester_freezed/tf_model.pb --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/ --input_shape=[1,30,64,64,1] --data_type FP32
Now i am facing another issue . The issue here is point no. 97 on this post.
So my model contains a cycle and model optimizer does not know a way to convert it. Has anybody faced this issue before ?
Please help.
Here is the model .
Here is the defination of the model in keras
from keras.models import Sequential
from keras.layers import Conv3D , MaxPool3D,Flatten ,Dense
from keras.layers.convolutional_recurrent import ConvLSTM2D
import keras
model = Sequential()
model.add(Conv3D(32, (3, 3, 3),
name="conv1" , input_shape=(30, 64, 64,1) , data_format='channels_last',
activation='relu') )
model.add(MaxPool3D(pool_size=(2, 2, 2), data_format='channels_last'))
model.add(Conv3D(64, (3, 3, 3), activation='relu', name="conv2", data_format='channels_last'))
model.add(MaxPool3D(pool_size=(2, 2,2), data_format='channels_last'))
model.add(ConvLSTM2D(40, (3, 3)))
model.add(Flatten(name="flatten"))
model.add(Dense(128, activation='relu', name="d1"))
model.add(Dense(6, activation='softmax', name="output"))
Actually the script to convert from h5 to .pb suggested by intel was not good enough. Always use the code from here to convert your keras model to .pb.
Once you obtain your .pb file now convert your model to IR using
python3 /opt/intel/openvino/deployment_tools/model_optimizer/mo_tf.py --input_model ml_playground/try_directory/tf_model.pb --output_dir /home/deepanshu/open_vino/udacity_project_custom_model/ --input_shape=[1,30,64,64,1] --data_type FP32
After the execution of this script we can obtain the intermediate representation of the keras model.
It seems that keras trainable attribute is ignored by tensorflow, which makes it very inconvenient to use keras as a syntactical shortcut in tensorflow.
For example:
import keras
import tensorflow as tf
import numpy as np
import keras.backend as K
Conv2 = keras.layers.Conv2D(filters=16, kernel_size=3, padding='same')
Conv2.trainable = False #This layers has been set to not trainable.
A=keras.layers.Input(batch_shape=(1,16,16,3))
B = Conv2(A)
x = np.random.randn(1, 16, 16,3)
y = np.random.randn(1,16, 16, 16)
True_y = tf.placeholder(shape=(1,16,16,16), dtype=tf.float32)
loss = tf.reduce_sum((B - True_y) ** 2)
opt_op = tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
print(tf.trainable_variables())
# [<tf.Variable 'conv2d_1/kernel:0' shape=(3, 3, 3, 16) dtype=float32_ref>, <tf.Variable 'conv2d_1/bias:0' shape=(16,) dtype=float32_ref>]
sess = K.get_session()
for _ in range(10):
out = sess.run([opt_op, loss], feed_dict={A:x, True_y:y})
print(out[1])
OutPut:
5173.94
4968.7754
4785.889
4624.289
4482.1
4357.5757
4249.1504
4155.329
4074.634
4005.6482
It simply means the loss is decreasing and the weights are trainable.
I read the blog ''Keras as a simplified interface to TensorFlow'', but it mentioned nothing about the trainable problem.
Any suggestion is appreciated.
Your conclusion is basically correct. Keras is a wrapper around TensorFlow, but not all Keras functionality transfers directly into TensorFlow, so you need to be careful when you mix Keras and raw TF.
Specifically, in this case, if you want to call the minimize function yourself, you need to specify which variables you want to train on using the var_list argument of minimize.
(1) I'm trying to fine-tune a VGG-16 network using TFSlim by loading pretrained weights into all layers except thefc8 layer. I achieved this by using the TF-SLIm function as follows:
import tensorflow as tf
import tensorflow.contrib.slim as slim
import tensorflow.contrib.slim.nets as nets
vgg = nets.vgg
# Specify where the Model, trained on ImageNet, was saved.
model_path = 'path/to/vgg_16.ckpt'
# Specify where the new model will live:
log_dir = 'path/to/log/'
images = tf.placeholder(tf.float32, [None, 224, 224, 3])
predictions = vgg.vgg_16(images)
variables_to_restore = slim.get_variables_to_restore(exclude=['fc8'])
restorer = tf.train.Saver(variables_to_restore)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
restorer.restore(sess,model_path)
print "model restored"
This works fine as long as I do not change the num_classes for the VGG16 model. What I would like to do is to change the num_classes from 1000 to 200. I was under the impression that if I did this modification by defining a new vgg16-modified class that replaces the fc8 to produce 200 outputs, (along with a variables_to_restore = slim.get_variables_to_restore(exclude=['fc8']) that everything will be fine and dandy. However, tensorflow complains of a dimensions mismatch:
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1,1,4096,200] rhs shape= [1,1,4096,1000]
So, how does one really go about doing this ? The documentation for TFSlim is really patchy and there are several versions scattered on Github - so not getting much help there.
You can try using slim's way of restoring — slim.assign_from_checkpoint.
There is related documentation in the slim sources:
https://github.com/tensorflow/tensorflow/blob/129665119ea60640f7ed921f36db9b5c23455224/tensorflow/contrib/slim/python/slim/learning.py
Corresponding part:
*************************************************
* Fine-Tuning Part of a model from a checkpoint *
*************************************************
Rather than initializing all of the weights of a given model, we sometimes
only want to restore some of the weights from a checkpoint. To do this, one
need only filter those variables to initialize as follows:
...
# Create the train_op
train_op = slim.learning.create_train_op(total_loss, optimizer)
checkpoint_path = '/path/to/old_model_checkpoint'
# Specify the variables to restore via a list of inclusion or exclusion
# patterns:
variables_to_restore = slim.get_variables_to_restore(
include=["conv"], exclude=["fc8", "fc9])
# or
variables_to_restore = slim.get_variables_to_restore(exclude=["conv"])
init_assign_op, init_feed_dict = slim.assign_from_checkpoint(
checkpoint_path, variables_to_restore)
# Create an initial assignment function.
def InitAssignFn(sess):
sess.run(init_assign_op, init_feed_dict)
# Run training.
slim.learning.train(train_op, my_log_dir, init_fn=InitAssignFn)
Update
I tried the following:
import tensorflow as tf
import tensorflow.contrib.slim as slim
import tensorflow.contrib.slim.nets as nets
images = tf.placeholder(tf.float32, [None, 224, 224, 3])
predictions = nets.vgg.vgg_16(images)
print [v.name for v in slim.get_variables_to_restore(exclude=['fc8']) ]
And got this output (shortened):
[u'vgg_16/conv1/conv1_1/weights:0',
u'vgg_16/conv1/conv1_1/biases:0',
…
u'vgg_16/fc6/weights:0',
u'vgg_16/fc6/biases:0',
u'vgg_16/fc7/weights:0',
u'vgg_16/fc7/biases:0',
u'vgg_16/fc8/weights:0',
u'vgg_16/fc8/biases:0']
So it looks like you should prefix scope with vgg_16:
print [v.name for v in slim.get_variables_to_restore(exclude=['vgg_16/fc8']) ]
gives (shortened):
[u'vgg_16/conv1/conv1_1/weights:0',
u'vgg_16/conv1/conv1_1/biases:0',
…
u'vgg_16/fc6/weights:0',
u'vgg_16/fc6/biases:0',
u'vgg_16/fc7/weights:0',
u'vgg_16/fc7/biases:0']
Update 2
Complete example that executes without errors (at my system).
import tensorflow as tf
import tensorflow.contrib.slim as slim
import tensorflow.contrib.slim.nets as nets
s = tf.Session(config=tf.ConfigProto(gpu_options={'allow_growth':True}))
images = tf.placeholder(tf.float32, [None, 224, 224, 3])
predictions = nets.vgg.vgg_16(images, 200)
variables_to_restore = slim.get_variables_to_restore(exclude=['vgg_16/fc8'])
init_assign_op, init_feed_dict = slim.assign_from_checkpoint('./vgg16.ckpt', variables_to_restore)
s.run(init_assign_op, init_feed_dict)
In the example above vgg16.ckpt is a checkpoint saved by tf.train.Saver for 1000 classes VGG16 model.
Using this checkpoint with all variables of 200 classes model (including fc8) gives the following error:
init_assign_op, init_feed_dict = slim.assign_from_checkpoint('./vgg16.ckpt', slim.get_variables_to_restore())
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
1 init_assign_op, init_feed_dict = slim.assign_from_checkpoint(
----> 2 './vgg16.ckpt', slim.get_variables_to_restore())
/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.pyc in assign_from_checkpoint(model_path, var_list)
527 assign_ops.append(var.assign(placeholder_value))
528
--> 529 feed_dict[placeholder_value] = var_value.reshape(var.get_shape())
530
531 assign_op = control_flow_ops.group(*assign_ops)
ValueError: total size of new array must be unchanged