How to create a Keras model using a frozen_interference_graph.pb? - tensorflow

I want to use a pre-trained model and add a segmentation head at the end of that, but the problem is that I just have the 'frozen_inference_graph.pb'. These are the files that I have from the model:
I have tried several ways:
1. loading the pre-trained model into a Keras model:
It seems to be impossible with the files that I have. It just gives me an AutoTrackable object instead of a model.
2. Accessing the Tensor Objects of the frozen model and make the model with tensors:
I found out how to access the tensors but couldn't make a Keras model with Tensor objects.
with self.graph.as_default():
graph = tf.import_graph_def(graph_def, name='')
graph = tf.compat.v1.import_graph_def(graph_def)
tf.compat.v1.Graph.as_default(graph)
self.sess = tf.Session(graph=self.graph)
self.tensors = [tensor for op in tf.compat.v1.get_default_graph().get_operations() for tensor in op.values()]
Here I can get the tensors but I can't use the tensors in the model:
model = tf.keras.models.Model(inputs=self.tensors[0], outputs=self.tensors[-1])
Is there any way to convert this frozen graph to a Keras model?
Or If there is another approach which I can train the model, I would be glad to know.
P.S. The pre-trained model is 'ssd_mobilenet_v3_small_coco_2020_01_14' which can be found Here.

You can use two methods:
The file 'frozen_inference_graph.pb' contains all necessary information about the weights and the model architecture. Use the following snippet to read the model and add a new layer:a
customModel = tf.keras.models.load_model('savedModel')
# savedModel is the folder with .pb data
pretrainedOutput = customModel.layers[-1].output
newOutput = tf.keras.layers.Dense(2)(pretrainedOutput) # change layer as needed
new_model = tf.keras.Model(inputs=customModel.inputs, outputs=[newOutput])
# create a new model with input of old model and new output tensors
where 'savedModel' is the name of the folder with 'frozen_inference_graph.pb' and other meta data. See details about using .pb files and finetuning the custom models in the TFguide.
Try using .meta file with model architecture and .ckpt to restore the weights in TF 1.x:
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph('my_test_model-1000.meta')
new_saver.restore(sess, tf.train.latest_checkpoint('./'))
Refer to the tutorial on how to load and customize restored models in TF 1.x.

Related

How to manually load pretrained model if I can't download it using TensorFlow

I am trying to download the VGG19 model via TensorFlow
base_model = VGG19(input_shape = [256,256,3],
include_top = False,
weights = 'imagenet')
However the download always gets stuck before it finishes downloading. I've tried with different models too like InceptionV3 and the same happens there.
Fortunately, the prompt makes the link available where the model can be downloaded manually
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg19/vgg19_weights_tf_dim_ordering_tf_kernels_notop.h5
19546112/80134624 [======>.......................] - ETA: 11s
After downloading the model from the given link I try to import the model using
base_model = load_model('vgg19_weights_tf_dim_ordering_tf_kernels_notop.h5')
but I get this error
ValueError: No model found in config file.
How do I load in the downloaded .h5 model manually?
You're using load_model on weights, instead of a model. You need to have a defined model first, then load the weights.
weights = "path/to/weights"
model = VGG19 # the defined model
model.load_weights(weights) # the weights
Got the same problem when learning on tensorflow tutorial, too.
Transfer learning and fine-tuning: Create the base model from the pre-trained convnets
# Create the base model from the pre-trained model MobileNet V2
IMG_SIZE = (160, 160)
IMG_SHAPE = IMG_SIZE + (3,)
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE, include_top=False, weights=None)
# load model weights manually
weights = 'mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_160_no_top.h5'
base_model.load_weights(weights)
I tried download the model.h5, and load manually. It works.
`

Keras models in tensorflow

I'm building image processing network in tensorflow and I want to make use of texture loss. Texture loss seems simple to implement if you have pretrained model loaded.
I'm using TF to build the computational graph for my model and I want to incorporate Keras.application.VGG19 model to get output from layer 'block4_conv4'.
The problem is: I have two TF tensors target and result from my main model, how to feed them into keras VGG19 in the same session to compute their diff and use it in main loss for my model?
It seems following code does the trick
with tf.variable_scope("") as scope:
phi_func = VGG19(include_top=False, weights=None, input_shape=(128, 128, 3))
text_1 = phi_func(predicted)
scope.reuse_variables()
text_2 = phi_func(x)
text_loss = tf.reduce_mean((text_1 - text_2)**2)
right after session created I call phi_func.load_weights(path) to initiate weights

Embedding feature vectors in Tensorflow

In text processing there is embedding to show up (if I understood it correctly) the database words as vector (after dimension reduction).
now, I am wondering, is there any method like this to show extracted features via CNN?
for example: consider we have a CNN and train and test sets. we want to train the CNN with train set and meanwhile see the extracted features (from dense layer) corresponding class labels via CNN in the embedding section of tensorboard.
the purpose of this work is seeing the features of input data in every batch and understand how close or far are they from together. and finally, in the trained model, we can find out accuracy of our classifier (like softmax or etc.).
thank you in advance for your help.
I have taken help of Tensorflow documentation.
For in depth information on how to run TensorBoard and make sure you are logging all the necessary information, see TensorBoard: Visualizing Learning.
To visualize your embeddings, there are 3 things you need to do:
1) Setup a 2D tensor that holds your embedding(s).
embedding_var = tf.get_variable(....)
2) Periodically save your model variables in a checkpoint in LOG_DIR.
saver = tf.train.Saver()
saver.save(session, os.path.join(LOG_DIR, "model.ckpt"), step)
3) (Optional) Associate metadata with your embedding.
If you have any metadata (labels, images) associated with your embedding, you can tell TensorBoard about it either by directly storing a projector_config.pbtxt in the LOG_DIR, or use our python API.
For instance, the following projector_config.ptxt associates the word_embedding tensor with metadata stored in $LOG_DIR/metadata.tsv:
embeddings {
tensor_name: 'word_embedding'
metadata_path: '$LOG_DIR/metadata.tsv'
}
The same config can be produced programmatically using the following code snippet:
from tensorflow.contrib.tensorboard.plugins import projector
# Create randomly initialized embedding weights which will be trained.
vocabulary_size = 10000
embedding_size = 200
embedding_var = tf.get_variable('word_embedding', [vocabulary_size,
embedding_size])
# Format: tensorflow/tensorboard/plugins/projector/projector_config.proto
config = projector.ProjectorConfig()
# You can add multiple embeddings. Here we add only one.
embedding = config.embeddings.add()
embedding.tensor_name = embedding_var.name
# Link this tensor to its metadata file (e.g. labels).
embedding.metadata_path = os.path.join(LOG_DIR, 'metadata.tsv')
#Use the same LOG_DIR where you stored your checkpoint.
summary_writer = tf.summary.FileWriter(LOG_DIR)
# The next line writes a projector_config.pbtxt in the LOG_DIR. TensorBoard will
# read this file during startup.
projector.visualize_embeddings(summary_writer, config)

How can I convert a trained Tensorflow model to Keras?

I have a trained Tensorflow model and weights vector which have been exported to protobuf and weights files respectively.
How can I convert these to JSON or YAML and HDF5 files which can be used by Keras?
I have the code for the Tensorflow model, so it would also be acceptable to convert the tf.Session to a keras model and save that in code.
I think the callback in keras is also a solution.
The ckpt file can be saved by TF with:
saver = tf.train.Saver()
saver.save(sess, checkpoint_name)
and to load checkpoint in Keras, you need a callback class as follow:
class RestoreCkptCallback(keras.callbacks.Callback):
def __init__(self, pretrained_file):
self.pretrained_file = pretrained_file
self.sess = keras.backend.get_session()
self.saver = tf.train.Saver()
def on_train_begin(self, logs=None):
if self.pretrian_model_path:
self.saver.restore(self.sess, self.pretrian_model_path)
print('load weights: OK.')
Then in your keras script:
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
restore_ckpt_callback = RestoreCkptCallback(pretrian_model_path='./XXXX.ckpt')
model.fit(x_train, y_train, batch_size=128, epochs=20, callbacks=[restore_ckpt_callback])
That will be fine.
I think it is easy to implement and hope it helps.
Francois Chollet, the creator of keras, stated in 04/2017 "you cannot turn an arbitrary TensorFlow checkpoint into a Keras model. What you can do, however, is build an equivalent Keras model then load into this Keras model the weights"
, see https://github.com/keras-team/keras/issues/5273 . To my knowledge this hasn't changed.
A small example:
First, you can extract the weights of a tensorflow checkpoint like this
PATH_REL_META = r'checkpoint1.meta'
# start tensorflow session
with tf.Session() as sess:
# import graph
saver = tf.train.import_meta_graph(PATH_REL_META)
# load weights for graph
saver.restore(sess, PATH_REL_META[:-5])
# get all global variables (including model variables)
vars_global = tf.global_variables()
# get their name and value and put them into dictionary
sess.as_default()
model_vars = {}
for var in vars_global:
try:
model_vars[var.name] = var.eval()
except:
print("For var={}, an exception occurred".format(var.name))
It might also be of use to export the tensorflow model for use in tensorboard, see https://stackoverflow.com/a/43569991/2135504
Second, you build you keras model as usually and finalize it by "model.compile". Pay attention that you need to give you define each layer by name and add it to the model after that, e.g.
layer_1 = keras.layers.Conv2D(6, (7,7), activation='relu', input_shape=(48,48,1))
net.add(layer_1)
...
net.compile(...)
Third, you can set the weights with the tensorflow values, e.g.
layer_1.set_weights([model_vars['conv7x7x1_1/kernel:0'], model_vars['conv7x7x1_1/bias:0']])
Currently, there is no direct in-built support in Tensorflow or Keras to convert the frozen model or the checkpoint file to hdf5 format.
But since you have mentioned that you have the code of Tensorflow model, you will have to rewrite that model's code in Keras. Then, you will have to read the values of your variables from the checkpoint file and assign it to Keras model using layer.load_weights(weights) method.
More than this methodology, I would suggest to you to do the training directly in Keras as it claimed that Keras' optimizers are 5-10% times faster than Tensorflow's optimizers. Other way is to write your code in Tensorflow with tf.contrib.keras module and save the file directly in hdf5 format.
Unsure if this is what you are looking for, but I happened to just do the same with the newly released keras support in TF 1.2. You can find more on the API here: https://www.tensorflow.org/api_docs/python/tf/contrib/keras
To save you a little time, I also found that I had to include keras modules as shown below with the additional python.keras appended to what is shown in the API docs.
from tensorflow.contrib.keras.python.keras.models import Sequential
Hope that helps get you where you want to go. Essentially once integrated in, you then just handle your model/weight export as usual.

tensorflow image retraining with serving

I am trying to serve retrained inception graph using tensorflow serving. For retraining, I am using this example. However I need to make changes to this graph to get it working with serving export code.
Since in tensorflow serving, you will receive serialized images as input, graph input should start with this:
serialized_tf_example = tf.placeholder(tf.string, name='tf_example')
feature_configs = {
'image/encoded': tf.FixedLenFeature(shape=[], dtype=tf.string),
}
tf_example = tf.parse_example(serialized_tf_example, feature_configs)
jpegs = tf_example['image/encoded']
images = tf.map_fn(preprocess_image, jpegs, dtype=tf.float32)
This images tensor should be input to retrained inception graph. However I don't know if its possible to prepend one graph to another in tensorflow like you can append easily using placeholder_with_input (which has been done in retraining code).
graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (
create_inception_graph())
Ideally, in image retraining code, I receive a placeholder tensor jpeg_input_data. I need to append tensor images to this placeholder tensor jpeg_data_tensor and export it as single graph using exporter so that it can be served using tensorflow serving. However I don't any tensorflow instruction that does it. Are there any other alternatives apart from this method?
One way of going about it is:
model_path = 'trained/export.pb'
with tf.Graph().as_default():
with tf.Session() as sess:
with gfile.FastGFile(model_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
# Your prepending ops here
images_placeholder = tf.placeholder(tf.string, name='tf_example')
...
images = tf.map_fn(preprocess_image, jpegs, dtype=tf.float32)
tf.import_graph_def(graph_def, name='inception', input_map={'ResizeBilinear:0': images})
Notice especially the input_map argument. ResizeBilinear:0 is likely not the correct name of the operation you need - you can list the ops by:
[n.name for n in tf.get_default_graph().as_graph_def().node]
I realize this is not a full answer and perhaps not the most efficient but hopefully it can get you started. Just a heads-up, there is also this blogpost.
So since you have already retrained the model, I'm assuming that the model is a Protobuf, but you can just load that up into a Python object and serve off of that object using a custom function that processes either a batch or an atomic operation.
And to your graph question, as far as I know, when you load a tf.Graph() object you are only working with that object and can't work with other graphs... that being said if you had another graph that is an extension of the existing Inception-V3 Graph then you can easily add that to the computation graph for your custom graph quite easily.