Several ways of retraining MobileNet for use with Tensorflow.js have failed for me. Is there any way to use a retrained model with Tensorflow.js?
Both using the modern, hub-based tutorial, as well as using retrain.py seem to fail.
Convert output of retrain.py to tensorflow.js
Error converting keras model to tfjs: duplicate weight name Variable
as well as some other open questions
Retrain an Image Classifier in tensorflow js
Loading of mobilenet v2 works, but pretrained mobilenet v2 fails
Can't convert TensorFlow saved model to tfjs_layers_model webmodel
The top two other questions show the code that failed in both instances, both are unsolved.
The aim is to load the mobilenet, retrain using custom data, and use it in Tensorflow.js. Following both tutorials seem to fail. Could this be done inside node.js? Is there another way? Where did I make mistakes (or is the software unable to use retrained models)? How can this work?
EDITs: latest github issue and one more question
I encountered the same problem and it seems that we use the wrong method.
There are loadGraphModel for TF converted models and loadLayersModel for Keras ones
my comment about the issue
The retrain.py python script does not generate a saved model, it actually generates a frozen graph model. That is why you cannot convert it using the tfjs 1.x converter. You need to use tfjs 0.8.5 pip to convert.
Also, the output node name is different from the mobilenet model graph, it is 'final_result' for retrained graph.
To convert it you need to use the tensorflowjs 0.8.5 pip:
use virtualenv to create an empty env.
pip install tensorflowjs==0.8.5
run the converter
tensorflowjs_converter \
--input_format=tf_frozen_model \
--output_node_names='final_result' \
--output_json=true /tmp/output_graph.pb \ /tmp/web_model
This should give you something like the following:
ls /tmp/web_model/
group1-shard10of21 group1-shard14of21 group1-shard18of21 group1-shard21of21 group1-shard5of21 group1-shard9of21
group1-shard11of21 group1-shard15of21 group1-shard19of21 group1-shard2of21 group1-shard6of21 model.json
group1-shard12of21 group1-shard16of21 group1-shard1of21 group1-shard3of21 group1-shard7of21
group1-shard13of21 group1-shard17of21 group1-shard20of21 group1-shard4of21 group1-shard8of21
To use the latest TFjs:
python retrain.py --tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/2 \
--image_dir /tmp/flower_photos --saved_model_dir /tmp/saved_retrained_model
tensorflowjs_converter --input_format=tf_saved_model \
--output_format=tfjs_graph_model \
--saved_model_tags=serve \
/tmp/saved_retrained_model/ /tmp/converted_model/
creates a model.json file. Command described in https://github.com/tensorflow/tfjs-converter#step-1-converting-a-savedmodel-keras-h5-tfkeras-savedmodel-or-tensorflow-hub-module-to-a-web-friendly-format.
Yet, loading the model with tf.loadLayersModel("file:///tmp/web_model/model.json") failed with
'className' and 'config' must set.
Maybe somebody can modify retain.py to support mobileV2 use my way.
The original retrain.py link. This link is Google's GitHub code, not my link.
I changed retrain.py, the below is my git diff:
diff --git a/scripts/retrain.py b/scripts/retrain.py
index 5fa9b0f..02a4f9a 100644
--- a/scripts/retrain.py
+++ b/scripts/retrain.py
## -1,3 +1,5 ##
+# -*- coding: utf-8 -*-
+
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
## -112,6 +114,13 ## from tensorflow.python.framework import graph_util
from tensorflow.python.framework import tensor_shape
from tensorflow.python.platform import gfile
from tensorflow.python.util import compat
+from tensorflow import saved_model as sm
+from tensorflow.python.saved_model import builder as saved_model_builder
+from tensorflow.python.saved_model import signature_constants
+from tensorflow.python.saved_model import signature_def_utils
+from tensorflow.python.saved_model import tag_constants
+from tensorflow.python.saved_model import utils as saved_model_utils
+
FLAGS = None
## -319,6 +328,7 ## def maybe_download_and_extract(data_url):
Args:
data_url: Web location of the tar file containing the pretrained model.
"""
+ print(FLAGS.model_dir)
dest_directory = FLAGS.model_dir
if not os.path.exists(dest_directory):
os.makedirs(dest_directory)
## -827,6 +837,7 ## def save_graph_to_file(sess, graph, graph_file_name):
sess, graph.as_graph_def(), [FLAGS.final_tensor_name])
with gfile.FastGFile(graph_file_name, 'wb') as f:
f.write(output_graph_def.SerializeToString())
+
return
## -971,6 +982,7 ## def main(_):
# Prepare necessary directories that can be used during training
prepare_file_system()
+ sigs = {}
# Gather information about the model architecture we'll be using.
model_info = create_model_info(FLAGS.architecture)
## -1002,6 +1014,9 ## def main(_):
FLAGS.random_brightness)
with tf.Session(graph=graph) as sess:
+ serialized_tf_example = tf.placeholder(tf.string, name='tf_example')
+ feature_configs = {'x': tf.FixedLenFeature(shape=[784], dtype=tf.float32),}
+ tf_example = tf.parse_example(serialized_tf_example, feature_configs)
# Set up the image decoding sub-graph.
jpeg_data_tensor, decoded_image_tensor = add_jpeg_decoding(
model_info['input_width'], model_info['input_height'],
## -1133,6 +1148,73 ## def main(_):
(test_filename,
list(image_lists.keys())[predictions[i]]))
+ """
+ # analyze SignatureDef protobuf
+ SignatureDef_d = graph.signature_def
+ SignatureDef = SignatureDef_d[sm.signature_constants.CLASSIFY_INPUTS]
+
+ # three TensorInfo protobuf
+ X_TensorInfo = SignatureDef.inputs['input_1']
+ scale_TensorInfo = SignatureDef.inputs['input_2']
+ y_TensorInfo = SignatureDef.outputs['output']
+
+ # Tensor details
+ # .get_tensor_from_tensor_info() to get default graph
+ X = sm.utils.get_tensor_from_tensor_info(X_TensorInfo, sess.graph)
+ scale = sm.utils.get_tensor_from_tensor_info(scale_TensorInfo, sess.graph)
+ y = sm.utils.get_tensor_from_tensor_info(y_TensorInfo, sess.graph)
+ """
+
+ """
+ output_graph_def = graph_util.convert_variables_to_constants(
+ sess, graph.as_graph_def(), [FLAGS.final_tensor_name])
+
+ X_TensorInfo = sm.utils.build_tensor_info(bottleneck_input)
+ scale_TensorInfo = sm.utils.build_tensor_info(ground_truth_input)
+ y_TensorInfo = sm.utils.build_tensor_info(output_graph_def)
+
+ # build SignatureDef protobuf
+ SignatureDef = sm.signature_def_utils.build_signature_def(
+ inputs={'input_1': X_TensorInfo, 'input_2': scale_TensorInfo},
+ outputs={'output': y_TensorInfo},
+ method_name='what'
+ )
+ """
+
+ #graph = tf.get_default_graph()
+ tensors_per_node = [node.values() for node in graph.get_operations()]
+ tensor_names = [tensor.name for tensors in tensors_per_node for tensor in tensors]
+ print(tensor_names)
+
+ export_dir = './tf_files/savemode'
+ builder = saved_model_builder.SavedModelBuilder(export_dir)
+
+ # name="" is important to ensure we don't get spurious prefixing
+ graph_def = tf.GraphDef()
+ tf.import_graph_def(graph_def, name="")
+ g = tf.get_default_graph()
+ inp1 = g.get_tensor_by_name("input:0")
+ inp2 = g.get_tensor_by_name("input_1/BottleneckInputPlaceholder:0")
+ inp3 = g.get_tensor_by_name("input_1/GroundTruthInput:0")
+ out = g.get_tensor_by_name("accuracy_1:0")
+
+ sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
+ tf.saved_model.signature_def_utils.predict_signature_def(
+ {'input_1': inp1, 'input_2': inp3}, {"output": out})
+
+ builder.add_meta_graph_and_variables(sess,
+ tags=[tag_constants.SERVING],
+ signature_def_map=sigs)
+
+ """
+ builder.add_meta_graph_and_variables(
+ sess=sess,
+ tags=[tag_constants.SERVING],
+ signature_def_map={sm.signature_constants.CLASSIFY_INPUTS: SignatureDef})
+ """
+
+ builder.save()
+
# Write out the trained graph and labels with the weights stored as
# constants.
save_graph_to_file(sess, graph, FLAGS.output_graph)
Using my diff, I can generate Tensorflow Served model.
And then I use the command to convert TensorFlow served model to Tfjs model.
tensorflowjs_converter \
--input_format=tf_saved_model \
--output_format=tfjs_graph_model \
./tf_files/savemode \
./tf_files/js_model
Still unsupported Ops for lasted Tensorflow JS version.
I just make a video here to explain why we cannot convert Tensorflow frozen model to Tensorflow JS model, tells how to find the input Tensor and Output Tensor. The running steps and result, finally, give unsupported Ops ScalarSummary and the reason.
Now that I cannot change the Mobilenet Model to Tensorflow JS model, so my workaround is using Python tensorflow and flask library on Server side, user upload the image to server and then return the result.
Related
Please how can I save this model using TensorFlow SaveModel.
train_steps = int(0.5 + (1.0 * num_epochs * nusers) / batch_size)
steps_in_epoch = int(0.5 + nusers / batch_size)
print("Will train for {} steps, evaluating once every {} steps".format(train_steps, steps_in_epoch))
def experiment_fn(output_dir):
return tf.contrib.learn.Experiment(
tf.contrib.factorization.WALSMatrixFactorization(
num_rows = nusers,
num_cols = nitems,
embedding_dimension = n_embeds,
model_dir = output_dir),
train_input_fn = read_dataset(tf.estimator.ModeKeys.TRAIN, input_path,batch_size, nitems, nusers, num_epochs,n_embeds, output_dir),
eval_input_fn = read_dataset(tf.estimator.ModeKeys.EVAL, input_path, batch_size, nitems, nusers, num_epochs, n_embeds, output_dir),
train_steps = train_steps,
eval_steps = 1,
min_eval_frequency = steps_in_epoch,
export_strategies = tf.contrib.learn.utils.saved_model_export_utils.make_export_strategy(serving_input_fn = create_serving_input_fn(nitems, nusers))
)
I have tried replacing the export_strategies with export_strategies=tf.export_saved_model(output_dir, serving_input_fn = create_serving_input_fn(nitems, nusers)) and it returns the following error message
AttributeError: module 'tensorflow' has no attribute 'export_saved_model
Also tried export_strategies=tf.saved_model(output_dir, serving_input_fn = create_serving_input_fn(nitems, nusers))
TypeError: 'DeprecationWrapper' object is not callable
The SavedModel format is another way to serialize models. Models saved in this format can be restored using tf.keras.models.load_model and are compatible with TensorFlow Serving. The SavedModel guide goes into detail about how to serve/inspect the SavedModel.
The below code illustrates the steps to save and restore the model.
# Create and train a new model instance.
model = create_model()
model.fit(train_images, train_labels, epochs=5)
# Save the entire model as a SavedModel.
!mkdir -p saved_model
model.save('saved_model/my_model')
# my_model directory
ls saved_model
# Contains an assets folder, saved_model.pb, and variables folder.
ls saved_model/my_model
# Reload a fresh Keras model from the saved model:
new_model = tf.keras.models.load_model('saved_model/my_model')
I followed the website: https://leimao.github.io/blog/Save-Load-Inference-From-TF2-Frozen-Graph/
However, I still do not know how to run inference with frozen_func(see my code below).
Please advise how to run inference using pb file in TensorFlow 2.2. Thanks.
import tensorflow as tf
def wrap_frozen_graph(graph_def, inputs, outputs, print_graph=False):
def _imports_graph_def():
tf.compat.v1.import_graph_def(graph_def, name="")
wrapped_import = tf.compat.v1.wrap_function(_imports_graph_def, [])
import_graph = wrapped_import.graph
print("-" * 50)
print("Frozen model layers: ")
layers = [op.name for op in import_graph.get_operations()]
if print_graph == True:
for layer in layers:
print(layer)
print("-" * 50)
return wrapped_import.prune(
tf.nest.map_structure(import_graph.as_graph_element, inputs),
tf.nest.map_structure(import_graph.as_graph_element, outputs))
# Load frozen graph using TensorFlow 1.x functions
with tf.io.gfile.GFile("/content/drive/My Drive/Model_file/froze_graph.pb", "rb") as f:
graph_def = tf.compat.v1.GraphDef()
loaded = graph_def.ParseFromString(f.read())
# Wrap frozen graph to ConcreteFunctions
frozen_func = wrap_frozen_graph(graph_def=graph_def,
inputs=["wav_data:0"],
outputs=["labels_softmax:0"],
print_graph=True)
You can use tf.graph_util.import_graph_def inside a tf.function to do that. For example, suppose you make a test GraphDef file my_func.pb like this:
import tensorflow as tf
# Test function to make into a GraphDef file
#tf.function
def my_func(x):
return tf.square(x, name='y')
# Get graph
g = my_func.get_concrete_function(tf.TensorSpec(None, tf.float32)).graph
# Write to file
tf.io.write_graph(g, '.', 'my_func.pb', as_text=False)
You can then load it and use it like this:
import tensorflow as tf
from tensorflow.core.framework.graph_pb2 import GraphDef
# Load GraphDef
with open('my_func.pb', 'rb') as f:
gd = GraphDef()
gd.ParseFromString(f.read())
#tf.function
def my_func2(x):
# Ensure the input is a tensor of the right type
x = tf.convert_to_tensor(x, tf.float32)
# Import the graph giving x as input and getting the output y
y = tf.graph_util.import_graph_def(
gd, input_map={'x:0': x}, return_elements=['y:0'])[0]
return y
tf.print(my_func2(2))
# 4
I'm using the google research github repository to run deeplab v3+ on my dataset to segment parts of a car. The crop size I've used is 513,513 (default) and the code adds a boundary to images smaller than that size (correct me if I'm wrong).
example!
The model seems to be performing poorly on the added boundary. Is there something I'm supposed to correct or will the model do fine with more training ?
Update: Here's the tensorboard graphs for training. Why is the regularization loss shooting like that? The output seems to be improving, can someone help me making inferences from these graphs?
Is there something I'm supposed to correct or will the model do fine with more training ?
its Ok, don't mind the boundary
To inference you can use this code
import cv2
import tensorflow as tf
import numpy as np
from PIL import Image
from skimage.transform import resize
class DeepLabModel():
"""Class to load deeplab model and run inference."""
INPUT_TENSOR_NAME = 'ImageTensor:0'
OUTPUT_TENSOR_NAME = 'SemanticPredictions:0'
INPUT_SIZE = 513
def __init__(self, path):
"""Creates and loads pretrained deeplab model."""
self.graph = tf.Graph()
graph_def = None
# Extract frozen graph from tar archive.
with tf.gfile.GFile(path, 'rb')as file_handle:
graph_def = tf.GraphDef.FromString(file_handle.read())
if graph_def is None:
raise RuntimeError('Cannot find inference graph')
with self.graph.as_default():
tf.import_graph_def(graph_def, name='')
self.sess = tf.Session(graph=self.graph)
def run(self, image):
"""Runs inference on a single image.
Args:
image: A PIL.Image object, raw input image.
Returns:
seg_map: np.array. values of pixels are classes
"""
width, height = image.size
resize_ratio = 1.0 * self.INPUT_SIZE / max(width, height)
target_size = (int(resize_ratio * width), int(resize_ratio * height))
resized_image = image.convert('RGB').resize(target_size, Image.ANTIALIAS)
batch_seg_map = self.sess.run(
self.OUTPUT_TENSOR_NAME,
feed_dict={self.INPUT_TENSOR_NAME: [np.asarray(resized_image)]})
seg_map = batch_seg_map[0]
seg_map = resize(seg_map.astype(np.uint8), (height, width), preserve_range=True, order=0, anti_aliasing=False)
return seg_map
the code is based on this file https://github.com/tensorflow/models/blob/master/research/deeplab/deeplab_demo.ipynb
model = DeepLabModel(your_model_pb_path)
img = Image.open(img_path)
seg_map = model.run(img)
to get your_model_pb_path you need to export your model to .pb file
you can do it using export_model.py file in Deeplab repo
https://github.com/tensorflow/models/blob/master/research/deeplab/export_model.py
if you were training xception_65 version
python3 <path to your deeplab folder>/export_model.py \
--logtostderr \
--checkpoint_path=<your ckpt> \
--export_path="./my_model.pb" \
--model_variant="xception_65" \
--atrous_rates=6 \
--atrous_rates=12 \
--atrous_rates=18 \
--output_stride=16 \
--decoder_output_stride=4 \
--num_classes=<NUMBER OF YOUR CLASSES> \
--crop_size=513 \
--crop_size=513 \
--inference_scales=1.0
<your ckpt> is a path to your trained model checkpoint you can find checkpoints in the folder that you passed as argument --train_logdir when training
you need to include only model name and number of iterations in path, or in other words you will have in your training folder, for example, files
model-1500.meta, model-1500.index and model-1000.data-00000-of-00001 you need to discard everything that goes after ., so the ckpt path will be model-1000
please make sure that atrous_rates are the same as you used to train the model
if you were training mobilenet_v2 version
python3 <path to your deeplab folder>/export_model.py \
--logtostderr \
--checkpoint_path=<your ckpt> \
--export_path="./my_model.pb" \
--model_variant="mobilenet_v2" \
--num_classes=<NUMBER OF YOUR CLASSES> \
--crop_size=513 \
--crop_size=513 \
--inference_scales=1.0
more you can find here
https://github.com/tensorflow/models/blob/master/research/deeplab/local_test_mobilenetv2.sh
https://github.com/tensorflow/models/blob/master/research/deeplab/local_test.sh
You can visualize results using this code
img_arr = np.array(img)
# as may colors as you have classes
colors = [(255, 0, 0), (0, 255, 0), ...]
for c in range(0, N_CLASSES):
img_arr[seg_map == c] = 0.5 * img_arr[seg_map == c] + 0.5 * np.array(colors[c])
cv2.imshow(img_arr)
cv2.waitKey(0)
I have looked on several posts on stackoverflow and have been at it for a few days now, but alas, I'm not able to properly serve an object detection model through tensorflow serving.
I have visited to the following links:
How to properly serve an object detection model from Tensorflow Object Detection API?
and
https://github.com/tensorflow/tensorflow/issues/11863
Here's what I have done.
I have downloaded the ssd_mobilenet_v1_coco_11_06_2017.tar.gz, which contains the following files:
frozen_inference_graph.pb
graph.pbtxt
model.ckpt.data-00000-of-00001
model.ckpt.index
model.ckpt.meta
Using the following script, I was able successfully convert the frozen_inference_graph.pb to a SavedModel (under directory ssd_mobilenet_v1_coco_11_06_2017/saved)
import tensorflow as tf
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants
import ipdb
# Specify version 1
export_dir = './saved/1'
graph_pb = 'frozen_inference_graph.pb'
builder = tf.saved_model.builder.SavedModelBuilder(export_dir)
with tf.gfile.GFile(graph_pb, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sigs = {}
with tf.Session(graph=tf.Graph()) as sess:
# name="" is important to ensure we don't get spurious prefixing
tf.import_graph_def(graph_def, name="")
g = tf.get_default_graph()
ipdb.set_trace()
inp = g.get_tensor_by_name("image_tensor:0")
outputs = {}
outputs["detection_boxes"] = g.get_tensor_by_name('detection_boxes:0')
outputs["detection_scores"] = g.get_tensor_by_name('detection_scores:0')
outputs["detection_classes"] = g.get_tensor_by_name('detection_classes:0')
outputs["num_detections"] = g.get_tensor_by_name('num_detections:0')
output_tensor = tf.concat([tf.expand_dims(t, 0) for t in outputs], 0)
# or use tf.gather??
# out = g.get_tensor_by_name("generator/Tanh:0")
sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
tf.saved_model.signature_def_utils.predict_signature_def(
{"in": inp}, {"out": output_tensor} )
sigs["predict_images"] = \
tf.saved_model.signature_def_utils.predict_signature_def(
{"in": inp}, {"out": output_tensor} )
builder.add_meta_graph_and_variables(sess,
[tag_constants.SERVING],
signature_def_map=sigs)
builder.save()
I get the following error:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
--port=9000 --model_base_path=/serving/ssd_mobilenet_v1_coco_11_06_2017/saved
2017-09-17 22:33:21.325087: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable default found under base path /serving/ssd_mobilenet_v1_coco_11_06_2017/saved/1
I understand I will need a client to connect to the server to do the prediction. However, I'm not even able to serve the model properly.
You need to change the export signature somewhat from what the original post did. This script does the necessary changes for you:
$OBJECT_DETECTION_CONFIG=object_detection/samples/configs/ssd_mobilenet_v1_pets.config
$ python object_detection/export_inference_graph.py \ --input_type encoded_image_string_tensor \ --pipeline_config_path ${OBJECT_DETECTION_CONFIG} \ --trained_checkpoint_prefix ${YOUR_LOCAL_CHK_DIR}/model.ckpt-${CHECKPOINT_NUMBER} \ --output_directory ${YOUR_LOCAL_EXPORT_DIR}
For more details on what the program is doing, see:
https://cloud.google.com/blog/big-data/2017/09/performing-prediction-with-tensorflow-object-detection-models-on-google-cloud-machine-learning-engine
Right now we are successfully able to serve models using Tensorflow Serving. We have used following method to export the model and host it with Tensorflow Serving.
------------
For exporting
------------------
from tensorflow.contrib.session_bundle import exporter
K.set_learning_phase(0)
export_path = ... # where to save the exported graph
export_version = ... # version number (integer)
saver = tf.train.Saver(sharded=True)
model_exporter = exporter.Exporter(saver)
signature = exporter.classification_signature(input_tensor=model.input,
scores_tensor=model.output)
model_exporter.init(sess.graph.as_graph_def(),
default_graph_signature=signature)
model_exporter.export(export_path, tf.constant(export_version), sess)
--------------------------------------
For hosting
-----------------------------------------------
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_name=default --model_base_path=/serving/models
However our issue is - we want keras to be integrated with Tensorflow serving. We would like to serve the model through Tensorflow serving using Keras.
The reason we would like to have that is because - in our architecture we follow couple of different ways to train our model like deeplearning4j + Keras ,
Tensorflow + Keras, but for serving we would like to use only one servable engine that's Tensorflow Serving. We don't see any straight forward way to achieve that. Any comments ?
Thank you.
Very recently TensorFlow changed the way it exports the model, so the majority of the tutorials available on web are outdated. I honestly don't know how deeplearning4j works, but I use Keras quite often. I managed to create a simple example that I already posted on this issue in TensorFlow Serving Github.
I'm not sure whether this will help you, but I'd like to share how I did and maybe it will give you some insights. My first trial prior to creating my custom model was to use a trained model available on Keras such as VGG19. I did this as follows.
Model creation
import keras.backend as K
from keras.applications import VGG19
from keras.models import Model
# very important to do this as a first thing
K.set_learning_phase(0)
model = VGG19(include_top=True, weights='imagenet')
# The creation of a new model might be optional depending on the goal
config = model.get_config()
weights = model.get_weights()
new_model = Model.from_config(config)
new_model.set_weights(weights)
Exporting the model
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import utils
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import build_signature_def, predict_signature_def
from tensorflow.contrib.session_bundle import exporter
export_path = 'folder_to_export'
builder = saved_model_builder.SavedModelBuilder(export_path)
signature = predict_signature_def(inputs={'images': new_model.input},
outputs={'scores': new_model.output})
with K.get_session() as sess:
builder.add_meta_graph_and_variables(sess=sess,
tags=[tag_constants.SERVING],
signature_def_map={'predict': signature})
builder.save()
Some side notes
It can vary depending on Keras, TensorFlow, and TensorFlow Serving
version. I used the latest ones.
Beware of the names of the signatures, since they should be used in the client as well.
When creating the client, all preprocessing steps that are needed for the
model (preprocess_input() for example) must be executed. I didn't try
to add such step in the graph itself as Inception client example.
With respect to serving different models within the same server, I think that something similar to the creation of a model_config_file might help you. To do so, you can create a config file similar to this:
model_config_list: {
config: {
name: "my_model_1",
base_path: "/tmp/model_1",
model_platform: "tensorflow"
},
config: {
name: "my_model_2",
base_path: "/tmp/model_2",
model_platform: "tensorflow"
}
}
Finally, you can run the client like this:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --config_file=model_config.conf
try this script i wrote, you can convert keras models into tensorflow frozen graphs, ( i saw that some models give rise to strange behaviours when you export them without freezing the variables).
import sys
from keras.models import load_model
import tensorflow as tf
from keras import backend as K
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants
K.set_learning_phase(0)
K.set_image_data_format('channels_last')
INPUT_MODEL = sys.argv[1]
NUMBER_OF_OUTPUTS = 1
OUTPUT_NODE_PREFIX = 'output_node'
OUTPUT_FOLDER= 'frozen'
OUTPUT_GRAPH = 'frozen_model.pb'
OUTPUT_SERVABLE_FOLDER = sys.argv[2]
INPUT_TENSOR = sys.argv[3]
try:
model = load_model(INPUT_MODEL)
except ValueError as err:
print('Please check the input saved model file')
raise err
output = [None]*NUMBER_OF_OUTPUTS
output_node_names = [None]*NUMBER_OF_OUTPUTS
for i in range(NUMBER_OF_OUTPUTS):
output_node_names[i] = OUTPUT_NODE_PREFIX+str(i)
output[i] = tf.identity(model.outputs[i], name=output_node_names[i])
print('Output Tensor names: ', output_node_names)
sess = K.get_session()
try:
frozen_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), output_node_names)
graph_io.write_graph(frozen_graph, OUTPUT_FOLDER, OUTPUT_GRAPH, as_text=False)
print(f'Frozen graph ready for inference/serving at {OUTPUT_FOLDER}/{OUTPUT_GRAPH}')
except:
print('Error Occured')
builder = tf.saved_model.builder.SavedModelBuilder(OUTPUT_SERVABLE_FOLDER)
with tf.gfile.GFile(f'{OUTPUT_FOLDER}/{OUTPUT_GRAPH}', "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sigs = {}
OUTPUT_TENSOR = output_node_names
with tf.Session(graph=tf.Graph()) as sess:
tf.import_graph_def(graph_def, name="")
g = tf.get_default_graph()
inp = g.get_tensor_by_name(INPUT_TENSOR)
out = g.get_tensor_by_name(OUTPUT_TENSOR[0] + ':0')
sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
tf.saved_model.signature_def_utils.predict_signature_def(
{"input": inp}, {"outout": out})
builder.add_meta_graph_and_variables(sess,
[tag_constants.SERVING],
signature_def_map=sigs)
try:
builder.save()
print(f'Model ready for deployment at {OUTPUT_SERVABLE_FOLDER}/saved_model.pb')
print('Prediction signature : ')
print(sigs['serving_default'])
except:
print('Error Occured, please checked frozen graph')
I have recently added this blogpost that explain how to save a Keras model and serve it with Tensorflow Serving.
TL;DR:
Saving an Inception3 pretrained model:
### Load a pretrained inception_v3
inception_model = keras.applications.inception_v3.InceptionV3(weights='imagenet')
# Define a destination path for the model
MODEL_EXPORT_DIR = '/tmp/inception_v3'
MODEL_VERSION = 1
MODEL_EXPORT_PATH = os.path.join(MODEL_EXPORT_DIR, str(MODEL_VERSION))
# We'll need to create an input mapping, and name each of the input tensors.
# In the inception_v3 Keras model, there is only a single input and we'll name it 'image'
input_names = ['image']
name_to_input = {name: t_input for name, t_input in zip(input_names, inception_model.inputs)}
# Save the model to the MODEL_EXPORT_PATH
# Note using 'name_to_input' mapping, the names defined here will also be used for querying the service later
tf.saved_model.simple_save(
keras.backend.get_session(),
MODEL_EXPORT_PATH,
inputs=name_to_input,
outputs={t.name: t for t in inception_model.outputs})
And then starting a TF serving Docker:
Copy the saved model to the hosts' specified directory. (source=/tmp/inception_v3 in this example)
Run the docker:
docker run -d -p 8501:8501 --name keras_inception_v3 --mount type=bind,source=/tmp/inception_v3,target=/models/inception_v3 -e MODEL_NAME=inception_v3 -t tensorflow/serving
Verify that there's network access to the Tensorflow service. In order to get the local docker ip (172.*.*.*) for testing run:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' keras_inception_v3