I beginner in Tensorflow so kindly forgive me for this simple question, but I am unable to find this answer any where. I am working on converting mobilenet Segmentation model (http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz) trained on Pascal dataset to Tensorflow-lite for mobile inference for more than a week but with no success. I am unable to define properly the input and output format for converter.
import tensorflow as tf
import numpy as np
img = tf.placeholder(name="Image", dtype=tf.float32, shape=(512,512, 3))
out = tf.placeholder(name="Output", dtype=tf.float32, shape=(512,512, 1))
localpb = 'frozen_inference_graph.pb'
tflite_file = 'retrained_graph_eyes1za.lite'
print("{} -> {}".format(localpb, tflite_file))
converter = tf.lite.TFLiteConverter.from_frozen_graph(
localpb, img, out
)
tflite_model = converter.convert()
open(tflite_file,'wb').write(tflite_model)
But, it is throwing lots of error like eager excecution. Kindly tell me how should I write the code to convert above Mobilenet model to tflite.
try this in command prompt or bash shell
You can use either of the following two ways
for tensorflow installed from package manager
python -m tensorflow.python.tools.optimize_for_inference \
--input=/path/to/frozen_inference_graph.pb \
--output=/path/to/frozen_inference_graph_stripped.pb \
--frozen_graph=True \
--input_names="sub_7" \
--output_names="ResizeBilinear_3"
build from source
bazel build tensorflow/python/tools:optimize_for_inference
bazel-bin/tensorflow/python/tools/optimize_for_inference \
--input=/path/to/frozen_inference_graph.pb \
--output=/path/to/frozen_inference_graph_stripped.pb \
--frozen_graph=True \
--input_names="sub_7" \
--output_names="ResizeBilinear_3"
Hope it works!!
Related
I have a model trained with TF 1.4 exported to a frozen inference graph with the file "models/research/object_detection/export_tflite_ssd_graph.py"
How can I can convert it tflite? I'm having a lot of issues
You can use the command line tool or the Python API.
Python API example:
converter = tf.lite.TFLiteConverter.from_frozen_graph(
graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
CLI example:
tflite_convert \
--output_file=/tmp/foo.tflite \
--graph_def_file=/tmp/mobilenet_v1_0.50_128/frozen_graph.pb \
--input_arrays=input \
--output_arrays=MobilenetV1/Predictions/Reshape_1
Tensorflow officially recommends using the Python API.
I downloaded a retrained_graph.pb and retrained_labels.txt file of a model I trained in Azure cognitive service. Now I want to make an Android app using that model and to do so I have to convert it to TFLite format. I used toco and I am getting the following error:
ValueError: Invalid tensors 'input' were found.
I am basically following this tutorial and have problem on step 4 and direcly
copy pasted the terminal code:
https://heartbeat.fritz.ai/neural-networks-on-mobile-devices-with-tensorflow-lite-a-tutorial-85b41f53230c
I am making a wild guess here, maybe you entered input_arrays=input.
Which may not be true. Use this script to find the name of the input and output arrays of the frozen inference graph
import tensorflow as tf
gf = tf.GraphDef()
m_file = open('frozen_inference_graph.pb','rb')
gf.ParseFromString(m_file.read())
with open('somefile.txt', 'a') as the_file:
for n in gf.node:
the_file.write(n.name+'\n')
file = open('somefile.txt','r')
data = file.readlines()
print "output name = "
print data[len(data)-1]
print "Input name = "
file.seek ( 0 )
print file.readline()
In my case they are:
output name: SemanticPredictions
input name: ImageTensor
You can use utility tflite_convert which is the part of tensorflow 1.10 (or higher) package.
The simple use for float inference is something like:
tflite_convert \
--output_file=/tmp/retrained_graph.tflite \
--graph_def_file=/tmp/retrained_graph.pb \
--input_arrays=input \
--output_arrays=output
Where input and output - are input and ouput tensors of your tensorflow graph
import tensorflow as tf
gf = tf.GraphDef()
m_file = open('frozen_inference_graph.pb','rb')
for n in gf.node:
print( n.name )
first one is input_arrays
last names are output_arrays (could be more than one depends on your number of output of the model)
my output
image_tensor <--- input_array
Cast
Preprocessor/map/Shape Preprocessor/map/strided_slice/stack
Preprocessor/map/strided_slice/stack_1
.
.
.
Postprocessor/BatchMultiClassNonMaxSuppression/map/
TensorArrayStack_5/TensorArrayGatherV3
Postprocessor/Cast_3
Postprocessor/Squeeze
add/y
add
detection_boxes <---output_array
detection_scores <---output_array
detection_multiclass_scores
detection_classes <---output_array
num_detections <---output_array
raw_detection_boxes
raw_detection_scores
Most of the answers here prove to be broken due to the version issues. This worked for me:
Note: First find the name of the input and output layers using Netron, as I mentioned here. In my case they are input and output.
!pip install tensorflow-gpu==1.15.0
# Convert
!toco --graph_def_file /content/yolo-v2-tiny-coco.pb \
--output_file yolo-v2-tiny-coco.tflite \
--output_format TFLITE \
--inference_type FLOAT \
--inference_input_type FLOAT \
--input_arrays input \
--output_arrays output
Also, as per zldrobit's amazing work, you can also fetch a better quantized version of this TFLite model as:
# Now let's quantize it
!toco --graph_def_file /content/yolo-v2-tiny-coco.pb \
--output_file quantized-yolo-v2-tiny-coco.tflite \
--output_format TFLITE \
--inference_type FLOAT \
--inference_input_type FLOAT \
--input_arrays input \
--output_arrays output \
--post_training_quantize
if you're using TF2 then the following will work for you to post quantized the .pb file.
import tensorflow as tf
converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph(
graph_def_file = 'path/to/frozen_inference__graph.pb',
input_arrays = ['Input_Tensor_Name'],
output_arrays = ['Output_Tensor_Name']
)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
with tf.io.gfile.GFile('model.tflite', 'wb') as f:
f.write(tflite_model)
incase if you want full int8 quantization then
import tensorflow as tf
converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph(
graph_def_file = 'path/to/frozen_inference__graph.pb',
input_arrays = ['Input_Tensor_Name'],
output_arrays = ['Output_Tensor_Name']
)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
image_shape=(input_width,input_height,no_of_channels) #change it according to your need
def representative_dataset_gen():
for i in range(10):
# creating fake images
image = tf.random.normal([1] + list(image_shape))
yield [image]
converter.representative_dataset = tf.lite.RepresentativeDataset(representative_dataset_gen)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8] # For EdgeTPU, no float ops allowed
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_model = converter.convert()
with tf.io.gfile.GFile('model.tflite', 'wb') as f:
f.write(tflite_model)
The error hints that you have not entered the correct
--input_arrays
From TF Lite Developer Guide
I quote :
"Setting the input_array and output_array arguments is not straightforward. The easiest way to find these values is to explore the graph using TensorBoard."
Using the Tensorboard isn't hard either, by simply running this command
tensorboard --logdir=path/to/log-directory
View the TensorBoard at
localhost:6006
To run the tflite converter on your local machine, you will need bazel and toco.
And if you read some issues in GitHub, in some versions of Tensrflow tflite causes a lot of trouble. To overcome this trouble, some recommend using tf-nightly!
To avoid all this, simply use Google Colab to convert your .pb into .lite or .tflite.
Since Colab started having the "upload" option for uploading your files into the current kernel, this I think is the most simple way without having to worry about other packages and their dependencies.
Here is the code for the same:
from google.colab import drive
drive.mount('/content/drive')
!cd drive/My\ Drive
from google.colab import files
pbfile = files.upload()
import tensorflow as tf
localpb = 'frozen_inference_graph_frcnn.pb'
tflite_file = 'frcnn_od.lite'
print("{} -> {}".format(localpb, tflite_file))
converter = tf.lite.TFLiteConverter.from_frozen_graph(
localpb,
["image_tensor"],
['detection_boxes']
)
tflite_model = converter.convert()
open(tflite_file,'wb').write(tflite_model)
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
"""**download optimized .lite file to local machine**"""
files.download(tflite_file)
There are two ways in which you can upload your .pb file to the current session:
i) (The easy way) After running the first cell in the above notebook, the drive will be mounted. So on your left part of the screen go to the files column and right click on the folder you want to upload your .pb file and choose upload.
Then use "ls" and "cd" commands to work your way into the folder and run the tflite converter cell.
ii) Run the cell with files.upload() command and click on browse and choose the .pb file from your local machine.
Once the file is uploaded, give its path to the variable "localpb" and also the name of the .lite model. Then simply run the cell having the "TFLiteConverter" comamnd.
And voila. You should have a tflite model appear in your drive. Simply right-click on it and download to your local machine to run inferences.
without bazel you can try the following code
pip uninstall tensorflow
pip install tf-nightly
pip show protobuf
If protobuf is version 3.6.1, then proceed to installing the pre-release version of 3.7.0.
pip uninstall protobuf
pip install protobuf==3.7.0rc2
I still couldn’t get the command line version to work. It kept returning the error: “tflite_convert: error: –input_arrays and –output_arrays are required with –graph_def_file” although both parameters were supplied. It worked in Python, however.
import tensorflow as tf
graph_def_file = "model.pb"
input_arrays = ["model_inputs"]
output_arrays = ["model_outputs"]
converter = tf.lite.TFLiteConverter.from_frozen_graph(
graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
Substituting Mul for input fixed it for me.
IMAGE_SIZE=299
tflite_convert \
--graph_def_file=tf_files/retrained_graph.pb \
--output_file=tf_files/optimized_graph.lite \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \
--input_array=Mul \
--output_array=final_result \
--inference_type=FLOAT \
--input_data_type=FLOAT
I am following up from my previous answer you can use the following script to convert your trained model on ssd mobilenet to tflte using
python object_detection/export_tflite_ssd_graph \
--pipeline_config_path ssd_0.75_export/pipeline.config \
--trained_checkpoint_prefix ssd_0.75_export/model.ckpt \
--output_directory ssd_to_tflite_output
To do this you will first need to be present in research folder of tensorflow object detection API, and change the dile path/name as per your names.
If this dosent work try running this script from research folder and rerun:
protoc object_detection/protos/*.proto --python_out=.
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
Most likely it's because during the retraining process the input and output tensors were renamed. If this is a retrained inceptionv3 graph, try using Mul as the input tensor name and final_result as the output tensor name.
bazel run --config=opt //tensorflow/contrib/lite/toco:toco -- \
... other options ...
--input_shape=1,299,299,3 \
--input_array=Mul \
--output_array=final_result
Similar adjustment if you use tflife_convert as Aleksandr suggest.
import tensorflow as tf
!tflite_convert \
--output_file "random.tflite" \
--graph_def_file "pb file path" \
--input_arrays "input tensor name" \
--output_arrays "output tensor name"
I have retrained an image classifier model on MobileNet, I have these files.
Further, I used toco to compress the retrained model to convert the model to .lite format, but I need it in .tflite format. Is there anyway I can get to tflite format from existing files?
Here is a simple python script which you can use to convert .pb format graph into tflite.
import tensorflow as tf
graph_def_file = "output_graph.pb" ##Your frozen graph
input_arrays = ["input"] ##Input Node
output_arrays = ["final_result"] ##Output Node
converter = tf.lite.TFLiteConverter.from_frozen_graph(graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open("converted_model.tflite","wb").write(tflite_model)
You can rename the .lite model to .tflite and it should work just fine.
Alternatively, with toco, you can rename the output as it is created :
toco \
--input_file=tf_files/retrained_graph.pb \
--output_file=tf_files/optimized_graph.lite \ //change this to tflite
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--input_shape=1,224,224,3 \
--input_array=input \
--output_array=final_result \
--inference_type=FLOAT \
--input_data_type=FLOAT
In order to convert TensorFlow checkpoints and GraphDef to a TensorFlow Lite FlatBuffer:
Freeze the checkpoints and graph using freeze_graph.py
Convert the frozen graph to a TensorFlow Lite FlatBuffer using
TOCO.
Your freeze_graph.py command will look similar to the following:
freeze_graph -- \
--input_graph=output_graph.pb \
--input_binary=true \
--input_checkpoint=checkpoint \
--output_graph=frozen_graph.pb \
--output_node_names= MobilenetV1/Predictions/Softmax
You can use either TocoConverter (Python API) or tflite_convert (command line tool) with your model. TocoConverter accepts a tf.Session, frozen graph def, SavedModel directory or a Keras model file. tflite_convert accepts the later three formats.
When using TOCO, specify the output_file parameter with a .tflite extension.
I have been trying TFLite to increase detection speed on Android but strangely my .tflite model now almost only detects 1 category.
I have done testing on the .pb model that I got after retraining a mobilenet and the results are good but for some reason, when I convert it to .tflite the detection is way off...
For the retraining I used the retrain.py file from Tensorflow for poets 2
I am using the following commands to retrain, optimize for inference and convert the model to tflite:
python retrain.py \
--image_dir ~/tf_files/tw/ \
--tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1 \
--output_graph ~/new_training_dir/retrainedGraph.pb \
-–saved_model_dir ~/new_training_dir/model/ \
--how_many_training_steps 500
sudo toco \
--input_file=retrainedGraph.pb \
--output_file=optimized_retrainedGraph.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TENSORFLOW_GRAPHDEF \
--input_shape=1,224,224,3 \
--input_array=Placeholder \
--output_array=final_result \
sudo toco \
--input_file=optimized_retrainedGraph.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--output_file=retrainedGraph.tflite \
--inference_type=FLOAT \
--inference_input_type=FLOAT \
--input_arrays=Placeholder \
--output_array=final_result \
--input_shapes=1,224,224,3
Am I doing anything wrong here? Where could the loss in accuracy come from?
I faced the same issue while I was trying to convert a .pb model into .lite.
In fact, my accuracy would come down from 95 to 30!
Turns out the mistake I was committing was not during the conversion of .pb to .lite or in the command involved to do so. But it was actually while loading the image and pre-processing it before it is passed into the lite model and inferred using
interpreter.invoke()
command.
The below code you see is what I meant by pre-processing:
test_image=cv2.imread(file_name)
test_image=cv2.resize(test_image,(299,299),cv2.INTER_AREA)
test_image = np.expand_dims((test_image)/255, axis=0).astype(np.float32)
interpreter.set_tensor(input_tensor_index, test_image)
interpreter.invoke()
digit = np.argmax(output()[0])
#print(digit)
prediction=result[digit]
As you can see there are two crucial commands/pre-processing done on the image once it is read using "imread()":
i) The image should be resized to the size that is the "input_height" and "input_width" values of the input image/tensor that was used during the training. In my case (inception-v3) this was 299 for both "input_height" and "input_width". (Read the documentation of the model for this value or look for this variable in the file that you used to train or retrain the model)
ii) The next command in the above code is:
test_image = np.expand_dims((test_image)/255, axis=0).astype(np.float32)
I got this from the "formulae"/model code:
test_image = np.expand_dims((test_image-input_mean)/input_std, axis=0).astype(np.float32)
Reading the documentation revealed that for my architecture input_mean = 0 and input_std = 255.
When I did the said changes to my code, I got the accuracy that was expected (90%).
Hope this helps.
Please file an issue on GitHub https://github.com/tensorflow/tensorflow/issues and add the link here.
Also please add more details on what you are retraining the last layer for.
OS Platform and Distribution: Linux Ubuntu 14.04
TensorFlow version: tensorflow (1.4.0) from binary,
CUDA/cuDNN version: cuda 8.0
I have trained a customized model with tensorflow and I am trying to make it a tensorflow lite model for mobile apps. My model defines like:
def P_Net(inputs,label=None,bbox_target=None,landmark_target=None,training=True):
#define common param
with slim.arg_scope([slim.conv2d],
activation_fn=prelu,
weights_initializer=slim.xavier_initializer(),
biases_initializer=tf.zeros_initializer(),
weights_regularizer=slim.l2_regularizer(0.0005),
padding='valid'):
print inputs.get_shape()
net = slim.conv2d(inputs, 28, 3, stride=1,scope='conv1')
......
conv4_1 = slim.conv2d(net,num_outputs=2,kernel_size=[1,1],stride=1,scope='conv4_1',activation_fn=tf.nn.softmax)
#conv4_1 = slim.conv2d(net,num_outputs=1,kernel_size=[1,1],stride=1,scope='conv4_1',activation_fn=tf.nn.sigmoid)
print conv4_1.get_shape()
#batch*H*W*4
bbox_pred = slim.conv2d(net,num_outputs=4,kernel_size=[1,1],stride=1,scope='conv4_2',activation_fn=None)
print bbox_pred.get_shape()
where conv4_1 and conv4_2 is the output layer.
I freeze the model with:
freeze_graph.freeze_graph('out_put_model/model.pb', '', False, model_path, 'Squeeze,Squeeze_1', '', '', 'out_put_model/frozen_model.pb', '', '')
After that, I could use tensorboard to view graphs. When I read it back to double check it, I get identical info to the checkpoint model.
Then, I try to save the frozen_model.pb to tensorflow lite model. While tensorflow 1.4.0 doesn't have tensorflow lite module, I checkout tensorflow from github and bazel run toco like:
bazel run --config=opt //tensorflow/contrib/lite/toco:toco -- --input_file='/home/sens/mtcnn_cat/MTCNN-Tensorflow/test/out_put_model/frozen_model.pb' --output_file='/home/sens/mtcnn_cat/MTCNN-Tensorflow/test/out_put_model/pnet.tflite' --inference_type=FLOAT --input_shape=1,128,128,3 --input_array=image_height,image_width,input_image --output_array=Squeeze,Squeeze_1 --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --dump_graphviz=/tmp
However, I get error about output array not found:
INFO: Running command line: bazel-bin/tensorflow/contrib/lite/toco/toco '--input_file=/home/sens/mtcnn_cat/MTCNN-Tensorflow/test/out_put_model/frozen_model.pb' '--output_file=/home/sens/mtcnn_cat/MTCNN-Tensorflow/test/out_put_model/pnet.tflite' '--inference_type=FLOAT' '--input_shape=1,128,128,3' '--input_array=image_height,image_width,input_image' '--output_array=Squeeze,Squeeze_1' '--input_format=TENSORFLOW_GRAPHDEF' '--output_format=TFLITE' '--dump_graphviz=/tmp'
2018-04-03 11:17:37.412589: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1172] Converting unsupported operation: Abs
2018-04-03 11:17:37.412660: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1172] Converting unsupported operation: Abs
2018-04-03 11:17:37.412699: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1172] Converting unsupported operation: Abs
2018-04-03 11:17:37.412880: F tensorflow/contrib/lite/toco/tooling_util.cc:686] Check failed: model.HasArray(output_array) Output array not found: Squeeze,Squeeze_1
Question:
How to set the --output_array=Squeeze,Squeeze_1 parameter? I think it's the same as output nodes in freeze_graph() in tensorboard, I do find the "Squeeze" and "Squeeze_1" node
How to set the --input_shape=1,128,128,3 --input_array=image_height,image_width,input_image parameter? I check and find the mobile do have a fixed size image input, but in my model, there's not fixed size of input image and fully convolution input like:
self.image_op = tf.placeholder(tf.float32, name='input_image')
self.width_op = tf.placeholder(tf.int32, name='image_width')
self.height_op = tf.placeholder(tf.int32, name='image_height')
image_reshape = tf.reshape(self.image_op, [1, self.height_op, self.width_op, 3])
and a reshape to 1widthheight*3
So, how to write this as input shape?
For Input array
[node.op.name for node in model.inputs]
For Output array
[node.op.name for node in model.outputs]
converting frozen model to tf_lite has never been a easy job, thanks to tensorflow. hope this code can help you summarize the graph and help you find output and input array
bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph={PATH_TO_FROZEN_GRAPH}/optimized_best.pb`
I came across this issue when trying to retrain and then convert to tflite.
This is the solution that worked for me:
With 1.9 and above (and possibly 1.8 too, haven't tested.) you need to drop the --input_format field and change the --input_file param to --graph_def_file
So you end up with a command that looks a bit like:
toco \
--graph_def_file=tf_files/retrained_graph.pb \
--output_file=tf_files/optimized_graph.lite \
--output_format=TFLITE \
--input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \
--input_array=input \
--output_array=final_result \
--inference_type=FLOAT \
--inference_input_type=FLOAT
I was then able to complete the poets example and get my tflite file to work on android.
Source:
https://github.com/googlecodelabs/tensorflow-for-poets-2/issues/68
You can use below tool to determine input and output array and model size along with other parameter for tflite conversion. This also creates a nice visualization of tensorflow frozen graph.
Github link for the tool