I have MobileNetV2 based model that uses the TimeDistributed layer. I want to convert that model to a TensorFlow Lite model in order to run it on a smartphone, but there is on undefined operation.
Here is the code:
import tensorflow as tf
IMAGE_SHAPE = (224, 224, 3)
mobilenet_model = tf.keras.applications.MobileNetV2(input_shape=IMAGE_SHAPE,
include_top=False,
pooling='avg',
weights='imagenet')
inputs = tf.keras.Input(shape=(5,) + IMAGE_SHAPE)
x = tf.keras.applications.mobilenet_v2.preprocess_input(inputs)
outputs = tf.keras.layers.TimeDistributed(mobilenet_model)(x)
model = tf.keras.Model(inputs, outputs)
model.compile()
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tfmodel = converter.convert() # fails
Here is the error message:
error: failed while converting: 'main':
Some ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select
TF Select ops: Mul
Details:
tf.Mul(tensor<?x5x224x224x3xf32>, tensor<f32>) -> (tensor<?x5x224x224x3xf32>)
The error is caused by the interaction between the input preprocessing and TimeDistributed layer. If I disable the input preprocessing, then the conversion succeeds, but obviously the network will not work properly without retraining. Also models that have the preprocessing but don't have TimeDistributed layer can be converted. Is it possible to move the preprocessing to a different place to avoid this error?
Also, adding the select ops helps to convert it successfully, but I'm not sure how to enable them at runtime. I'm using the Mediapipe framework to build an Android app. and I don't think Mediapipe supports linking to extra operations.
Try this to conversion of model.
converter = TFLiteConverter.from_saved_model(model_dir)
converter.experimental_new_converter = False
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
converter.allow_custom_ops=True
tflite_model = converter.convert()
Check the documentation on how to convert it and use it inside android. Maybe you will have success with that. If not come back and ping me again.
You can give it a try without:
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
if your tensorflow version is 2.6.0 or 2.6.1
Check the allow list operators here
Best
Related
I am working on detection task on android using TFLite. However I am faced with problem conversion from SavedModel. I modified the conversion code as below and my model convert to .tflite sucsessfully but gives this error on mobile testing:
import tensorflow as tf
saved_model_dir = 'save/fine_tuning'
# tf.saved_model.save(model, saved_model_dir)
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
inference error is . I add
implementation 'org.tensorflow:tensorflow-lite-select-tf-ops:0.0.0-nightly' to dependency and model works, but inference is very slow. Can anyone help for this problem?
I've trained resnet50v2 and densenet 169 models. TensorFlow nightly 2.3.0-dev20200608. The model works fine and I tried some optimization such as "simple" tf lite, tf lite dynamic range, tf lite 16float, and they all work fine (the accuracy is either identical to the original or slightly lower as expected).
I want to convert my model to use full-integer post-training quantization with uint8. I converted my model from SavedModel format with:
converter = tf.lite.TFLiteConverter.from_saved_model('/path/to/my/saved_models')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
def representative_dataset_gen():
for i in range(100):
yield [x_train[i].astype(np.float32)]
converter.representative_dataset = representative_data_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_model = converter.convert()
with open('resnet.tflite', 'wb') as f:
f.write(tflite_model)
as written in the TensorFlow lite website. I then compiled the model for edge tpu. It works, in the sense that edge tpu allows me to run it without errors but results are gibberish. It predicts always the same value. I tried then on cpu with tf lite interpreter. Input/output tensor are correctly uint8, but again it predicts again the same value. On cpu wiht tf lite issue persists moving to int8. Has anyone else experienced the same issue?
Please find here a Google folder with the code I use to convert, the model before and after conversion, and the converted model.
https://drive.google.com/drive/folders/11XruNeJzdIm9DTn7FnuIWYaSalqg2F0B?usp=sharing
I have retrained my model on Darknet and use https://github.com/qqwweee/keras-yolo3 to convert my darknet weights to h5.
I have replace relu for leaky relu for quantization purpose.
My model then has been converted to tflite model successfully by using tf-nightly.
However, I can't parse the model to edgetpu by resize nearest neighbor error.
To my understanding, resize nearest neighbor is supported in https://coral.ai/docs/edgetpu/models-intro/#supported-operations
So why did this error happened?
Any way to fix?
Here is my tflite convert code:
import tensorflow as tf
import numpy as np
import sys
def representative_dataset_gen():
for _ in range(250):
yield [np.random.uniform(0.0, 1.0, size=(1, 416, 416, 3)).astype(np.float32)]
if len(sys.argv) != 3:
print(f"Usage: {sys.argv[0]} <keras-model> <output-filename>")
sys.exit(1)
model_fn = sys.argv[1]
out_fn = sys.argv[2]
# Convert and apply full integer quantization
converter = tf.compat.v1.lite.TFLiteConverter.from_keras_model_file(model_fn)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8,
tf.lite.OpsSet.SELECT_TF_OPS]
#converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# Set inputs and outputs of network to 8-bit unsigned integer
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_dataset_gen
tflite_model = converter.convert()
open(sys.argv[2], "wb").write(tflite_model)
ERROR
I'm from the coral team and I'm planning on investigating the yolov3 model myself, just haven't got the bandwidth to do so yet :)
Here are some tips from what I've gathered:
Users have been able to successfully compile the model after changing the leaky_relu to relu, although accuracy may decrease. I know you've mention this, but I wanted to put this on the list for other users to reference.
Secondly, I suspect that you are using tf-nightly or some newer tf version for your conversion? If so, I suggest downgrading to maybe tf2.2, some newer version of the same ops are not yet supported by the compiler.
Try turning off MLIR converter also, released version of the edgetpu_compiler doesn't play well with MLIR
lmk if you found some success, would love to give this a shot also!
fyi: I got yolov4 converted but the architect only allows 1/962 ops to run on the edgetpu so it's a bummer no go.
I'am trying to convert a tf.keras model based on mobilenetv2 with transpose convolution using latest tf-nighlty. Here is the conversion code
#saved_model_dir='/content/ksaved' # tried from saved model also
#converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter = tf.lite.TFLiteConverter.from_keras_model(reshape_model)
converter.experimental_new_converter=True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_dataset_gen
tflite_quant_modell = converter.convert()
open("/content/model_quant.tflite", "wb").write(tflite_quant_modell)
The conversion was successful(in google colab); but it has quantize and dequantize operators at the ends(as seen using netron). All operators seems to be supported. Representative data set images are float32 in generator and the model has a 4 channel float32 input by default. It looks like we need a UINT8 input and output inside model for coral TPU. How can we properly carry out this conversion?
Ref:-
Full integer quantization of weights and activations
How to quantize inputs and outputs of optimized tflite model
Coral Edge TPU Compiler cannot convert tflite model: Model not quantized
I tried with 'tf.compat.v1.lite.TFLiteConverter.from_keras_model_file' instead of v2 version.I got error: "Quantization not yet supported for op: TRANSPOSE_CONV" while trying to quantize the model in latest tf 1.15 (using representative dataset) and "Internal compiler error. Aborting! " from coral tpu compiler using tf2.0 quantized tflite
Tflite model # https://github.com/tensorflow/tensorflow/issues/31368
It seems to work until the last constitutional block (1x7x7x160)
The compiler error(Aborting) does not give any information regarding the potential cause and all types of convolutional layers seems to be supported as per coral docs.
Coral doc: https://coral.ai/docs/edgetpu/models-intro/#quantization
Here is a dummy model example of quantizing a keras model. Notice I'm using strict tf1.15 for the example, because tf2.0 deprecated:
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
with the from_keras_model api. I think the most confusing thing about this is that you can still call it but nothing happens. This means that model will still take in float inputs. I notice that you are using tf2.0, because from_keras_model is a tf2.0 api. Coral still suggest using tf1.15 for converting model for now. I suggest downgrading tensorflow or maybe even just use this (while keeping tf2.0, it may or may not work):
tf.compat.v1.lite.TFLiteConverter.from_keras_model_file
More on it here.
I always make sure not to use the experimental converter:
converter.experimental_new_converter = False
import tensorflow as tf
from tensorflow import lite
converter = tf.lite.TFLiteConverter.from_saved_model("C:/tmp")
model = converter.convert()
open("converted_model.lite", "wb").write(model)
it gives
ValueError: This converter can only convert a single ConcreteFunction. Converting multiple functions is under development.
Is there a way to fix this or converting my .pb model into a keras (h5) model???
Related questions: How do i convert tensorflow 2.0 estimator model to tensorflow lite?
Try to use a concrete function:
import tensorflow as tf
from tensorflow import lite
saved_model_obj = tf.saved_model.load(export_dir="C:/tmp")
concrete_func = saved_model_obj.signatures['serving_default']
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])
# print(saved_model_obj.signatures.keys())
# converter.optimizations = [tf.lite.Optimize.DEFAULT]
# converter.experimental_new_converter = True
model = converter.convert()
open("converted_model.lite", "wb").write(model)
serving_default is the default key for signatures in a SavedModels.
If not working try to uncomment converter.experimental_new_converter = True and the two lines above it.
Short explanation
Based on Concrete functions guide
Eager execution in TensorFlow 2 evaluates operations immediately, without building graphs.
To save the model you need graph/s which is wrapped in a python callables: a concrete functions.