I am converting a frozen pb model to onnx, but onnx does not support uint8. How do I go about replacing uint8 with int8 in the model file? I can also retrain if I have to, but I am not sure which file to modify and how. I would have guessed that I needed to modify the file which contains the neural net architecture:
https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet_v2.py
but that does not seem to be the case...
There's a MobileNet v2 ONNX Model in the ONNX model zoo that has already been converted (although the source model was originally from mxnet). This might be useful.
ONNX operators do support uint8 datatypes, as described in the ONNX IR documentation.
Related
Is there a way to convert support vector classifier with rbf kernel to tf model?
I am aware of converting support vector classifier with linear kernel because there exists coef_ where we can find parameters and assign to tf model. Got this idea from how to convert saved model from sklearn into tensorflow/lite.
However _coef wont be there for rbf , so i am not sure on how to convert this model to an tf model.
I have retrained some tensorflow2.0 model, it's working as 1 class object detector, prepared with object detection api v2 (https://tensorflow-object-detection-api-tutorial.readthedocs.io/).
After that I have converted it to onnx (tf2onnx.convert) and tested - got the same inference results.
I have tested all pretrained models (downloaded from tf model zoo https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md):
ssd_mobilenet_v2_320x320_coco17_tpu-8
ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8
ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8
ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
I have retrained it by using some small batch of data.
The problem is with using it with gstreamer/deepstream. As I have seen, gstreamer consumes the onnx model, or model after converting it to TensorRT. (If I will provide onnx - model is also converted to TensorRT of course, but it's done by gstreamer right before running)
I was also trying to same pipeline with train->convert to onnx->convert to trt (or just provide onnx model to gstreamer). Same issue.
Error:
ERROR: [TRT]: [graph.cpp::computeInputExecutionUses::519] Error Code
9: Internal Error ((Unnamed Layer* 747) [Recurrence]: IRecurrenceLayer
cannot be used to compute a shape tensor)
TensorRT Version: 8.2.1.8
tf2onnx Version: 1.9.3
Is there any chance to get some help?
Or maybe I should skip the onnx model and just convert it from tensorflow to tensorRT engine? Is it possible?
Of course I can upload the model if it would help.
BR!
I have my own pytorch, onnx, tf1.x(frozen_graph.pb), and tf2.x(saved_model.pb) models.
Each of those model outputs is exactly same as I expected.
The original model is PyTorch model, but I converted it into the following sequences to run the model on my objective embedded platform:
PyTorch -> onnx -> tf2.x saved_model.pb -> tf1.x frozen_graph.pb
The problem is that. The objective embedded platform only supports data in NHWC format. But my whole models have NCHW format.
So the question is, is there any way to convert the data formats of the trained model from NCHW to NHWC using tf1.x or tf2.x?
Please advice.
I am trying to convert DeepLab trained on the Cityscapes dataset from here to TFLite. From viewing the frozen graph in Netron, the input and output tensors both are of type uint8. I was able to use the default DeepLab model provided for the TFLite GPU delegate, which had float32 input and output tensors. I didn't think the model was supposed to be quantized, so when trying the following code without the commented lines, I got this error:
F tensorflow/lite/toco/tooling_util.cc:2241] Check failed: array.data_type == array.final_data_type Array "ImageTensor" has mis-matching actual and final data types (data_type=uint8, final_data_type=float).
After this, I found that I should try to quantize the model. I inserted the commented lines to use uint8 instead of float32, but I got this error, which seems like an unsupported op.
F ./tensorflow/lite/toco/toco_tooling.h:38] Check failed: s.ok() Unimplemented: this graph contains anoperator of type Cast for which the quantized form is not yet implemented. Sorry, and patches welcome (that's a relatively fun patch to write, mostly providing the actual quantized arithmetic code for this op).
Is it right to use the quantized script? The off-the-shelf TFLite DeepLab model provided uses float32. Thanks!
Does the converted tensorflow lite model always have quantized calculation and output?
Or it depends on the tensorflow model's input and inference type?
It depends on the inference type.
First the input model should be instrumented with quantization operations, https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize can be helpful with that.
The resulting eval graph should be provided to TOCO for conversion with --inference_type=QUANTIZED_UINT8 and the correct --mean_values and --std_values for the input arrays.
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md for some examples of how to invoke TOCO for quantized models.
UPDATE: We have added a new post training quantization tool: https://medium.com/tensorflow/tensorflow-model-optimization-toolkit-post-training-integer-quantization-b4964a1ea9ba that should be easier than the old methods of quantization.