I have my own pytorch, onnx, tf1.x(frozen_graph.pb), and tf2.x(saved_model.pb) models.
Each of those model outputs is exactly same as I expected.
The original model is PyTorch model, but I converted it into the following sequences to run the model on my objective embedded platform:
PyTorch -> onnx -> tf2.x saved_model.pb -> tf1.x frozen_graph.pb
The problem is that. The objective embedded platform only supports data in NHWC format. But my whole models have NCHW format.
So the question is, is there any way to convert the data formats of the trained model from NCHW to NHWC using tf1.x or tf2.x?
Please advice.
Related
Is there a way to convert support vector classifier with rbf kernel to tf model?
I am aware of converting support vector classifier with linear kernel because there exists coef_ where we can find parameters and assign to tf model. Got this idea from how to convert saved model from sklearn into tensorflow/lite.
However _coef wont be there for rbf , so i am not sure on how to convert this model to an tf model.
I have retrained some tensorflow2.0 model, it's working as 1 class object detector, prepared with object detection api v2 (https://tensorflow-object-detection-api-tutorial.readthedocs.io/).
After that I have converted it to onnx (tf2onnx.convert) and tested - got the same inference results.
I have tested all pretrained models (downloaded from tf model zoo https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md):
ssd_mobilenet_v2_320x320_coco17_tpu-8
ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8
ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8
ssd_resnet50_v1_fpn_640x640_coco17_tpu-8
I have retrained it by using some small batch of data.
The problem is with using it with gstreamer/deepstream. As I have seen, gstreamer consumes the onnx model, or model after converting it to TensorRT. (If I will provide onnx - model is also converted to TensorRT of course, but it's done by gstreamer right before running)
I was also trying to same pipeline with train->convert to onnx->convert to trt (or just provide onnx model to gstreamer). Same issue.
Error:
ERROR: [TRT]: [graph.cpp::computeInputExecutionUses::519] Error Code
9: Internal Error ((Unnamed Layer* 747) [Recurrence]: IRecurrenceLayer
cannot be used to compute a shape tensor)
TensorRT Version: 8.2.1.8
tf2onnx Version: 1.9.3
Is there any chance to get some help?
Or maybe I should skip the onnx model and just convert it from tensorflow to tensorRT engine? Is it possible?
Of course I can upload the model if it would help.
BR!
I am converting a frozen pb model to onnx, but onnx does not support uint8. How do I go about replacing uint8 with int8 in the model file? I can also retrain if I have to, but I am not sure which file to modify and how. I would have guessed that I needed to modify the file which contains the neural net architecture:
https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet_v2.py
but that does not seem to be the case...
There's a MobileNet v2 ONNX Model in the ONNX model zoo that has already been converted (although the source model was originally from mxnet). This might be useful.
ONNX operators do support uint8 datatypes, as described in the ONNX IR documentation.
I've trained a simple CNN model on Cifar-10 in tensorflow with fake quantization (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize). I then generated a .tflite file using toco. Now I want to use a python interpreter to test the tflite model.
Since I used tf.image.per_image_standardization to subtract mean and divide by variance during training. I need to do the same thing to the testing data right? But, the problem is, my model is already fully quantized by tflite, and it only takes uint8 data as inputs. To do image standardization, I need to convert my image to float32. So how do I convert it back to uint8, or is image standardization even necessary for the testing data in this case? Thanks.
So, it turns out I need to do standardization on the testing data for a good accuracy.
To do it, I directly feed uint8 input images to the tf.image.per_image_standardization function. The function would convert the uint8 data to float32, and then do standardization (subtract mean, divide by std). You can find source code of the function here: https://github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/python/ops/image_ops_impl.py
Now, I have the standardized float32 input images. What I did is writing a quantization function to quantize the float32 images back to uint8. The math comes from this paper: https://arxiv.org/abs/1803.08607
Now, I have the standardized uint8 input images, I then use tflite interpreter python API to test the model. It works as expected.
Does the converted tensorflow lite model always have quantized calculation and output?
Or it depends on the tensorflow model's input and inference type?
It depends on the inference type.
First the input model should be instrumented with quantization operations, https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize can be helpful with that.
The resulting eval graph should be provided to TOCO for conversion with --inference_type=QUANTIZED_UINT8 and the correct --mean_values and --std_values for the input arrays.
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md for some examples of how to invoke TOCO for quantized models.
UPDATE: We have added a new post training quantization tool: https://medium.com/tensorflow/tensorflow-model-optimization-toolkit-post-training-integer-quantization-b4964a1ea9ba that should be easier than the old methods of quantization.