I want to migrate my code for fine-tuning an object detection model for inference on Coral devices to TensorFlow 2, but I don't see quantized models in the TF2 model zoo.
Is it possible to fine-tune a model in TF2 for this purpose and use a technique like quantization-aware training or post-training quantization? I haven't seen any related tutorials or issues. I've also seen some reports of issues with quantization with TFLite converter in TF2 so I'm not even sure if it's possible to do it in TF2.
Even for the TF1.x models, the conversions are not that straightforward which I am struggling with them at this time. I am pretty disappointed as I think Coral should have better support for tensorflow than Intel NCS USB originally as they are in the same family. However, it seems that I am wrong.
Quantization aware training is possible by adding graph_rewriter at the end of the config file before fine-tuning of the pretrained model:
graph_rewriter {
quantization {
delay: 48000
weight_bits: 8
activation_bits: 8
}
}
Source: https://neuralet.com/article/quantization-of-tensorflow-object-detection-api-models/
Related
I'm using tensorflow's object detection api to train my own object detection model. Currently using ssd_inception_v2 on tensorflow 1.15. I've trained many models previously using this api but what I'm trying to do is improve my inference time. I want to try quantization aware training I've found out by adding those lines to config file I can do qat.
graph_rewriter {
quantization {
delay: 48000
weight_bits: 8
activation_bits: 8
}
}
Now my question is after I have done the training do I need to convert my frozen model to tflite in order to make use of 8 bit precision? Or if I use my frozen model directly would it use 8bit precision? If not how do I convert my pb to tflite and do inference on that?
How do I find more info on how the ssd_mobilenet_v1 tflite model on TFHub was trained?
Was it trained in such a way that made it easy to convert it to tflite by avoiding certain ops not supported by tflite? Or was it trained normally, and then converted using the tflite converter with TF Select and the tips on this github issue?
Also, does anyone know if there's an equivalent mobilenet tflite model trained on OpenImagesV6? If not, what's the best starting point for training one?
I am not sure about about the exact origin of the model, but looks like it does have TFLite-compatible ops. From my experience, the best place to start for TFLite-compatible SSD models is with the TF2 Detection Zoo. You can convert any of the SSD models using these instructions.
To train your own model, you can follow these instructions that leverage Google Cloud.
Is it possible to re-quantize already quantized models?
I have some models that I have trained with Quantization Aware Training (QAT) with Full Integer Quantization. However, I am failing to do GPU Delegation with those models. Is there a way to make the models I already have with Float16 Quantization in order to be able to run them with GPU Delegate.
Are you looking for some ways to convert integer quantized model fo float16 quantized model?
Which version of TFLite are you using? TFLite 2.3 supports running quantized models with GPU delegate. However, as GPU only supports float operations, so it internally dequantize integer weights into float weights.
Please see the doc for how to enable (experimental) quantized model support. https://www.tensorflow.org/lite/performance/gpu_advanced#running_quantized_models_experimental
I have trained my custom data-set using the Tensor Flow Object Detection API. I run my "prediction" script and it works fine on the GPU. Now , I want to convert the model to lite and run it on the Google Coral Edge TPU Board to detect my custom objects. I have gone through the documentation that Google Coral Board Website provides but I found it very confusing.
How to convert and run it on the Google Coral Edge TPU Board?
Thanks
Without reading the documentation, it will be very hard to continue. I'm not sure what your "prediction script" means, but I'm assuming that the script loaded a .pb tensorflow model, loaded some image data, and run inference on it to produce prediction results. That means you have a .pb tensorflow model at the "Frozen graph" stage of the following pipeline:
Image taken from coral.ai.
The next step would be to convert your .pb model to a "fully quantized .tflite model" using the post training quantization technique. The documentation to do that are given here. I also created a github gist, containing an example of Post Training Quantization here. Once you have produced the .tflite model, you'll need to compile the model via the edgetpu_compiler. Although everything you need to know about the edgetpu compiler is in that link, for your purpose, compiling a model is as simple as:
$ edgetpu_compiler your_model_name.tflite
Which will creates a your_model_name_edgetpu.tflite model that is compatible with the EdgeTPU. Now, if at this stage, instead of creating an edgetpu compatible model, you are getting some type of errors, then that means your model did not meets the requirements that are posted in the models-requirements section.
Once you have produced a compiled model, you can then deploy it on an edgetpu device. Currently are 2 main APIs that can be use to run inference with the model:
EdgeTPU API
python api
C++ api
tflite API
C++ api
python api
Ultimately, there are many demo examples to run inference on the model here.
The previous answer works with general classification models, but not with TF object detection API trained models.
You cannot do post-training quantization with TF Lite converter on TF object detection API models.
In order to run object detection models on EdgeTPU-s:
You must train the models in quantized aware training mode with this addition in model config:
graph_rewriter {
quantization {
delay: 48000
weight_bits: 8
activation_bits: 8
}
}
This might not work with all the models provided in the model-zoo, try a quantized model first.
After training, export the frozen graph with: object_detection/export_tflite_ssd_graph.py
Run tensorflow/lite/toco tool on the frozen graph to make it TFLite compatible
And finally run edgetpu_complier on the .tflite file
You can find more in-depth guide here:
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md
I downloaded a tensorflow model from Custom Vision and want to run it on a coral tpu. I therefore converted it to tensorflow-lite and applying hybrid post-training quantization (as far as I know that's the only way because I do not have access to the training data).
You can see the code here: https://colab.research.google.com/drive/1uc2-Yb9Ths6lEPw6ngRpfdLAgBHMxICk
When I then try to compile it for the edge tpu, I get the following:
Edge TPU Compiler version 2.0.258810407
INFO: Initialized TensorFlow Lite runtime.
Invalid model: model.tflite
Model not quantized
Any idea what my problem might be?
tflite models are not fully quantized using converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]. You might have a look on post training full integer quantization using the representation dataset: https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization_of_weights_and_activations Simply adapt your generator function to yield representative samples (e.g. similar images, to what your image classification network should predict). Very few images are enough for the converter to identify min and max values and quantize your model. However, typically your accuracy is less in comparison to quantization aware learning.
I can't find the source but I believe the edge TPU currently only supports 8bit-quantized models, and no hybrid operators.
EDIT: On Corals FAQ they mention that the model needs to be fully quantized.
You need to convert your model to TensorFlow Lite and it must be
quantized using either quantization-aware training (recommended) or
full integer post-training quantization.