I have quantized a Tensorflow model using Tensorflow Lite, then I run inference on it. It seems that it only uses CPU.
Is it possible to run inference with quantized model on GPU?
Related
I have a trained Tensorflow model and a Tensorflow Lite model converted from it. If I provide them with the same input, should they give me the exact same output?
Is it possible to re-quantize already quantized models?
I have some models that I have trained with Quantization Aware Training (QAT) with Full Integer Quantization. However, I am failing to do GPU Delegation with those models. Is there a way to make the models I already have with Float16 Quantization in order to be able to run them with GPU Delegate.
Are you looking for some ways to convert integer quantized model fo float16 quantized model?
Which version of TFLite are you using? TFLite 2.3 supports running quantized models with GPU delegate. However, as GPU only supports float operations, so it internally dequantize integer weights into float weights.
Please see the doc for how to enable (experimental) quantized model support. https://www.tensorflow.org/lite/performance/gpu_advanced#running_quantized_models_experimental
I need to quantify my model for use in the full version of tensorflow. And I do not find how to do this (in the official manual for quantization of the model, the model is saved in the format tflite)
AFAIK the only supported quantization scheme in tensorflow is tflite. What do you plan to do with a quantized tensorflow graph? If it is inference only, why not simply use tflite?
I am training to train tensorflow with cpu,it is slow, i have no problem with this but my question is
"will my mdel be trained as like gpu, is the only difference between
cpu and gpu in deep learning is speed?
my train code
python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config
I am using windows 10
as you know the SVM in ski-learn doesn't support GPU. by the way, the Tensorflow has a SVM module but I don't know tensorflow's svm works on GPU or not? if it doesn't support GPU can I force it to use GPU by code?