Is there a general statement if or how much the accuracy and runtime at the inference decrease when using a TFLite model (.tflite) instead the original TensorFlow model (.h5)?
The simple answer is no. This really relies on how the model looks like along with what sort of optimization was applied on converting the H5 to tflite. But generally if you do not apply any sort of additional optimization but just performed the float32 conversion to tflite, I'd say most of the time tflite would generate the equivalent level of accuracy compared to the original model.
Related
I’m transforming a TensorFlow model to Pytorch. And I’d like to initialize the mean and variance of BatchNorm2d using TensorFlow model.
I’m doing it in this way:
bn.running_mean = torch.nn.Parameter(torch.Tensor(TF_param))
And I get this error:
RuntimeError: the derivative for 'running_mean' is not implemented
But is works for bn.weight and bn.bias. Is there any way to initialize the mean and variance using my pre-trained Tensorflow model? Is there anything like moving_mean_initializer and moving_variance_initializer in Pytorch?
Thanks!
The running mean and variance of a batch norm layer are not nn.Parameters, but rather a buffer of the layer.
I think you can simply assign a torch.tensor, no need to wrap a nn.Parameter around it.
I've trained a simple CNN model on Cifar-10 in tensorflow with fake quantization (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize). I then generated a .tflite file using toco. Now I want to use a python interpreter to test the tflite model.
Since I used tf.image.per_image_standardization to subtract mean and divide by variance during training. I need to do the same thing to the testing data right? But, the problem is, my model is already fully quantized by tflite, and it only takes uint8 data as inputs. To do image standardization, I need to convert my image to float32. So how do I convert it back to uint8, or is image standardization even necessary for the testing data in this case? Thanks.
So, it turns out I need to do standardization on the testing data for a good accuracy.
To do it, I directly feed uint8 input images to the tf.image.per_image_standardization function. The function would convert the uint8 data to float32, and then do standardization (subtract mean, divide by std). You can find source code of the function here: https://github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/python/ops/image_ops_impl.py
Now, I have the standardized float32 input images. What I did is writing a quantization function to quantize the float32 images back to uint8. The math comes from this paper: https://arxiv.org/abs/1803.08607
Now, I have the standardized uint8 input images, I then use tflite interpreter python API to test the model. It works as expected.
Does the converted tensorflow lite model always have quantized calculation and output?
Or it depends on the tensorflow model's input and inference type?
It depends on the inference type.
First the input model should be instrumented with quantization operations, https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize can be helpful with that.
The resulting eval graph should be provided to TOCO for conversion with --inference_type=QUANTIZED_UINT8 and the correct --mean_values and --std_values for the input arrays.
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/toco/g3doc/cmdline_examples.md for some examples of how to invoke TOCO for quantized models.
UPDATE: We have added a new post training quantization tool: https://medium.com/tensorflow/tensorflow-model-optimization-toolkit-post-training-integer-quantization-b4964a1ea9ba that should be easier than the old methods of quantization.
When I started with Neural it seemed I understood Optimizers and Estimators well.
Estimators: Classifier to classify the value based on sample set and Regressor to predict the value based on sample set.
Optimizer: Using different optimizers (Adam, GradientDescentOptimizer) to minimise the loss function, which could be complex.
I understand every estimators come up with an Default optimizer internally to perform minimising the loss.
Now my question is how do they fit in together and optimize the machine training?
short answer: loss function link them together.
for example, if you are doing a classification, your classifier can take input and output a prediction. then you can calculate your loss by take predicted class and ground truth class. the task of your optimizer is to minimize the loss by modifying the parameter of your classifier.
I did quantization on inception-resnet-v2 model using https://www.tensorflow.org/performance/quantization#how_can_you_quantize_your_models.
Size of freezed graph(input for quantization) is 224,6 MB and quantized graph is 58,6 MB. I ran accuracy test for certain dataset wherein, for freezed graph the accuracy is 97.4% whereas for quantized graph it is 0%.
Is there a different way to quantize the model for inception-resnet versions? or, for inception-resnet model, quantization is not support at all?
I think they transitioned from quantize_graph to graph_transforms. Try using this:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms
And what did you use for the input nodes/output nodes when testing?