Accuracy drop for Tensorflow object detection Post Quantization - tensorflow

I am fine-tuning SSD Mobilenet v2 for a custom dataset. I am fine-tuning the model for 50k steps and quantization aware training kicks in at 48k step count.
graph_rewriter {
quantization {
delay: 48000
weight_bits: 8
activation_bits: 8
}
}
I am observing a 95%+ training, validation and testing mAP post training.
After quantization using the commands
python object_detection/export_tflite_ssd_graph.py
--pipeline_config_path=${CONFIG_FILE}
--trained_checkpoint_prefix=${CHECKPOINT_PATH}
--output_directory=${OUTPUT_DIR} --add_postprocessing_op=true
./bazel-bin/tensorflow/contrib/lite/toco/toco
--input_file=${OUTPUT_DIR}/tflite_graph.pb \
--output_file=${OUTPUT_DIR}/detect.tflite \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--inference_type=QUANTIZED_UINT8 \
--input_shapes="1,300,300,3" \
--input_arrays=normalized_input_image_tensor \
--output_arrays="TFLite_Detection_PostProcess","TFLite_Detection_PostProcess:1","TFLite_Detection_PostProcess:2","TFLite_Detection_PostProcess:3" \
--std_values=128.0 --mean_values=128.0 --allow_custom_ops --default_ranges_min=0 --default_ranges_max=6
I tested the generated detect.tflite model using same test set. I see a drop in mAP to about 85%.
Is this mAP number drop to be expected? How can I improve the post quantization mAP?

Related

How to use EfficientNet Lite models as backbone for keypoint regression?

I would like to employ EfficientNet Lite 0 model as a backbone to perform a keypoint regression task. However, I get stuck at loading the model from the either Tensorflow Hub or the official GitHub repository. Could you please explain how can I:
import such model in Tensorflow with checkpoints from ImageNet
modify the last layers of the network
modify the loss according to my task
retrain the network
I am looking forward to apply Efficient Lite since I would like to convert everything to TF Lite.
TensorFlow Lite currently doesn't support EfficientNet Lite, but they do support mobile (CPU & GPU) friendly CenterNet. See this Colab that demonstrates how to use this model.
Commands to convert the keypoints model:
# Get mobile-friendly CenterNet for Keypoint detection task.
# See TensorFlow 2 Detection Model Zoo for more details:
# https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
wget http://download.tensorflow.org/models/object_detection/tf2/20210210/centernet_mobilenetv2fpn_512x512_coco17_kpts.tar.gz
tar -xf centernet_mobilenetv2fpn_512x512_coco17_kpts.tar.gz
rm centernet_mobilenetv2fpn_512x512_coco17_kpts.tar.gz*
# Export the intermediate SavedModel that outputs 10 detections & takes in an
# image of dim 320x320.
# Modify these parameters according to your needs.
python models/research/object_detection/export_tflite_graph_tf2.py \
--pipeline_config_path=centernet_mobilenetv2_fpn_kpts/pipeline.config \
--trained_checkpoint_dir=centernet_mobilenetv2_fpn_kpts/checkpoint \
--output_directory=centernet_mobilenetv2_fpn_kpts/tflite \
--centernet_include_keypoints=true \
--keypoint_label_map_path=centernet_mobilenetv2_fpn_kpts/label_map.txt \
--max_detections=10 \
--config_override=" \
model{ \
center_net { \
image_resizer { \
fixed_shape_resizer { \
height: 320 \
width: 320 \
} \
} \
} \
}"
tflite_convert --output_file=centernet_mobilenetv2_fpn_kpts/model.tflite \
--saved_model_dir=centernet_mobilenetv2_fpn_kpts/tflite/saved_model

Reshaping tensorflow output tensors

I am training an object detection model with Azure customvision.ai. The model output is with tensorflow, either saved model .pb, .tf or .tflite.
The model output type is designated as float32[1,13,13,50]
I then push the .tflite onto a Google Coral Edge device and attempt to run it (previous .tflite models trained with Google Cloud worked, but I'm now bound to corporate Azure and need to use customvision.ai). These commands are with
$ mdt shell
$ export DEMO_FILES="/usr/lib/python3/dist*/edgetpu/demo"
$ export DISPLAY=:0 && edgetpu_detect \
$ --source /dev/video1:YUY2:1280x720:20/1 \
$ --model ${DEMO_FILES}/model.tflite
Finally, the model attempts to run, but results in a ValueError
'This model has a {}.'.format(output_tensors_sizes.size)))
ValueError: Detection model should have 4 output tensors! This model has 1.
What is happening here? How do I reshape my tensorflow model to match the device requirements of 4 output tensors?
The model that works
The model that does not work
Edit, this outputs a tflite model, but still has only one output
python tflite_convert.py \
--output_file=model.tflite \
--graph_def_file=saved_model.pb \
--saved_model_dir="C:\Users\b0588718\AppData\Roaming\Python\Python37\site-packages\tensorflow\lite\python" \
--inference_type=FLOAT \
--input_shapes=1,416,416,3 \
--input_arrays=Placeholder \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--mean_values=128 \
--std_dev_values=128 \
--allow_custom_ops \
--change_concat_input_ranges=false \
--allow_nudging_weights_to_use_fast_gemm_kernel=true
You are running an object detection demo where the engine expects 4 outputs from the model and your model only have one outputs. Maybe you had the tflite conversion incorrect? For instance, if you grabbed the Face SSD model from our zoo, conversion should be like this:
$ tflite_convert \
--output_file=face_ssd.tflite \
--graph_def_file=tflite_graph.pb \
--inference_type=QUANTIZED_UINT8 \
--input_shapes=1,320,320,3 \
--input_arrays normalized_input_image_tensor \
--output_arrays "TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3" \
--mean_values 128 \
--std_dev_values 128 \
--allow_custom_ops \
--change_concat_input_ranges=false \
--allow_nudging_weights_to_use_fast_gemm_kernel=true
Take a look at a similar query for more details:
https://github.com/google-coral/edgetpu/issues/135#issuecomment-640677917

How to convert ssd_resnet_50 tensorflow checkpoint to .tflite?

I'm trying to convert the ssd_resnet_50 model from the tensorflow Object Detection API to .tflite format but it doesn't work.
Some background:
I'm able to successfully convert the out of the box and retrained ssd_mobilenet_v2_quantized model to .tflite and run the .tflite model.
Because the ssd_resnet_50 model is not quantized, I've added the following to the ssd_resnet_50 pipeline.config file and retrained the model:
graph_rewriter {
quantization {
delay: 48000
weight_bits: 8
activation_bits: 8
}
}
After retraining ssd_resnet_50, I try to convert the model to .tflite format with the following commands:
# Produces tflite_graph.pb
python3 object_detection/export_tflite_ssd_graph.py \
--pipeline_config_path=pipeline.config \
--trained_checkpoint_prefix=model.ckpt-50000 \
--output_directory=$OUTPUT_DIR \
--add_postprocessing_op=true
# Produces detect.tflite
bazel run -c opt tensorflow/lite/toco:toco -- \
--input_file=$OUTPUT_DIR/tflite_graph.pb \
--output_file=$OUTPUT_DIR/detect.tflite \
--input_shapes=1,640,640,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--inference_type=QUANTIZED_UINT8 \
--mean_values=128 \
--std_values=128 \
--change_concat_input_ranges=false \
--allow_custom_ops
Normally, TOCO would produce a valid detect.tflite that could be run. However, TOCO runs into the following error regarding quantization and Relu6.
Can anyone help?
Error :
2019-05-21 10:41:07.885065: F tensorflow/lite/toco/tooling_util.cc:1718] Array WeightSharedConvolutionalBoxPredictor_2/BoxPredictionTower/conv2d_0/BatchNorm/feature_2/FusedBatchNorm_mul_0, which is an input to the Add operator producing the output array WeightSharedConvolutionalBoxPredictor_2/Relu6, is lacking min/max data, which is necessary for quantization. If accuracy matters, either target a non-quantized output format, or run quantized training with your model from a floating point checkpoint to change the input graph to contain min/max information. If you don't care about accuracy, you can pass --default_ranges_min= and --default_ranges_max= for easy experimentation.
run_toco.sh: line 25: 3280 Aborted (core dumped) bazel run -c opt tensorflow/lite/toco:toco -- --input_file=$OUTPUT_DIR/tflite_graph.pb --output_file=$OUTPUT_DIR/detect.tflite --input_shapes=1,640,640,3 --input_arrays=normalized_input_image_tensor --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' --inference_type=QUANTIZED_UINT8 --mean_values=128 --std_values=128 --change_concat_input_ranges=false --allow_custom_ops
Reading your error it seems that the array WeightSharedConvolutionalBoxPredictor_2/BoxPredictionTower/ conv2d_0/BatchNorm /feature_2/FusedBatchNorm_mul_0 from WeightSharedConvolutionalBoxPredictor_2/ Relu6 does not have min/max information which is needed to do post-training quantization.
You can look at Use "dummy-quantization" to try out quantized inference on a float graph. section for an example and some details.
You can add --default_ranges_min=0 --default_ranges_max=255 to your command but you will lose accuracy doing so.
bazel run -c opt tensorflow/lite/toco:toco -- \
--input_file=$OUTPUT_DIR/tflite_graph.pb \
--output_file=$OUTPUT_DIR/detect.tflite \
--input_shapes=1,640,640,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--inference_type=QUANTIZED_UINT8 \
--mean_values=128 \
--std_values=128 \
--default_ranges_min=0 \
--default_ranges_max=255 \
--change_concat_input_ranges=false \
--allow_custom_ops
From the Tensorflow Converter command line reference :
--default_ranges_min, --default_ranges_max. Type: floating-point. Default value for the (min, max) range values used for all arrays without a specified range. Allows user to proceed with quantization of non-quantized or incorrectly-quantized input files. These flags produce models with low accuracy. They are intended for easy experimentation with quantization via "dummy quantization"

TensorFlow lite: High loss in accuracy after converting model to tflite

I have been trying TFLite to increase detection speed on Android but strangely my .tflite model now almost only detects 1 category.
I have done testing on the .pb model that I got after retraining a mobilenet and the results are good but for some reason, when I convert it to .tflite the detection is way off...
For the retraining I used the retrain.py file from Tensorflow for poets 2
I am using the following commands to retrain, optimize for inference and convert the model to tflite:
python retrain.py \
--image_dir ~/tf_files/tw/ \
--tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1 \
--output_graph ~/new_training_dir/retrainedGraph.pb \
-–saved_model_dir ~/new_training_dir/model/ \
--how_many_training_steps 500
sudo toco \
--input_file=retrainedGraph.pb \
--output_file=optimized_retrainedGraph.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TENSORFLOW_GRAPHDEF \
--input_shape=1,224,224,3 \
--input_array=Placeholder \
--output_array=final_result \
sudo toco \
--input_file=optimized_retrainedGraph.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--output_file=retrainedGraph.tflite \
--inference_type=FLOAT \
--inference_input_type=FLOAT \
--input_arrays=Placeholder \
--output_array=final_result \
--input_shapes=1,224,224,3
Am I doing anything wrong here? Where could the loss in accuracy come from?
I faced the same issue while I was trying to convert a .pb model into .lite.
In fact, my accuracy would come down from 95 to 30!
Turns out the mistake I was committing was not during the conversion of .pb to .lite or in the command involved to do so. But it was actually while loading the image and pre-processing it before it is passed into the lite model and inferred using
interpreter.invoke()
command.
The below code you see is what I meant by pre-processing:
test_image=cv2.imread(file_name)
test_image=cv2.resize(test_image,(299,299),cv2.INTER_AREA)
test_image = np.expand_dims((test_image)/255, axis=0).astype(np.float32)
interpreter.set_tensor(input_tensor_index, test_image)
interpreter.invoke()
digit = np.argmax(output()[0])
#print(digit)
prediction=result[digit]
As you can see there are two crucial commands/pre-processing done on the image once it is read using "imread()":
i) The image should be resized to the size that is the "input_height" and "input_width" values of the input image/tensor that was used during the training. In my case (inception-v3) this was 299 for both "input_height" and "input_width". (Read the documentation of the model for this value or look for this variable in the file that you used to train or retrain the model)
ii) The next command in the above code is:
test_image = np.expand_dims((test_image)/255, axis=0).astype(np.float32)
I got this from the "formulae"/model code:
test_image = np.expand_dims((test_image-input_mean)/input_std, axis=0).astype(np.float32)
Reading the documentation revealed that for my architecture input_mean = 0 and input_std = 255.
When I did the said changes to my code, I got the accuracy that was expected (90%).
Hope this helps.
Please file an issue on GitHub https://github.com/tensorflow/tensorflow/issues and add the link here.
Also please add more details on what you are retraining the last layer for.

How to train TensorFlow's deeplab model on Cityscapes?

Is it possible to train the current deeplab model in TensorFlow to reasonable accuracy using 4 GPUs with 11GB? I seem to be able to fit 2 batches per GPU, so am running a total batch size of 8 across 4 clones.
Following the instructions included with the model, I get a mean IoU of < 30% after 90,000 iterations.
PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim python deeplab/train.py \
--logtostderr --training_number_of_steps=90000 \
--train_split="train" --model_variant="xception_65" \
--atrous_rates=6 --atrous_rates=12 --atrous_rates=18 \
--output_stride=16 --decoder_output_stride=4 --train_crop_size=769 \
--train_crop_size=769 --train_batch_size=8 --num_clones=4 \
--dataset="cityscapes" \
--tf_initial_checkpoint=deeplab/models/xception/model.ckpt \
--train_logdir=$LOGDIR \
--dataset_dir=deeplab/datasets/cityscapes/tfrecord
I have tried with batch norm both enabled and disabled without much difference in outcome.
Thanks!
It seems I needed a much larger step length than the default. 1e-2 gives results closer to the published results, with batch size 15 and a smaller crop window size.
if you check this link https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
It has links to pretrained models for MobileNet v2 and DeepLab trained on Cityscapes. You can modify the existing shell scripts present here to train on cityscapes.