I'm trying to implement a trained-from-scratch model onto a USB webcam attached to a google dev board. I cut the training at 15k. The model had not converged fully at 15k but I wanted to workout how to freeze the graph, tflite and move it onto a google coral dev board. I realise this may be an issue in terms of accuracy. MobileNetv2.
Below are that of a ceiling and a desk. The detection boxes appear in the exact same place each and everytime, no matter what the camera is looking at. The boxes do not move, nor does the percent change when the camera moves.
Train model
python train.py --logtostderr --train_dir=training --pipeline_config_path=training/ssd_mobilenet_v1_coco.config
Freeze model
python export_tflite_ssd_graph.py --pipeline_config_path=training/ssd_mobilenet_v2_coco.config --trained_checkpoint_prefix=training/model.ckpt-9070 --output_directory=inference_graph --add_postprocessing_op=true
Quantize
tflite_convert --graph_def_file=inference_graph/tflite_graph.pb --output_file=inference_graph/detect.tflite --inference_type=QUANTIZED_UINT8 --input_shapes=1,300,300,3 --input_arrays=normalized_input_image_tensor --output_arrays=TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3 --mean_values=128 --std_dev_values=127 --allow_custom_ops --default_ranges_min=0 --default_ranges_max=6
The model is ran as follows on the google coral dev board;
export DISPLAY=:0 && edgetpu_detect \
--source /dev/video1:YUY2:1280x720:20/1 \
--model ${DEMO_FILES}/converted_tflite_quant_model.tflite
The google coral came prepacked with a demo facial recognition model (below). The only difference is the model. The below command works perfectly and tracks my face across the screen. The above command results in the photos below.
export DISPLAY=:0 && edgetpu_detect \
--source /dev/video1:YUY2:1280x720:20/1 \
--model ${DEMO_FILES}/mobilenet_ssd_v2_face_quant_postprocess_edgetpu.tflite
What could possibly be going wrong here?
Edit, a few months later, I've retrained it with mobilenet_v1, to the exact same result.
Humm,I wonder if the source camera could be a problem also?
Can you double check if the pixel format, size, and FPS values are correct?
Also, on both images that you posted, the process seems to be loading your converted_tflite_quant_model.tflite model, are you sure you tried other model?
Were you able to run the demo with the correct result?
Related
I am trying to run deep-sort for real-time object tracking on yolov4-tiny model on webcam from this github repository.
https://github.com/theAIGuysCode/yolov4-deepsort
But there is only command for yolov4 for real time object detection using webcam. How can I modify the code and what will be the command to run on yolov4-tiny to run on webcam for real time object detection.
I will be glad if you suggest me some other way to run object tracking(less resource hungry way) or in tensorflow lite.
The answer is on the setup page https://github.com/theAIGuysCode/yolov4-deepsort. You don't need to make changes you just need to get the right weights and then run it pointing to the correct weights like:
{#Run yolov4-tiny object tracker
python object_tracker.py --weights ./checkpoints/yolov4-tiny-416 --model yolov4 --video ./data/video/test.mp4 --output ./outputs/tiny.avi --tiny}
You can look at the yolov4.py code and you will seen how it checks for normal or tiny when you give it the command line string see https://github.com/theAIGuysCode/yolov4-deepsort/blob/master/core/yolov4.py.
I trained the model according to this tutorial, but it doesn't work, seems to be too new
Custom object detection in the browser using TensorFlow.js
Training a model for custom object detection (TF 2.x) on Google Colab
After I load the model in this example, I got some errors
https://github.com/tensorflow/tfjs-models/tree/master/coco-ssd
Error：https://github.com/tensorflow/tfjs/issues/4638
I don't understand the reason, but I also don't understand the model output information
Here is a working sample code
https://github.com/mtrucc/react-ts-fpkqes
Here is the working model
https://github.com/mtrucc/react-ts-fpkqes/tree/main/public/mask_model
This is the model I trained myself and converted using the command, it doesn't work
https://github.com/mtrucc/react-ts-fpkqes/tree/main/public/mask_model2
You can switch models by modifying the path in the code
https://github.com/mtrucc/react-ts-fpkqes/blob/6cb2ce09f4c44673098749041244b0f998f8d8f4/src/components/mask.js#L35
Here is my conversion command
tensorflowjs_converter --input_format=tf_saved_model \
--output_format=tfjs_graph_model \
/content/gdrive/MyDrive/customTF2/data/inference_graph/saved_model \
/content/gdrive/MyDrive/customTF2/data/inference_graph/saved_model_js
I don't know where is the problem? If you need more details, I can add.
I am doing a research project that consists in an object detection AI, capable of detecting by a webcam 7 classes of objects.
Using google colab, I successfully trained the ssd_mobilenet_v2_quantized_300x300_coco model, using tensorflow 1.15.
The objective is to run the model in a Raspberry Pi 3b+, using the official camera and Google Coral EdgeTPU device, so the model must be quantized in order to use it.
The issue comes with the testing part; so after training the model, converted it to tflite using:
!python export_tflite_ssd_graph.py --pipeline_config_path="/content/drive/My Drive/Colab Data/models/research/object_detection/training/ssd_mobilenet_v2_quantized_300x300_coco.config" --trained_checkpoint_prefix=training/model.ckpt-28523 --output_directory=compiler/ --add_postprocessing_op=true
and
!tflite_convert --graph_def_file=compiler/tflite_graph.pb --output_file=compiler/detect.tflite --output_format=TFLITE --input_shapes=1,300,300,3 --input_arrays=normalized_input_image_tensor --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' --inference_type=QUANTIZED_UINT8 --mean_values=128 --std_dev_values=128 --change_concat_input_ranges=false --allow_custom_ops
Converted model: https://gofile.io/d/kOe2Ac
Tried to test the model using Edje Electronics webcam script. found here
And outputs this error:
RuntimeError: tensorflow/lite/kernels/detection_postprocess.cc:404 ValidateBoxes(decoded_boxes, num_boxes) was not true.Node number 98 (TFLite_Detection_PostProcess) failed to invoke.
The weirdest thing is that if I try to run the same script in my current workstation (with tensorflow 1.15.1), the code runs flawlessly, so there should be something wrong with the rpi.
The rpi is running tensorflow 1.15.2, built from the WHL source. Actually y tried with all the versions that I can, but always the same error.
I will be so grateful with any help that could bring to me. Thanks in advance.
Ok, thanks for the info! But how is that it works flawlessly on other pc and not on the rpi? Do i need to install tf-nightly instead of tensorflow? This might be of the cpu architecture?
ValidateBoxes verifies if the the xmin, xmax, ymin, ymax of the detected box makes sense or not. And the issue is usually caused by detected boxes with the same xmin/xmax or ymin/ymax, which should not be flagged as an issue, and the rest of the code and tolerant with the case very well. The issue has been fixed in TFLite and it is now available through the tf-nightly build.
If you'd like to build TF from source, here is a workaround solution.
I used Object Detection API from Tensorflow to train my model. The eval script also worked fine.
However, I want to get the coordinates of the bounding box in eval images. Is that possible to do so in eval script? How can I do it?
I used this script.
A few examples (how many is set by num_visualizations in the config file) are shown on TensorBoard.
If you want more than that, I suggest to export the model and use inference.
See the TF OD tutorial notebook for how to use an exported model for inference and visualizing results.
You can refer to the following link.
Although this does not have any relation with the eval.py script, it gets the job done. If you specify a directory consisting of the test images, you can get the bounding box coordinates of all the detected objects using the model you have trained.
An custom model takes around 4 seconds for running the Tensor Flow (TF) module of TF camera demo on a Lenovo Yoga Tab 3 tablet https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
It is after optimizing the interface using TF tools.
How to improve the latency? Can the image size to TF be reduced? Now it takes 299x299 image as input. If "Yes" what is the recommended image size? or could any in-building TF optimizing technique be used?
To improve the latency, perhaps you can try the 8-bit quantization tools provided by TensorFlow. They have a special tool just to do this: Graph Transform You can read more here.
For instance, this is what I do to my frozen model before I build it into an APK:
/home/kwotsin/tensorflow-android/tensorflow/bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=./frozen_model_mobilenet.pb \
--out_graph=./quantized_model_mobilenet.pb \
--inputs='Placeholder_only' \
--outputs='MobileNet/Predictions/Softmax' \
--transforms='
add_default_attributes
strip_unused_nodes(type=float, shape="1,299,299,3")
remove_nodes(op=Identity, op=CheckNumerics)
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms
quantize_weights
quantize_nodes
strip_unused_nodes
sort_by_execution_order'
To change the input size, I guess that will depend on what input size your model requires from the start. Currently, for my app, I use an image size of 299 as well and I can say the inference speed is very fast, typically under 1 second (of course, this depends on hardware as well. I ran it on an LG G3). If your model takes in a batch of tensors with shape [None, 299, 299, 3], then I guess your image size needs to be 299. Otherwise, you can try changing the image size in the ClassifierActivity.java file.
I don't think there's a recommended image size, since the classifier performance will depend on the task required to be done. For example, if you're doing a very simple task of simply classifying colors, I don't think you require a very high resolution image to do it. On the other hand, if you try to classify objects with very similar appearance, a larger image size will probably help in giving your classifier more information. Again, this depends on what training your model has done.