If I use the Pre-Trained TFLite Object detection model in MLKit, I get the following error:
CalculatorGraph::Run() failed in Run:
Calculator::Open() for node "BoxClassifierCalculator" failed: #vk Unexpected number of dimensions for output index 0: got 3D, expected either 2D (BxN with B=1) or 4D (BxHxWxN with B=1, W=1, H=1).
Any Idea what I might be doing wrong?
ML Kit does not currently support custom object detection model yet. ML Kit currently only allows developers to use custom image classification models. All TFLite models that are compatible with ML Kit are listed here:
https://tfhub.dev/ml-kit/collections/image-classification/1
If you want to do object detection, you can try out ML Kit's Object Detection API: https://developers.google.com/ml-kit/vision/object-detection
If you want to use a custom object detection model, you can try TFLite task library:
https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview.
Well, since I found out through this post here on SO, I went on to look for other options.
And the tutorial posted here: Tensor Flow Lite OBJ detection did the trick just nicely.
First of all added the
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.2.0'
To build.gradle, and this simple code worked like a charm. Obviously I have to make some changes to reflect my specific needs, but it's detecting without errors.
val image = TensorImage.fromBitmap(bitmap)
val options = ObjectDetector.ObjectDetectorOptions.builder()
.setMaxResults(5)
.setScoreThreshold(0.5f)
.build()
val detector = ObjectDetector.createFromFileAndOptions(
this, // the application context
"model.tflite", // must be same as the filename in assets folder
options
)
A more in depth explanation is given in that link, too much to be thrown here.
Hope this helps somebody else.
You can now create MLKit Custom Models with Google's Vertex AI/AutoML Cloud service and it works well with the CameraX API using an ImageAnalysis Analyzer. Follow the guide here to make and train the TFLite model with your own images via browser (or use the Google Cloud CLI if you have many to upload). After that, this guide walks you through pretty much everything you need to add the model to your project, and how detect and label objects with MLKit.
Related
A client of mine wants to run their Google Vertex AI model on NVIDIA Jetson boards using TensorRT as accelerator. The problem with this is that their model uses certain operators (DecodeJpeg) that are not supported by ONNX. I've been able to isolate the feature extrator subgraph from the model, so everything supported by ONNX is being used, while the preprocess and postprocess will be written separate from the model.
I'm asking because I need to be provided the pre/postprocess of the model so I could implement them separately, so is there a way to get pre/postprocess from Google Vertex AI console?
I've tried running a loop that rescales the image to a squared tile from 0 to 512, but none of those gave the adequate result.
I have a custom object detection model that I can call with model = MyModel() and model.loadweights(checkpoint) and I want to evaluate it using the Object Detection API.
From what I understood there are two possibilities, either I use the legacy eval.py, there I don't know, what to put into the pipeline_config file
Or I use the newer version that is implemented in model_main_tf2.py, but there I would have to save my model as model.config and I don't know what to put the pipeline file either.
Since my model is a YOLO model, it is not included in the sample once yet.
https://github.com/tensorflow/models/tree/master/research/object_detection/configs/tf2
Would really appreciate the help!
You can't calculate the mAP using the Object Detection API because there's no pipeline.config file for Yolo.
However, you can check this repo out. It's a Tensorflow based implementation of YoloV3. They have working code for calculating mAP. You can modify this accordingly to calculate the mAP of your model.
I'm fairly new to TensorFlow so I apologize if I'm saying something absurd.
I've been playing with the PoseNet model in the browser using TensorFlow.js. In this project, I can change the algorithm and parameters so I can get better results on the detection of certain poses. The most important params in my use case are the Multiplier, Quant Bytes and Output Stride.
So far so good, I have the results I want. However, I want to convert these results to TensorFlow Lite so I can use it in an iOS application. I managed to find the PoseNet model in a TensorFlow Lite file (tflite) and I even found an iOS app example provided by TensorFlow to I'm able to load up the model file and have it working on iOS.
The problem is...I'm unable to change the params (Multiplier, Quant Bytes and Output Stride) on the iOS app. I can't find it anywhere how I can do this. I've tried searching for these params in the iOS app source code, I've tried to find ways to convert a TensorFlow.js model to TensorFlow Lite so I can load the model with the params I want in the app but no luck.
I'm writing this post so maybe you guys can point me in the right direction so I'm able to "translate" what I have on TensorFlow.js to TensorFlow Lite.
EDIT:
This is what I've learned in the last couple of days:
TFLite is designed for serving fixed model with lightweight runtime. Thus, modifying model parameters on demand is not a design goal for it.
I looked at the TF.js code for PoseNet, and found similar design. It seems you can modify parameters, because they actually have different models for each params. https://github.com/tensorflow/tfjs-models/blob/b72c10bdbdec6b04a13f780180ed904736fa52a5/posenet/src/checkpoints.ts#L37
TFLite models generally don't support dynamic parameters. Output stride Multiplier and Quant Bytes are fixed params when the neural network is created.
So what I want to do is to extract weights from TF.js model, and put then into existing MobileNet code.
And that's where I need help now. Could anyone point me in the direction to load and change the model so I can then convert it to tflite with my own params?
EDIT2:
I found a repo that is helping me convert TF.js models to TF Lite Griffin98/posenet_tfjs2tflite. I still can't define the Quant Bytes tho.
I am trying to use an object detection model, that was created using the TF Object Detection API, in TensorFlow.js.
I converted the model using:
tensorflowjs_converter --input_format=tf_saved_model inference_graph/saved_model inference_graph/web_model
It gets converted without any problems and loads in my javascript code.
Now I am a bit unsure about what my next steps should be.
I have to translate the Python into JavaScript, but certain areas I am unsure about.
With the object detection API in python, there are many steps; (1)preprocessing the image, such as convert to RGB, numpy array reshape, expand dimensions (I have an idea of how I would approach it) and (2) the run inference for single image function, I am not sure how I would go about it in tensorflow.js.
I tried to find some general information about deploying an object detection model in tensorflow.js, but I could not find much, except with pre-trained models.
Any information about this topic would be great!
Thanks!
as mentioned by #edkeveked you will need to perform similar input processing and output processing in JavaScript as is being done in Python. i can't say exactly what you will need to do since i am not familiar with the model. however, you can find an example using a specific object detection model here:
https://github.com/vabarbosa/tfjs-model-playground/blob/master/object-detector/demo/object-detector.js
see also
https://medium.com/codait/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3
You would need to replicate the same process in javascript before giving it to the model. In js, the image use by default the RGB channel, so there is no need to make that conversion again.
I am very new to CNTK.
I wanted to train a set of images (to detect objects like alcohol glasses/bottles) using CNTK - ResNet/Fast-R CNN.
I am trying to follow below documentation from GitHub; However, it does not appear to be a straight forward procedure. https://github.com/Microsoft/CNTK/wiki/Object-Detection-using-Fast-R-CNN
I cannot find proper documentation to generate ROI's for the images with different sizes and shapes. And how to create object labels based on the trained models? Can someone point out to a proper documentation or training link using which I can work on the cntk model? Please see the attached image in which I was able to load a sample image with default ROI's in the script. How do I properly set the size and label the object in the image ? Thanks in advance!
sample image loaded for training
Not sure what you mean by proper documentation. This is an implementation of the paper (https://arxiv.org/pdf/1504.08083.pdf). Looks like you are trying to generate ROI's. Can you look through the helper functions as documented at the site to parse what you might need:
To run the toy example, make sure that in PARAMETERS.py the datasetName is set to "grocery".
Run A1_GenerateInputROIs.py to generate the input ROIs for training and testing.
Run A2_RunCntk_py3.py to train a Fast R-CNN model using the CNTK Python API and compute test results.
The algo will work on several candidate regions and then generate outputs: one for the classes of objects and another one that generates the bounding boxes for the objects belonging to those classes. Please refer to the code for getting the details of the implementation.
Can someone point out to a proper documentation or training link using which I can work on the cntk model?
You can take a look at my repository on GitHub.
It will guide you through all the steps required to train your own model for object detection and classification with CNTK.
But in short the proper steps should look something like this:
Setup environment
Prepare data
Tag images (ground truth)
Download pretrained model and create mappings for your custom dataset
Run training
Evaluate the model on test set