I have used tensorflow.js to do person segmentation of live video in a web browser. Is there a similar tensorflowlite/mobile version of the same? Is there any android SDK available to do person/video segmentation on mobile clients? Any pointers would be very helpful.
thanks
Sowmya
Yes, TensorFlow Lite is available for Android. Click here for more.
You can use TensorFlow Lite's Segmentation Model. You can see an example GIF on the page also. The segmentation model ( DeepLab v3 ) can segment various objects in an image. Since we are concerned with person segmentation, DeepLabv3 can handle that too.
If you want to use BodyPix, which is a TensorFlow.js implementation, you need to find the model's frozen graph along with the input/output nodes. Only then you can get it into TensorFlow Lite. On investigating the repo, I found this file which contains the URLs for the model.
Else, we can use Real-Time Pose Estimation, which is also a TF JS implementation.
Related
I'm looking into training an object detection network using Tensorflow, and I had a look at the TF2 Model Zoo. I noticed that there are noticeably less models there than in the directory /models/research/models/, including the MobileDet with SSDLite developed for the jetson xavier.
To clarify, the readme says that there is a MobileDet GPU with SSDLite, and that the model and checkpoints trained on COCO are provided, yet I couldn't find them anywhere in the repo.
How is one supposed to use those models?
I already have a custom-trained MobileDetv3 for image classification, and I was hoping to see a way to turn the network into an object detection network, in accordance with the MobileDetv3 paper. If this is not straightforward, training one network from scratch could be ok too, I just need to know where to even start from.
If you plan to use the object detection API, you can't use your existing model. You have to choose from a list of models here for v2 and here for v1
The documentation is very well maintained and the steps to train or validate or run inference (test) on custom data is very well explained here by the TensorFlow team. The link is meant for TensorFlow version v2. However, if you wish to use v1, the process is fairly similar and there are numerous blogs/videos explaining how to go about it
If I use the Pre-Trained TFLite Object detection model in MLKit, I get the following error:
CalculatorGraph::Run() failed in Run:
Calculator::Open() for node "BoxClassifierCalculator" failed: #vk Unexpected number of dimensions for output index 0: got 3D, expected either 2D (BxN with B=1) or 4D (BxHxWxN with B=1, W=1, H=1).
Any Idea what I might be doing wrong?
ML Kit does not currently support custom object detection model yet. ML Kit currently only allows developers to use custom image classification models. All TFLite models that are compatible with ML Kit are listed here:
https://tfhub.dev/ml-kit/collections/image-classification/1
If you want to do object detection, you can try out ML Kit's Object Detection API: https://developers.google.com/ml-kit/vision/object-detection
If you want to use a custom object detection model, you can try TFLite task library:
https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview.
Well, since I found out through this post here on SO, I went on to look for other options.
And the tutorial posted here: Tensor Flow Lite OBJ detection did the trick just nicely.
First of all added the
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.2.0'
To build.gradle, and this simple code worked like a charm. Obviously I have to make some changes to reflect my specific needs, but it's detecting without errors.
val image = TensorImage.fromBitmap(bitmap)
val options = ObjectDetector.ObjectDetectorOptions.builder()
.setMaxResults(5)
.setScoreThreshold(0.5f)
.build()
val detector = ObjectDetector.createFromFileAndOptions(
this, // the application context
"model.tflite", // must be same as the filename in assets folder
options
)
A more in depth explanation is given in that link, too much to be thrown here.
Hope this helps somebody else.
You can now create MLKit Custom Models with Google's Vertex AI/AutoML Cloud service and it works well with the CameraX API using an ImageAnalysis Analyzer. Follow the guide here to make and train the TFLite model with your own images via browser (or use the Google Cloud CLI if you have many to upload). After that, this guide walks you through pretty much everything you need to add the model to your project, and how detect and label objects with MLKit.
Let's assume I made an app that has machine learning in it using a tflite file.
Is it possible that I could retrain this model right inside the app?
I have tried to use the Model Maker which is provided by TensorFlow, but, without this, i don't think there's any other way to retrain your model with just the app i made.
Do you mean training on the device when the app is deployed? If yes, TFLite currently doesn't support training in general. But there's some experimental work in this direction with limited support as shown by https://github.com/tensorflow/examples/blob/master/lite/examples/model_personalization.
Currently, the retraining of a TFLite model, as you found out w/ Model Maker, has to happen offline w/ TF before the app is deployed.
My goal is to test out Google's BERT algorithm in Google Colab.
I'd like to use a pre-trained custom model for Finnish (https://github.com/TurkuNLP/FinBERT). The model can not be found on TFHub library. I have not found a way to load model with Tensorflow Hub.
Is there a neat way to load and use a custom model with Tensorflow Hub?
Fundamentally: yes. Everyone can create the kind of models that TF Hub hosts, and I hope authors of interesting models do consider that.
For TF1 and the hub.Module format tailored to it, see
https://www.tensorflow.org/hub/tf1_hub_module#creating_a_new_module
For TF2 and its revised SavedModel format, see
https://www.tensorflow.org/hub/tf2_saved_model#creating_savedmodels_for_tf_hub
That said, a sophisticated model like BERT requires a bit of attention to export it with all bells and whistles, so it helps to have some tooling to build on. The BERT reference implementation for TF2 at https://github.com/tensorflow/models/tree/master/official/nlp/bert comes with an open-sourced export_tfhub.py script, and anyone can use that to export custom BERT instances created from that code base.
However, I understand from https://github.com/TurkuNLP/FinBERT/blob/master/nlpl_tutorial/training_bert.md#general-info that you are using Nvidia's fork of the original TF1 implementation of BERT. There are Hub modules created from the original research code, but the tooling to that end has not been open-sourced, and Nvidia doesn't seem to have added their own either.
If that's not changing, you'll probably have to resort to doing things the pedestrian way and get acquainted with their codebase and load their checkpoints into it.
I am working on project Audio classifier app. I am beginner in java Also if some one can help me figure out how to extract MFFC features from audio signal I would like give them credit,also provide me contact details if you are interested.
In Android, you can use TensorFlow Lite , a lightweight solution for TensorFlow.
You can convert a TensorFlow model, or Keras model to a TF Lite model ( .tflite ). See here.
This TF Lite model could be inferenced on an Android or iOS device using the TF Lite's Java and Swift API.
It may not support some layers like LSTM or BatchNormalization. Dense and all Conv layers work well.
Another method is using Tensorflow Mobile.
TensorFlow Mobile runs a protocol buffers file ( .pb ). It has been deprecated but can be still used. But, Google suggests its developers to use TF Lite.
You can find a complete tutorial here.