I have a bunch of TFRecords that I used to train a model. I'd like to use them with TensorFlow Serving as well. So far, I've just been using the RESTful TensorFlow serving endpoint and have been turning TFRecords into JSON request bodies.
Is there some special way I can do inference on TFRecords directly without manually munging individual TFRecords into TF serving requests?
TFRecords are binary format, would be hard to pass through RESTFul API directly.
The alternative is to use the GRPC end point of the tf serving. But it may not save you much.
GRPC request requires tensor_proto as input, see here for an example call in Python. In this case, your tensor proto could be a one dimensional data containing a serialized tf.Example object that comes from TFRecord. When you save your model during the training phase, you can define custom serving input processing function, that can accept a serialized tf.Example data as input for serving. Refer to the tf.estimator.Estimator.export_saved_model on how to define your custom function serving_input_receiver_fn for processing inputs at serving time.
Related
here I want to train a object detection model, so I have annotated the data using roboflow and then exported it as tfrecords and also got the (.pbtxt file) and after that I don't have any clue on how to train a can model from scratch with just 2,3 number of hidden layers. am not getting on how to use that tfrecord to fit in my model which I have created. please help me out.
tfrecord files are usually used with Tensorflow Object Detection. It's pretty old and I haven't seen it used in practice recently, but there's a Tensorflow Object Detection tutorial here that uses these tfrecord files.
If there's not a particular reason you need to use TF Object Detection I'd recommend using a newer and more well-supported model like YOLOv5 or YOLOv7.
I am trying to use this for my object detection task. The problems I am facing are:
On running the saved_model_cli command, I am getting the following output. There is no signature defined with tag-set "serve" also the method name is empty
The variable folder in the model directory only contains a few bytes of data which means the weights are not actually written to disk.
The model format seems to be HubModule V1 which seems to be the issue, any tips on making the above model servable are highly appreciated.
TF2 SavedModels should not have this problem, only Hub.Modules from TF1 since Hub.Modules use the signatures for other purposes. You can take a hub.Module and build a servable SavedModel, but it's quite complex and involves building the signatures yourself.
Instead, I recommend checking out the list of TF2 object detection models on TFHub.dev for a model you can use instead of the model you are using: https://tfhub.dev/s?module-type=image-object-detection&tf-version=tf2
These models should be servable with TF Serving
I have an image classification deep learning CNN model (.h5 file) trained using Keras and Tensorflow 2 that I want to use online for predictions. I want an API that takes the single input image over HTTP and responds with the predicted class labels using the trained model. Is there an API provided by Keras or Tensorflow to do the same?
There's two basic options:
Use TensorFlow Serving - it provides ready-to-go REST API server, the only thing that you need to do is to convert your model to .pb format.
Write your own simple REST server (on Flask, for example) which will call model.predict() on the inputs (that approach may be easier to start with, but it will be hard to scale/optimize for heavy load.
Is there good documentation/explanation of what actually tensorflow input_fn and serving_input_receiver_fn does?
What is the internal concept behind these functions?
From the documentation:
During training, an input_fn() ingests data and prepares it for use by the model. At serving time, similarly, a serving_input_receiver_fn() accepts inference requests and prepares them for the model.
I am trying to serve the tensorflow object detection api model in tensorflow serving, And I am confused by the 3 different SignatureDefs. What are the differences, When to choose one over another?
Tensorflow Serving uses a different way of updating models weights and different signature mechanism is used in serving. In order to save model in serving se uses SavedModel. SavedModel provides a language-neutral format to save machine-learned models that is recoverable and hermetic. It enables higher-level systems and tools to produce, consume and transform TensorFlow models.
This support SignatureDefs
Graphs that are used for inference tasks typically have a set of inputs and outputs. This is called a Signature.
SavedModel uses SignatureDefs to allow generic support for signatures that may need to be saved with the graphs.
For those who previously used TF-Exporter/SessionBundle, Signatures in TF-Exporter will be replaced by SignatureDefs in SavedModel.
A SignatureDef requires specification of:
inputs as a map of string to TensorInfo.
outputs as a map of string to TensorInfo.
method_name (which corresponds to a supported method name in the loading tool/system).
Classification SignatureDefs support structured calls to TensorFlow Serving's Classification API. These prescribe that there must be an inputs Tensor, and that there are two optional output Tensors: classes and scores, at least one of which must be present.
Predict SignatureDefs support calls to TensorFlow Serving's Predict API. These signatures allow you to flexibly support arbitrarily many input and output Tensors. For the example below, the signature my_prediction_signature has a single logical input Tensor images that are mapped to the actual Tensor in your graph x:0.
Regression SignatureDefs support structured calls to TensorFlow Serving's Regression API. These prescribe that there must be exactly one inputs Tensor, and one outputs Tensor.
Please refer:
https://www.tensorflow.org/serving/signature_defs
https://github.com/tensorflow/serving/issues/599
The Classify API is higher-level and more specific than the Predict API. Classify accepts tensorflow.serving.Input (which wraps a list of tf.Examples) as input and produces classes and scores as output. It is used for classification problems. Predict, on the other than, accepts tensors as input and outputs tensors. It can be used for regression, classification and other types of inference problems.