How to a make a model ready for TensorFlow Serving REST interface with a base64 encoded image? - tensorflow-serving

Question

My understanding is that I should be able to grab a TensorFlow model from Google's AI Hub, deploy it to TensorFlow Serving and use it to make predictions by POSTing images via REST requests using curl.
I could not find any bbox predictors on AI Hub at this time but I did find one on the TensorFlow model zoo:
http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz
I have the model deployed to TensorFlow serving, but the documentation is unclear with respect to exactly what should be included in the JSON of the REST request.
My understanding is that
The SignatureDefinition of the model determines what the JSON should look like
I should base64 encode the images
I was able to get the signature definition of the model like so:
>python tensorflow/tensorflow/python/tools/saved_model_cli.py show --dir /Users/alexryan/alpine/git/tfserving-tutorial3/model-volume/models/bbox/1/ --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['in'] tensor_info:
dtype: DT_UINT8
shape: (-1, -1, -1, 3)
name: image_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['out'] tensor_info:
dtype: DT_FLOAT
shape: unknown_rank
name: detection_boxes:0
Method name is: tensorflow/serving/predict
I think the shape info here is telling me that the model can handle images of any dimensions?
The input layer looks like this in Tensorboard:
But how do I convert this SignatureDefinition to a valid JSON request?
I'm assuming that I'm supposed to use the predict API ...
and Google's doc says ...
URL
POST
http://host:port/v1/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]:predict
/versions/${MODEL_VERSION} is optional. If omitted the latest version
is used.
Request format
The request body for predict API must be JSON object
formatted as follows:
{
// (Optional) Serving signature to use.
// If unspecifed default serving signature is used.
"signature_name": <string>,
// Input Tensors in row ("instances") or columnar ("inputs") format.
// A request can have either of them but NOT both.
"instances": <value>|<(nested)list>|<list-of-objects>
"inputs": <value>|<(nested)list>|<object>
}
Encoding binary values JSON uses UTF-8 encoding. If you have input
feature or tensor values that need to be binary (like image bytes),
you must Base64 encode the data and encapsulate it in a JSON object
having b64 as the key as follows:
{ "b64": "base64 encoded string" }
You can specify this object as a value for an input feature or tensor.
The same format is used to encode output response as well.
A classification request with image (binary data) and caption features
is shown below:
{ "signature_name": "classify_objects", "examples": [
{
"image": { "b64": "aW1hZ2UgYnl0ZXM=" },
"caption": "seaside"
},
{
"image": { "b64": "YXdlc29tZSBpbWFnZSBieXRlcw==" },
"caption": "mountains"
} ] }
Uncertainties include:
should I use "instances" in my JSON
should I base64 encode a JPG or PNG or something else?
Should the image be of a particular
width and height?
In Serving Image-Based Deep Learning Models with TensorFlow-Serving’s RESTful API this format is suggested:
{
"instances": [
{"b64": "iVBORw"},
{"b64": "pT4rmN"},
{"b64": "w0KGg2"}
]
}
I used this image:
https://tensorflow.org/images/blogs/serving/cat.jpg
and base64 encoded it like so:
# Download the image
dl_request = requests.get(IMAGE_URL, stream=True)
dl_request.raise_for_status()
# Compose a JSON Predict request (send JPEG image in base64).
jpeg_bytes = base64.b64encode(dl_request.content).decode('utf-8')
predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
But when I use curl to POST the base64 encoded image like so:
{"instances" : [{"b64": "/9j/4AAQSkZJRgABAQAASABIAAD/4QBYRXhpZgAATU0AKgAA
...
KACiiigAooooAKKKKACiiigAooooA//Z"}]}
I get a response like this:
>./test_local_tfs.sh
HEADER=|Content-Type:application/json;charset=UTF-8|
URL=|http://127.0.0.1:8501/v1/models/saved_model/versions/1:predict|
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8501 (#0)
> POST /v1/models/saved_model/versions/1:predict HTTP/1.1
> Host: 127.0.0.1:8501
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type:application/json;charset=UTF-8
> Content-Length: 85033
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Date: Tue, 17 Sep 2019 10:47:18 GMT
< Content-Length: 85175
<
{ "error": "Failed to process element: 0 of \'instances\' list. Error: Invalid argument: JSON Value: {\n \"b64\": \"/9j/4AAQSkZJRgABAQAAS
...
ooooA//Z\"\n} Type: Object is not of expected type: uint8" }
I've tried converting a local version of the same file to base64 like so (confirming that the dtype is uint8) ...
img = cv2.imread('cat.jpg')
print('dtype: ' + str(img.dtype))
_, buf = cv2.imencode('.jpg', img)
jpeg_bytes = base64.b64encode(buf).decode('utf-8')
predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
But posting this JSON generates the same error.
However, when the json is formated like so ...
{'instances': [[[[112, 71, 48], [104, 63, 40], [107, 70, 20], [108, 72, 21], [109, 77, 0], [106, 75, 0], [92, 66, 0], [106, 80, 0], [101, 80, 0], [98, 77, 0], [100, 75, 0], [104, 80, 0], [114, 88, 17], [94, 68, 0], [85, 54, 0], [103, 72, 11], [93, 62, 0], [120, 89, 25], [131, 101, 37], [125, 95, 31], [119, 91, 27], [121, 93, 29], [133, 105, 40], [119, 91, 27], [119, 96, 56], [120, 97, 57], [119, 96, 53], [102, 78, 36], [132, 103, 44], [117, 88, 28], [125, 89, 4], [128, 93, 8], [133, 94, 0], [126, 87, 0], [110, 74, 0], [123, 87, 2], [120, 92, 30], [124, 95, 33], [114, 90, 32],
...
, [43, 24, 33], [30, 17, 36], [24, 11, 30], [29, 20, 38], [37, 28, 46]]]]}
... it works.
The problem is this json file is >11 MB in size.
How do I make the base64 encoded version of the json work?
UPDATE: It seems that we have to edit the pretrained model to accept base64 images at the input layer
This article describes how to edit the model ...
Medium: Serving Image-Based Deep Learning Models with TensorFlow-Serving’s RESTful API
... unfortunately, it assumes that we have access to the code which generated the model.
user260826's solution provides a work-around using an estimator but it assumes the model is a keras model. Not true in this case.
Is there a generic method to make a model ready for TensorFlow Serving REST interface with a base64 encoded image that works with any of the TensorFlow model formats?

Answer 1

The first step is to export the trained model in the appropriate format. Use export_inference_graph.py like this
python export_inference_graph \
--input_type encoded_image_string_tensor \
--pipeline_config_path path/to/ssd_inception_v2.config \
--trained_checkpoint_prefix path/to/model.ckpt \
--output_directory path/to/exported_model_directory
in the above code snippet, it is important to specify
--input_type encoded_image_string_tensor
after exporting the model, run the tensorflow server as usual with the newly exported model.
The inference code will look like this:
from __future__ import print_function
import base64
import requests
SERVER_URL = 'http://localhost:8501/v1/models/vedNet:predict'
IMAGE_URL = 'test_images/19_inp.jpg'
def main():
with open(IMAGE_URL, "rb") as image_file:
jpeg_bytes = base64.b64encode(image_file.read()).decode('utf-8')
predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
response = requests.post(SERVER_URL, predict_request)
response.raise_for_status()
prediction = response.json()['predictions'][0]
if __name__ == '__main__':
main()

Answer 2

As you mentioned JSON is a very inefficient approach, as payload normally exceeds original filesize, you need to convert the model to be able to process the image bytes written to a string using Base64 encoding:
{"b64": base64_encoded_string}
This new conversion will reduce the prediction time and bandwidth utilization used to transfer image from prediction client to your infrastructure.
I recently used a Transfer Learning model with TF Hub and Keras which was using a JSON as input, as you mentioned this is not optimal for prediction.
I used the following function to overwrite it:
Using the following code we add a new serving function which will be able to process Base64 encoded images.
Using TF estimator model:
h5_model_path = os.path.join('models/h5/best_model.h5')
tf_model_path = os.path.join('models/tf')
estimator = keras.estimator.model_to_estimator(
keras_model_path=h5_model_path,
model_dir=tf_model_path)
def image_preprocessing(image):
"""
This implements the standard preprocessing that needs to be applied to the
image tensors before passing them to the model. This is used for all input
types.
"""
image = tf.expand_dims(image, 0)
image = tf.image.resize_bilinear(image, [HEIGHT, WIDTH], align_corners=False)
image = tf.squeeze(image, axis=[0])
image = tf.cast(image, dtype=tf.uint8)
return image
def serving_input_receiver_fn():
def prepare_image(image_str_tensor):
image = tf.image.decode_jpeg(image_str_tensor, channels=CHANNELS)
return image_preprocessing(image)
input_ph = tf.placeholder(tf.string, shape=[None])
images_tensor = tf.map_fn(
prepare_image, input_ph, back_prop=False, dtype=tf.uint8)
images_tensor = tf.image.convert_image_dtype(images_tensor, dtype=tf.float32)
return tf.estimator.export.ServingInputReceiver(
{'input': images_tensor},
{'image_bytes': input_ph})
export_path = os.path.join('/tmp/models/json_b64', version)
if os.path.exists(export_path): # clean up old exports with this version
shutil.rmtree(export_path)
estimator.export_savedmodel(
export_path,
serving_input_receiver_fn=serving_input_receiver_fn)
A good example here

Answer 3

I have been struggling with the same problem. Finally I could make it work. I just had to add a new signature to the model:
import tensorflow as tf
model = tf.saved_model.load("/path/to/the/original/model")
# This is the current signature, that only accepts image tensors as input
signature = model.signatures["default"]
#tf.function()
def my_predict(image_b64):
# Model doesn't support batch!!
img_dec = tf.image.decode_png(image_b64[0], channels=3)
img_tensor = tf.image.convert_image_dtype(img_dec, tf.float32)[tf.newaxis, ...]
prediction = signature(img_tensor)
return prediction
# Create new signature, to read b64 images
new_signature = my_predict.get_concrete_function(
image_b64=tf.TensorSpec([None], dtype=tf.string, name="image_b64")
)
tf.saved_model.save(
model,
export_dir="/path/to/the/saved/model",
signatures=new_signature
)
Finally, after serving I can make predictions passing an input like this:
{
"instances": [
{
"b64": "youBase64ImageHere"
}
]
}