Tensorflow Serving Problem For A.K Nain Keras Model - tensorflow

Question

I'm new to ML and try to find some interesting prediction based on keras. I follow AK Nain's post of solving captcha image using tensorflow + keras, it all goes okay. When I use flask, a simple api built which can successfully predict output. Problem is that, I want to scale this api's performance. Currently it took 40 milis for every prediction, so I can achieve 20-25 prediction per second using this api. I need to scale it using more cpu cores and memory and get its optimal performance. So after searching about 7 days, I came near to run it on Tensorflow Serving Docker Image.
Problem arise here, I cannot get the predict feature of tensorflow serving, cause it expect a different input but I failed to analyze it's actual input. Here I attach working prediction from flask api,
img = flask.request.files["image"].read()
# 2. Decode and convert to grayscale
img = tf.io.decode_png(img, channels=1)
# 3. Convert to float32 in [0, 1] range
img = tf.image.convert_image_dtype(img, tf.float32)
# 4. Resize to the desired size
img = tf.image.resize(img, [img_height, img_width])
# 5. Transpose the image because we want the time
# dimension to correspond to the width of the image.
img = tf.transpose(img, perm=[1, 0, 2])
img = tf.expand_dims(img, axis=0)
preds = prediction_model.predict(img)
pred_text = decode_batch_predictions(preds)
# indicate that the request was a success
data["success"] = True
data["text"]=pred_text[0]
This flask api server me actual output. I export the same model to docker image, and change this api a little bit to work,
img = flask.request.files["image"].read()
# 2. Decode and convert to grayscale
img = tf.io.decode_png(img, channels=1)
# 3. Convert to float32 in [0, 1] range
img = tf.image.convert_image_dtype(img, tf.float32)
# 4. Resize to the desired size
img = tf.image.resize(img, [img_height, img_width])
# 5. Transpose the image because we want the time
# dimension to correspond to the width of the image.
img = tf.transpose(img, perm=[1, 0, 2])
img = tf.expand_dims(img, axis=0)
payload = json.dumps({"signature_name": "serving_default", "instances":
[{"image":img.numpy().tolist()}]})
r = requests.post(KERAS_REST_API_URL, data=payload,headers=headers).json()
It fails with the following error,
{'error': 'Failed to process element: 0 of \'instances\' list. JSON object: {\n "image": [\n [\n [\n [\n 1.0\n ],\n [\n 1.0\n ],\n [\n 1.0\n ],\n [\n 1.0\n ],\n [\n 1.0\n ],\n [\n 0.03921568766236305\n ],\n [\n 1.0\n ],\n [\n 1.0\n ],\n
[\n 1.0\n ],\n [\n 1.0\n ],\n [\n 1.0\n
],\n [\n 1.0\n ],\n [\n 1.0\n ],\n [\n
1.0\n ],\n [\n 1.0\n ],\n [\n 0.03921568766236305\n ],\n [\n 1.0\n ],\n [\n 1.0\n ],\n
My model metadata is like this,
{"model_spec":{"name":"my_model","signature_name":"","version":"1"},"metadata":
{"signature_def":{"signature_def":{"serving_default":{"inputs":{"label":
{"dtype":"DT_FLOAT","tensor_shape":{"dim":[{"size":"-1","name":""},
{"size":"-1","name":""}],"unknown_rank":false},"name":"serving_default_label:0"},"image":
{"dtype":"DT_FLOAT","tensor_shape":{"dim":[{"size":"-1","name":""},{"size":"100","name":""},
{"size":"36","name":""},
{"size":"1","name":""}],"unknown_rank":false},"name":"serving_default_image:0"}},"outputs":
{"ctc_loss":{"dtype":"DT_FLOAT","tensor_shape":{"dim":[{"size":"-1","name":""},
{"size":"25","name":""},
{"size":"12","name":""}],"unknown_rank":false},"name":"StatefulPartitionedCall:0"}},
"method_name":"tensorflow/serving/predict"},"__saved_model_init_op":{"inputs":{},"outputs":
{"__saved_model_init_op":{"dtype":"DT_INVALID","tensor_shape":{"dim":
[],"unknown_rank":true},"name":"NoOp"}},"method_name":""}}}}}
Can you please tell me where I'm wrong?

Answer 1

So with tensorflow serving, you have to serve up the saved model itself. This is why integrating the pipeline into the model is so critically important. Here are two places where you can learn about the tensorflow serving architecture and how you can actually serve models.
Why is the architecture this way? From my understanding, this is architected to execute everything in the graph itself, so you don't have the latency of python getting in the way.