I am using tensorflow_model_server to serve a SavedModel. I keep getting this response code 400 and following error:
{ "error": "The first dimension of paddings must be the rank of inputs[4,2] [1,1,1,208,770,3]\\n\\t [[{{node Generator/FlatConv/sequential/zero_padding2d/Pad}}]]" }
Output from saved-model-cli show ...
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['__saved_model_init_op']:
The given SavedModel SignatureDef contains the following input(s):
The given SavedModel SignatureDef contains the following output(s):
outputs['__saved_model_init_op'] tensor_info:
dtype: DT_INVALID
shape: unknown_rank
name: NoOp
Method name is:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, -1, 3)
name: serving_default_input_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['output_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, -1, 3)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
WARNING:tensorflow:From /tensorflow-1.15.0/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Defined Functions:
Function Name: '_default_save_signature'
Option #1
Callable with:
Argument #1
input_1: TensorSpec(shape=(?, ?, ?, 3), dtype=tf.float32, name='input_1')
Pre-processing
img_path = "/content/input_images/my_img.jpg"
img = np.array(Image.open(img_path).convert("RGB"))
img = np.expand_dims(img, 0).astype(np.float32) / 127.5 - 1
Request code:
payload = {
"instances": [{'input_1': [input_image.tolist()]}]
}
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8501/v1/models/my_model:predict', data=json.dumps(payload), headers=headers)
print("Request complete")
print (json_response)
response_text = json_response.text
response_text
Response / Output
Request complete
<Response [400]>
'{ "error": "The first dimension of paddings must be the rank of inputs[4,2] [1,1,1,449,674,3]\\n\\t [[{{node Generator/FlatConv/sequential/zero_padding2d/Pad}}]]" }'
Code is run on Colab
I do not understand what is wrong here.
It means that your input data shoud be four dimensions array, while you have 6d
try this:
"instances": [{'input_1': np.squeeze(input_image).tolist()}]
Remove the outer [] and squeezing it seems to reduce 6d to 4d and this will likely works.
I ran into a similar error when I have a multi-modal inputs where 1st item is an image and 2nd is a string. I tried removing the batch dimension in the image input and it works. In my opinion, this seems to be a bug in tensorflow-serving. It should be able to handle a hash of batch of image.
(alternative, you don't need the squeeze if you didn't np.expand_dims earlier.
Related
I made a tensorflow Model and i use tensorflow serving to deploy this model, but when i can build the restful request params the model need
curl -d '{"instances": [[1], [1], [1], [1], [1], [1], [1], [1], [1], [1]]}'
-X POST http://localhost:8501/v1/models/shipping_predict:predict
it call back
"error": "instances is a plain list, but expecting list of objects as multiple input tensors required as per tensorinfo_map"
this is my model
# prepare each input head
in_layers = list()
em_layers = list()
#
# customer
in_layer_customer = Input(shape=(1,))
em_layer_customer = Embedding(5000, 10)(in_layer_customer)
em_layer_customer = layers.Reshape([1 * 10])(em_layer_customer)
in_layers.append(in_layer_customer)
em_layers.append(em_layer_customer)
# salesman
in_layer_sale = Input(shape=(1,))
em_layer_sale = Embedding(500, 10)(in_layer_sale)
em_layer_sale = layers.Reshape([1 * 10])(em_layer_sale)
in_layers.append(in_layer_sale)
em_layers.append(em_layer_sale)
# business_type
in_layer_businessType = Input(shape=(1,))
em_layer_businessType = Embedding(100, 10)(in_layer_businessType)
em_layer_businessType = layers.Reshape([1 * 10])(em_layer_businessType)
in_layers.append(in_layer_businessType)
em_layers.append(em_layer_businessType)
# 20
in_layer_20 = Input(shape=(1,))
em_layer_20 = layers.Dense(16, activation='relu')(in_layer_20)
in_layers.append(in_layer_20)
em_layers.append(em_layer_20)
# 40
in_layer_40 = Input(shape=(1,))
em_layer_40 = layers.Dense(16, activation='relu')(in_layer_40)
in_layers.append(in_layer_40)
em_layers.append(em_layer_40)
# 45
in_layer_45 = Input(shape=(1,))
em_layer_45 = layers.Dense(16, activation='relu')(in_layer_45)
in_layers.append(in_layer_45)
em_layers.append(em_layer_45)
# other
in_layer_other = Input(shape=(1,))
em_layer_other = layers.Dense(16, activation='relu')(in_layer_other)
in_layers.append(in_layer_other)
em_layers.append(em_layer_other)
# dischargingPort
in_layer_dischargingPortName = Input(shape=(1,))
em_layer_dischargingPortName = Embedding(5000, 10)(in_layer_dischargingPortName)
em_layer_dischargingPortName = layers.Reshape([1 * 10])(em_layer_dischargingPortName)
in_layers.append(in_layer_dischargingPortName)
em_layers.append(em_layer_dischargingPortName)
# MBL Method
in_layer_mbl = Input(shape=(1,))
em_layer_mbl = layers.Dense(16, activation='relu')(in_layer_mbl)
in_layers.append(in_layer_mbl)
em_layers.append(em_layer_mbl)
# atdMouth
in_layer_atdMouth = Input(shape=(1,))
em_layer_atdMouth = Embedding(100, 10)(in_layer_atdMouth)
em_layer_atdMouth = layers.Reshape([1 * 10])(em_layer_atdMouth)
in_layers.append(in_layer_atdMouth)
em_layers.append(em_layer_atdMouth)
merge = Concatenate()(em_layers)
dense = Dense(32, activation='relu')(merge)
output = Dense(1)(dense)
model = Model(inputs=in_layers, outputs=output)
instances is a plain list, but expecting list of objects as multiple
input tensors
It sounds like your model, for whatever reason, is expecting named tensors. This is not something I've worked with before but there appears to be another way of sending requests to your your model.
curl -X POST -i 'http://192.168.1.16:8501/v1/models/export:predict' --data '
{
"signature_name": "serving_default",
"inputs": [
{
"tokens_0" :["text text text text text text text text text text"],
"length_0": [1],
"tokens_1": ["01 01 01 01 01 01 01 01 01 01"],
"length_1": [1],
"tokens_2": ["4 4 4 1 1 4 4 4 4 4"],
"length_2": [1]
}
]
}'
I've just copied this example from here (credit to #ishaan-sharma).
Your model is non-trivial so I won't try and create the exact request for you. If you're unsure about the tensor names etc, you can check out the expected shape using the saved_model_cli:
saved_model_cli show [-h] --dir DIR [--all]
A bit late for this answer, but I hope it helps to someone. I have saved a Keras model where it excepts multiple inputs. The model expects two inputs: text and a vector of integers
import tensorflow as tf
from tensorflow.keras.models import Model
model = Model(inputs=[text_input, input2], outputs=out)
### some code for training and data preparation
export_path = "serving/5/"
tf.saved_model.save(model, export_path)
This finally saved the model to be served using Tensorflow serving. A brief summary of the model is as follows:
$ saved_model_cli show --dir ./serving/5 --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['input_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 23)
name: serving_default_input_1:0
inputs['text'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: serving_default_text:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_2'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 1)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
Finally to post my request to the serving container, I did:
import requests
import json
params = json.dumps(
{
"instances": [
{
"text": [features[0]],
"input_1": features[1:]
}
]
}
)
response = requests.post(
"http://localhost:8038/v1/models/serving:predict",
headers={"Content-Type": "application/json"},
data=params
)
The main issue was that I had to wrap all my inputs as a JSON in instances with their tensor names (which I got from saved_model_cli).
Important thing to note is that the values needs to be given as a list. features[0] is a text so I wrapped it up in a list while features[1:] itself is a vector which does fine.
My motivation is to build a custom objection detection web application. I downloaded a tf2 pretrained SSD Resnet1010 model from model zoo. My idea is if this implementation works, I will train the model with my own data. I ran $saved_model_cli show --dir saved_model --tag_set serve --signature_def serving_default to figure out input and output nodes.
The given SavedModel SignatureDef contains the following input(s):
inputs['input_tensor'] tensor_info:
dtype: DT_UINT8
shape: (1, -1, -1, 3)
name: serving_default_input_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_anchor_indices'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:0
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: StatefulPartitionedCall:1
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:2
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 91)
name: StatefulPartitionedCall:3
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:4
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (1)
name: StatefulPartitionedCall:5
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 51150, 4)
name: StatefulPartitionedCall:6
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 51150, 91)
name: StatefulPartitionedCall:7
Method name is: tensorflow/serving/predict
Then I converted the model to tensorflowjs model, by running
tensorflowjs_converter --input_format=tf_saved_model --output_node_names='detection_anchor_indices,detection_boxes,detection_classes,detection_multiclass_scores,detection_scores,num_detections,raw_detection_boxes,raw_detection_scores' --saved_model_tags=serve --output_format=tfjs_graph_model saved_model js_model
Here is my javascript code (this goes inside vue methods)
loadTfModel: async function(){
try {
this.model = await tf.loadGraphModel(this.MODEL_URL);
} catch(error) {
console.log(error);
}
},
predictImg: async function() {
const imgData = document.getElementById('img');
let tf_img = tf.browser.fromPixels(imgData);
tf_img = tf_img.expandDims(0);
const predictions = await this.model.executeAsync(tf_img);
const data = []
for (let i = 0; i < predictions.length; i++){
data.push(predictions[i].dataSync());
}
console.log(data);
}
The output looks like this:
My question is does these eight items in the array corresponds to eight defined output nodes? How to make sense of this data? and how to convert this into a human-readable format like the python one?
Update 1:
I have tried this answer and edited my predict method:
predictImg: async function() {
const imgData = document.getElementById('img');
let tf_img = tf.browser.fromPixels(imgData);
tf_img = tf_img.expandDims(0);
const predictions = await this.model.executeAsync(tf_img, ['detection_classes']).then(predictions => {
const data = predictions.dataSync()
console.log('Predictions: ', data);
})
}
I ended up getting, "Error: The output 'detection_classes' is not found in the graph". I would appreciate any help.
There might probably a mistake in the output node specified in this.model.executeAsync(tf_img, ['detection_classes']). Additionnally, there is no need to use await here await this.model.executeAsync(tf_img, ['detection_classes']). Either await is used or then is used.
The other option to get the detection_classes is to index the array of output:
predictions[i].dataSync()[2]
I think you first need to check the web_model/model.json file and investigate the name of the outputs. Those are the ones you will need to use when filtering what to display (the following is my example file).
I am learning to use Tensorflow serving but I have a very hard time finding examples including in Stackoverflow.
I am using flower example in TensorFlow website. The training part is successful and having a hard time deploying.
I ran the server as follows :
docker run -p 8501:8501 --mount type=bind,source=C:\tmp\saved_models,target=/models/flowers -e MODEL_NAME=flowers -t tensorflow/serving &
To look at all SignatureDef, I used the following
saved_model_cli show --dir c:\tmp\saved_models\1 --all
The response is
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['image'] tensor_info:
dtype: DT_STRING
shape: ()
name: DecodeJpeg/contents:0
The given SavedModel SignatureDef contains the following output(s):
outputs['classes'] tensor_info:
dtype: DT_STRING
shape: (5)
name: Const:0
outputs['prediction'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 5)
name: final_result:0
Method name is: tensorflow/serving/predict
As I have seen here it suppose to have "Method name is: tensorflow/serving/classify"
Why is it missing in my example ?
My RESTAPI request is as follows :
http://localhost:8501/v1/models/flowers/versions/1:classify
Body :
{
"signature_name": "serving_default",
"flowers": [
{
"image": { "b64": "c:/Users/pubud/OneDrive/Pictures/bird.jpg=" },
"rose": "flower"
}
]
}
The error I get is :
{
"error": "Expected classification signature method_name to be tensorflow/serving/classify. Was: tensorflow/serving/predict"
}
Any help is really appreciated.
Your RESTful URL specifies ...:classify, which is different from what's in the signature_def['serving_default'] - predict.
Try changing the RESTful URL to be ...:predict, and then refer to the instructions here for the json body of the request.
Here is more information about the SignatureDefs:
https://www.tensorflow.org/serving/signature_defs
Good luck!
I have an estimator using a TensorHub text_embedding column, like so:
my_dataframe = pandas.DataFrame(columns=["title"})
# populate data
labels = []
# populate labels with 0|1
embedded_text_feature_column = hub.text_embedding_column(
key="title"
,module_spec="https://tfhub.dev/google/nnlm-en-dim128-with-normalization/1")
estimator = tf.estimator.LinearClassifier(
feature_columns = [ embedded_text_feature_column ]
,optimizer=tf.train.FtrlOptimizer(
learning_rate=0.1
,l1_regularization_strength=1.0
)
,model_dir=model_dir
)
estimator.train(
input_fn=tf.estimator.inputs.pandas_input_fn(
x=my_dataframe
,y=labels
,batch_size=128
,num_epochs=None
,shuffle=True
,num_threads=5
)
,steps=5000
)
export(estimator, "/tmp/my_model")
How can I export and serve the model so that it accepts strings as input to predictions? I have a serving_input_receiver_fn as follows, and tried quite a few more, but I'm quite confused as to what it needs to look like so that I can serve it (with saved_model_cli, say) and call it with title strings (or a simple JSON structure) as input.
def export(estimator, dir_path):
def serving_input_receiver_fn():
feature_spec = tf.feature_column.make_parse_example_spec(hub.text_embedding_column(
key="title"
,module_spec="https://tfhub.dev/google/nnlm-en-dim128-with-normalization/1"))
return tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)
estimator.export_savedmodel(
export_dir_base=dir_path
,serving_input_receiver_fn=serving_input_receiver_fn()
)
If you want to feed raw strings, you might want to consider using the raw input receiver. This code:
feature_placeholder = {'title': tf.placeholder('string', [1], name='title_placeholder')}
serving_input_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(feature_placeholder)
estimator.export_savedmodel(dir_path, serving_input_fn)
will give you a SavedModel with the following input specification according to the SavedModel CLI:
saved_model_cli show --dir ./ --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: title_placeholder_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['classes'] tensor_info:
dtype: DT_STRING
shape: (-1, 2)
name: linear/head/Tile:0
outputs['scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 2)
name: linear/head/predictions/probabilities:0
You can provide a python expression to the CLI to serve an input to the model to validate that it works:
saved_model_cli run --dir ./ --tag_set serve --signature_def \
serving_default --input_exprs "inputs=['this is a test sentence']"
Result for output key classes:
[[b'0' b'1']]
Result for output key scores:
[[0.5123377 0.4876624]]
The complete code for exporting the model: (I've already trained it and now loading from weights file)
def cnn_layers(inputs):
conv_base= keras.applications.mobilenetv2.MobileNetV2(input_shape=(224,224,3), input_tensor=inputs, include_top=False, weights='imagenet')
for layer in conv_base.layers[:-200]:
layer.trainable = False
last_layer = conv_base.output
x = GlobalAveragePooling2D()(last_layer)
x= keras.layers.GaussianNoise(0.3)(x)
x = Dense(1024,name='fc-1')(x)
x = keras.layers.BatchNormalization()(x)
x = keras.layers.advanced_activations.LeakyReLU(0.3)(x)
x = Dropout(0.4)(x)
x = Dense(512,name='fc-2')(x)
x = keras.layers.BatchNormalization()(x)
x = keras.layers.advanced_activations.LeakyReLU(0.3)(x)
x = Dropout(0.3)(x)
out = Dense(10, activation='softmax',name='output_layer')(x)
return out
model_input = layers.Input(shape=(224,224,3))
model_output = cnn_layers(model_input)
test_model = keras.models.Model(inputs=model_input, outputs=model_output)
weight_path = os.path.join(tempfile.gettempdir(), 'saved_wt.h5')
test_model.load_weights(weight_path)
export_path='export'
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import utils
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import build_signature_def, predict_signature_def
from tensorflow.contrib.session_bundle import exporter
builder = saved_model_builder.SavedModelBuilder(export_path)
signature = predict_signature_def(inputs={'image': test_model.input},
outputs={'prediction': test_model.output})
with K.get_session() as sess:
builder.add_meta_graph_and_variables(sess=sess,
tags=[tag_constants.SERVING],
signature_def_map={'predict': signature})
builder.save()
And the output of (dir 1 has saved_model.pb and models dir) :
python /tensorflow/python/tools/saved_model_cli.py show --dir /1 --all is
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
signature_def['predict']:
The given SavedModel SignatureDef contains the following input(s):
inputs['image'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 224, 224, 3)
name: input_1:0
The given SavedModel SignatureDef contains the following output(s):
outputs['prediction'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 107)
name: output_layer/Softmax:0
Method name is: tensorflow/serving/predict
To accept b64 string:
The code was written for (224, 224, 3) numpy array. So, the modifications I made for the above code are:
_bytes should be added to input when passing as b64. So,
predict_signature_def(inputs={'image':......
changed to
predict_signature_def(inputs={'image_bytes':.....
Earlier, type(test_model.input) is : (224, 224, 3) and dtype: DT_FLOAT. So,
signature = predict_signature_def(inputs={'image': test_model.input},.....
changed to (reference)
temp = tf.placeholder(shape=[None], dtype=tf.string)
signature = predict_signature_def(inputs={'image_bytes': temp},.....
Edit:
Code to send using requests is : (As mentioned in the comments)
encoded_image = None
with open('/1.jpg', "rb") as image_file:
encoded_image = base64.b64encode(image_file.read())
object_for_api = {"signature_name": "predict",
"instances": [
{
"image_bytes":{"b64":encoded_image}
#"b64":encoded_image (or this way since "image" is not needed)
}]
}
p=requests.post(url='http://localhost:8501/v1/models/mnist:predict', json=json.dumps(object_for_api),headers=headers)
print(p)
I'm getting <Response [400]> error. I think there's no error in the way I'm sending. Something needs to be changed in the code for exporting the model and specifically in
temp = tf.placeholder(shape=[None], dtype=tf.string).
Looking at the docs you've provided what you're looking to do is to take the image and send it in to the API. Images are easily transferable in a text format if you encode them, base64 being pretty much the standard. So what we want to do is create a json object with the image as base64 in the right place and then send this json object into the REST api. python has the requests library which makes sending in a python dictionary as JSON very easy.
So take the image, encode it, put it in a dictionary and send it off using requests:
import requests
import base64
encoded_image = None
with open("image.png", "rb") as image_file:
encoded_image = base64.b64encode(image_file.read())
object_for_api = {"signature_name": "predict",
"instances": [
{
"image": {"b64": encoded_image}
}]
}
requests.post(url='http://localhost:8501/v1/models/mnist:predict', json=object_for_api)
You can also encode your numpy array into JSON but it doesn't seem that the API docs are looking for that.
Two side notes:
I encourage you to use tf.saved_model.simple_save
You may find model_to_estimator convenient.
While your model seems like it will work for requests (the output of saved_model_cli shows the outer dimension is None for both inputs and outputs), it's fairly inefficient to send JSON arrays of floats
To the last point, it's often easier to modify the code to do the image decoding server side so you're sending a base64 encoded JPG or PNG over the wire instead of an array of floats. Here's one example for Keras (I plan to update that answer with simpler code).