Is Tensorflow Serving example expected to have 10% error rate? - tensorflow-serving

Question

I followed the docs Building an optimized serving binary then Testing the development environment, and I got
Inference error rate: 10.4%
Is it expected that a fresh install of a release build of TensorFlow Serving would give a 10% error rate on the provided example model?
My environment:
AWS EC2
OS: Amazon Linux AMI release 2018.03
Instance Type: r5.large
Steps to reproduce:
# download tensorflow serving code
git clone https://github.com/tensorflow/serving
cd serving
# build optimized serving binary
docker build --pull -t $USER/tensorflow-serving-devel -f tensorflow_serving/tools/docker/Dockerfile.devel .
# run & open shell for generated docker image
docker run -it -p 8600:8600 ec2-user/tensorflow-serving-devel:latest
# train the mnist model
python tensorflow_serving/example/mnist_saved_model.py /tmp/mnist_model
# serve the model
tensorflow_model_server --port=8500 --model_name=mnist --model_base_path=/tmp/mnist_model/ &
# test the client
python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=localhost:8500

Answer 1

The example mnist_saved_model.py as part of tensorflow_serving's example is more focused on speed to create a model and a simple example on how to save a model then accuracy.
In https://www.tensorflow.org/tfx/serving/serving_advanced, it shows that when the above code is trained with 100 iterations it has a 13.1% error rate, and when trained with 2000 iterations it has a 9.5% error rate.
The default if --training_iteration is not specified is 1000, so your 10.4 error rate is inline with these results.
You will find this mnist model provides better accuracy (and takes much longer to train): https://github.com/tensorflow/models/tree/master/official/mnist
This model will work with the slight changes to the mnist_client.py example.
try this:
train the mnist model
git clone https://github.com/tensorflow/models
export PYTHONPATH="$PYTHONPATH:$PWD/models"
pushd models/official/mnist/
python mnist.py --export_dir /tmp/mnist_model
serve the model
tensorflow_model_server --port=8500 --model_name=mnist --model_base_path=/tmp/mnist_model/ &
switch back the original directory
popd
make the follow changes to mnist_client.py to work with the new model
diff --git a/tensorflow_serving/example/mnist_client.py b/tensorflow_serving/example/mnist_client.py
index 592969d..85ef0bf 100644
--- a/tensorflow_serving/example/mnist_client.py
+++ b/tensorflow_serving/example/mnist_client.py
## -112,8 +112,8 ## def _create_rpc_callback(label, result_counter):
sys.stdout.write('.')
sys.stdout.flush()
response = numpy.array(
- result_future.result().outputs['scores'].float_val)
- prediction = numpy.argmax(response)
+ result_future.result().outputs['classes'].int64_val)
+ prediction = response[0]
if label != prediction:
result_counter.inc_error()
result_counter.inc_done()
## -143,9 +143,9 ## def do_inference(hostport, work_dir, concurrency, num_tests):
for _ in range(num_tests):
request = predict_pb2.PredictRequest()
request.model_spec.name = 'mnist'
- request.model_spec.signature_name = 'predict_images'
+ request.model_spec.signature_name = 'classify'
image, label = test_data_set.next_batch(1)
- request.inputs['images'].CopyFrom(
+ request.inputs['image'].CopyFrom(
tf.contrib.util.make_tensor_proto(image[0], shape=[1, image[0].size]))
result_counter.throttle()
result_future = stub.Predict.future(request, 5.0) # 5 seconds
test the client
python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=localhost:8500
Inference error rate: 0.8%

Answer 2

Is Tensorflow Serving an example expected to have a 10% error rate?
Yes, this particular example is expected to have a 10% error rate as the accuracy of this model on training and testing data is almost the same (around 90%) i.e., this is a very basic neural network as shown here.
If you want a good prediction accuracy, you might have to use resnet_client.py or you can actually add more layers and tune the hyper-parameters to get higher prediction accuracy or lesser inference error rate.
A tutorial on how to use the resent model to serve is given here. This should give you a much lesser inference error rate.