Tensorflow serving object detection predict using Kubeflow - tensorflow

Question

I followed the steps given in this post to deploy my tensorflow model for prediction using GPUs on Google Kubernetes Engine and Kubeflow. I have exposed the service as a load balancer by modifying the YAML file in this way where I changed the type from ClusterIP to LoadBalancer.
spec:
clusterIP: A.B.C.D
externalTrafficPolicy: Cluster
ports:
- name: grpc-tf-serving
nodePort: 30098
port: 9000
protocol: TCP
targetPort: 9000
- name: http-tf-serving-proxy
nodePort: 31399
port: 8000
protocol: TCP
targetPort: 8000
selector:
app: my-model
sessionAffinity: None
type: LoadBalancer
The status changed to:
status:
loadBalancer:
ingress:
- ip: W.X.Y.Z
Service specs (kubectl describe services my-model):
Name: my-model
Namespace: default
Labels: app=my-model
app.kubernetes.io/deploy-manager=ksonnet
ksonnet.io/component=model2
Annotations: getambassador.io/config:
---
apiVersion: ambassador/v0
kind: Mapping
name: tfserving-mapping-my-model-get
prefix: /models/my-model/
rewrite: /
method: GET
service: my-model.default:8000
---
apiVersion: ambassador/v0
kind: Mapping
name: tfserving-mapping-my-model-post
prefix: /models/my-model/
rewrite: /model/my-model:predict
method: POST
service: my-model.default:8000
ksonnet.io/managed:
{"pristine":"H4sIAAAAAAAA/7SRMY/UQAyFe35F5DpzCVweRcHW4QQBWKlQzQMhS/jZEckHmvGt9xplf+OZvfYjXRCgoIyz+/L8xsfgTR+5VxiEkA4vIYWfkQJgHDH+RAHhhYWNgpkB...
Selector: app=my-model
Type: LoadBalancer
IP: A.B.C.D
LoadBalancer Ingress: W.X.Y.Z
Port: grpc-tf-serving 9000/TCP
TargetPort: 9000/TCP
NodePort: grpc-tf-serving 30098/TCP
Endpoints: P.Q.R.S:9000
Port: http-tf-serving-proxy 8000/TCP
TargetPort: 8000/TCP
NodePort: http-tf-serving-proxy 31399/TCP
Endpoints: R.Q.R.S:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Pods Specs (kubectl describe pods):
Name: my-model-v1-bd6ccb757-qrwdv
Namespace: default
Node: gke-kuberflow-xyz-gpu-pool-5d4ebf17-56mf/SOME_IP
Start Time: Mon, 18 Feb 2019 18:11:24 +0530
Labels: app=my-model
pod-template-hash=682776313
version=v1
Annotations: <none>
Status: Running
IP: P.Q.R.S
Controlled By: ReplicaSet/my-model-v1-bd6ccb757
Containers:
my-model:
Container ID: docker://d14e8261ddfe606393da2ee45badac0136cee98rwa5611c47ad85733ce5d2c925
Image: tensorflow/serving:1.11.1-gpu
Image ID: docker-pullable://tensorflow/serving#sha256:907d7db828b28ewer234d0b3ca10e2d66bcd8ef82c5cccea761fcd4f1190191d2f
Port: 9000/TCP
Host Port: 0/TCP
Command:
/usr/bin/tensorflow_model_server
Args:
--port=9000
--model_name=my-model
--model_base_path=gs://xyz_kuber_app-xyz-identification/export/
State: Running
Started: Mon, 18 Feb 2019 18:11:25 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 4
memory: 4Gi
nvidia.com/gpu: 1
Requests:
cpu: 1
memory: 1Gi
nvidia.com/gpu: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
my-model-http-proxy:
Container ID: docker://c98e06ad75f3456c353395e9ad2e2e3bcbf0b38cd2634074704439cd5ebf335d
Image: gcr.io/kubeflow-images-public/tf-model-server-http-proxy:v20180606-asdasda
Image ID: docker-pullable://gcr.io/kubeflow-images-public/tf-model-server-http-proxy#sha256:SHA
Port: 8000/TCP
Host Port: 0/TCP
Command:
python
/usr/src/app/server.py
--port=8000
--rpc_port=9000
--rpc_timeout=10.0
State: Running
Started: Mon, 18 Feb 2019 18:11:25 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 500m
memory: 500Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-b6dpn (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-b6dpn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-fsdf3
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
nvidia.com/gpu:NoSchedule
Events: <none>
I used the command python predict.py --url=http://W.X.Y.Z:8000/model/my-model:predict to perform the prediction from the serving_script folder but I am getting the a 500 Internal server error as the response. What is going wrong here?
The code for prediction can be found here: https://github.com/kubeflow/examples/tree/master/object_detection/serving_script

Answer 1

It was a mistake from my end. I was using a different input image array format for the model. I was sending an image tensor instead of encoded image string tensor.