sagemaker invoke_endpoint signature_def feature prep - tensorflow

Question

I have a sagemaker tensorflow model using a custom estimator, similar to the abalone.py sagemaker tensorflow example, using build_raw_serving_input_receiver_fn in the serving_input_fn:
def serving_input_fn(params):
tensor = tf.placeholder(tf.float32, shape=[1, NUM_FEATURES])
return build_raw_serving_input_receiver_fn({INPUT_TENSOR_NAME: tensor})()
Predictions are being request from java-script using json:
response = #client.invoke_endpoint(
endpoint_name: #name,
content_type: "application/json",
accept: "application/json",
body: values.to_json
)
Everything fine so far. Now I want to add some feature engineering (scaling transformations on the features using a scaler derived from the training data). Following the pattern of the answer for Data Normalization with tensorflow tf-transform
I've now got serving_input_fn like this:
def serving_input_fn(params):
feature_placeholders = {
'f1': tf.placeholder(tf.float32, [None]),
'f2': tf.placeholder(tf.float32, [None]),
'f3': tf.placeholder(tf.float32, [None]),
}
features = {
key: tf.expand_dims(tensor, -1)
for key, tensor in feature_placeholders.items()
}
return tf.estimator.export.ServingInputReceiver(add_engineering(features), feature_placeholders)
From saved_model_cli show --dir . --all I can see the input signature has changed:
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['f1'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: Placeholder_1:0
inputs['f2'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: Placeholder_2:0
inputs['f3'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: Placeholder:0
How do I prepare features for prediction from this new model? In python I've been unsuccessfully trying things like
requests = [{'f1':[0.1], 'f2':[0.1], 'f3':[0.2]}]
predictor.predict(requests)
also need to send prediction requests from java-script.

Answer 1

You can define an
def input_fn(data=None, content_type=None):
This would be called directly when a call is made to SageMaker. You can do your feature preparation in this function. model_fn would be called after this function.
Make sure you return a dict of string and TensorProto.
dict{"input tensor name", TensorProto} from the input_fn method.
You can find more details where
https://docs.aws.amazon.com/sagemaker/latest/dg/tf-training-inference-code-template.html
A sample input_fn would look something like below
def input_fn(data=None, content_type=None):
"""
Args:
data: An Amazon SageMaker InvokeEndpoint request body
content_type: An Amazon SageMaker InvokeEndpoint ContentType value for data.
Returns:
object: A deserialized object that will be used by TensorFlow serving as input.
"""
# `inputs` is based on the parameters defined in the model spec's signature_def
return {"inputs": tf.make_tensor_proto(data, shape=(1,))}

Answer 2

Have managed to make the feature values available on their way into prediction via a sagemaker input_fn definition, as suggested by Raman. It means going back to the build_raw_serving_input_receiver_fn serving_input_fn I started with (top of post). The input_fn looks like this:
def input_fn(data=None, content_type=None):
if content_type == 'application/json':
values = np.asarray(json.loads(data))
return {"inputs": tf.make_tensor_proto(values=values, shape=values.shape, dtype=tf.float32)}
else:
return {"inputs": data}
Although I can't pass e.g. a scaler from training into this procedure, it will probably work to embed it in the model.py file that sagemaker requires (which contains this input_fn defn). What I have is responding correctly to by addressed from python either by
data = [[0.1, 0.2, 0.3]]
payload = json.dumps(data)
response = client.invoke_endpoint(
EndpointName=endpoint_name,
Body=payload,
ContentType='application/json'
)
result = json.loads(response['Body'].read().decode())
or
values = np.asarray([[0.1, 0.2, 0.3]])
prediction = predictor.predict(values)
This is all new to me... please recommend improvements/alert me to potential problems if you know of any.