I want to deploy a model with tensorflow serving. In the model, there are some ops that can't place into gpu. While deploy my model, I should config the tensorflow serving config file platform_config_file. The most important is allow_soft_placement. How to write it?
platform_configs: {
key: "tensorflow",
value: {
source_adapter_config: {
type_url: "type.googleapis.com/tensorflow.serving.SavedModelBundleSourceAdapterConfig"
value:
And, it use any.proto, the value must be bytes.How to write this.
thanks very much.
Related
In order to serve a model in triton Inference server, the model repository should look like below
<model-repository-path>/
<model-name>/
config.pbtxt
1/
model.graphdef
But I am stuck at creating the above repository.
After you trained a model in TensorFlow,
1)How do you create model.graphdef?
2)how do you create config.pbtxt?
'model.graphdef' is your trained tensorflow model, it will have the extension .pb. When you train a model in tensorflow the weights are saved as a pb file. You can add that to this folder.
In this case you do not need to create a config.pbtxt file because triton inference server can automatically generate the configurations from tensorflow, tensorRT and ONNX models. You can simply start the server with the command --strict-model-config = false and it will generate the config.pbtxt file for you.
If you do wish to create your own config.pbtxt you can do so as well. The details are available in the official triton documentation.
The auto generated config file can be viewed using
curl localhost:8000/v2/models/<model_name>/config | jq
Use information from this output to create your own config file
I have a trained sklearn SVM model in .pkl format and a Keras .h5 model. Can I load these models using tensorflow.js on a browser?
I do most of my coding in python and not sure how to work with tensorflow.js
My model saving code looks like this
from sklearn.externals import joblib
joblib.dump(svc,'model.pkl')
model = joblib.load('model.pkl')
prediction = model.predict(X_test)
#------------------------------------------------------------------
from keras.models import load_model
model.save('model.h5')
model = load_model('my_model.h5')
In order to deploy your model with tensorflow-js, you need to use the tensorflowjs_converter, so you also need to install the tensorflowjs dependency.
You can do that in python via pip install tensorflowjs.
Next, you convert your trained model via this operation, according to your custom names: tensorflowjs_converter --input_format=keras /tmp/model.h5 /tmp/tfjs_model, where the last path is the output path of the conversion result.
Note that, after the conversion you will get a model.json (architecture of your model) and a list of N shards (weights split in N shards).
Then, in JavaScript, you need to us the function tf.loadLayersModel(MODEL_URL), where MODEL_URL is the url pointing to your model.json. Ensure that, at the same location with the model.json, the shards are also located.
Since this is an asynchronous operation(you do not want your web-page to get blocked while your model is loading), you need to use the JavaScript await keyword; hence await tf.loadLayersModel(MODEL_URL)
Please have a look at the following link to see an example: https://www.tensorflow.org/js/guide/conversion
After I've trained and deployed the model with AWS SageMaker, I want to evaluate it on several csv files:
- category-1-eval.csv (~700000 records)
- category-2-eval.csv (~500000 records)
- category-3-eval.csv (~800000 records)
...
The right way to do this is with using Estimator.evaluate() method, as it is fast.
The problem is - I cannot find the way to restore SageMaker model into Tensorflow Estimator, is it possible?
I've tried to restore a model like this:
tf.estimator.DNNClassifier(
feature_columns=...,
hidden_units=[...],
model_dir="s3://<bucket_name>/checkpoints",
)
In AWS SageMaker documentation a different approach is described - to test the actual endpoint from the Notebook - but it takes to much time and requires a lot of API calls to the endpoint.
if you used the built-in Tensorflow container, your model has been saved in Tensorflow Serving format, e.g.:
$ tar tfz model.tar.gz
model/
model/1/
model/1/saved_model.pb
model/1/variables/
model/1/variables/variables.index
model/1/variables/variables.data-00000-of-00001
You can easily load it with Tensorflow Serving on your local machine, and send it samples to predict. More info at https://www.tensorflow.org/tfx/guide/serving
Is there a simple way in tensorflow to make any model run in parallel without reprogramming the model? Like in PyTorch using the DataParallel method
I am new to tensorflow so any reference or code example is very much appreciated.
maybe you can modify the config proto of tensorflow and use:
config = tf.ConfigProto(device_count={'GPU': GPU},
inter_op_parallelism_threads=NBR_THREADS,
intra_op_parallelism_threads=NBR_THREADS)
with tf.Session(config=config) as sess:
<run_your_code>
i tried to follow this tutorial on how to convert a Keras H5 Model zu ProtoBuff and serving it using Tensorflow Serve:
https://towardsdatascience.com/deploying-keras-models-using-tensorflow-serving-and-flask-508ba00f1037
That tutorial among many other resources on the web use "tf.saved_model.simple_save", which is deprecated and removed by now (March 2019).
Converting the h5 into pb using freeze_session as shown here:
How to export Keras .h5 to tensorflow .pb?
Seems to miss a "serve" Tag, as the tensorflow_model_server outputs:
Loading servable: {name: ImageClassifier version: 1} failed: Not found: Could not find meta graph def matching supplied tags: { serve }. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: saved_model_cli
checked it with saved_model_cli, there are no tags.
What is the way to make a h5 model serveable in tensorflow_server nowadays?
NOTE: This applies to TF 2.0+
I'm assuming you have your Keras model in model.h5.
Firstly, just load the model with tensorflow's implementation of Keras:
from tensorflow import keras
model = keras.models.load_model('model.h5')
Then, simply export a SavedModel
keras.experimental.export_saved_model(model, 'path_to_saved_model')
Finally, apply any transformation you nomally d to go from SavedModel to the .pb inference file (e.g.: freezing, optimizing for inference, etc)
You can hve more details and a full example in TF's official guide for saving and serializing models in TF 2.0