I am working on an HPC with no internet access on worker nodes and the only option to save a SetFit trainer after training, is to push it to HuggingFace hub. How do I go about saving it locally to disk?
https://github.com/huggingface/setfit
setfit has this class method
model._save_pretrained(save_directory)
and to load it
saved_model = SetFitModel._from_pretrained(save_directory)
I think you can do this with either pickle or joblib
import pickle
import joblib
pickle.dump(trainer, open('model.pkl', 'wb'))
joblib.dump(trainer, 'model.joblib')
And load in the future with:
job_model = joblib.load('model.joblib')
pkl_model = pickle.load(open('model.pkl', 'rb'))
As an alternative to pushing your Trainer to the Hub as described in SetFit for Text Classification, you can save your trainer locally and use it for prediction.
There is a predict method in the source code. You can use that same method to make predictions from your SetFit object
Save your model locally:
import joblib
# trainer is you SetFit object: setfit.trainer.SetFitTrainer
joblib.dump(trainer, 'my-awesome-setfit-model.joblib')
Load your model and make a classification or inference from your model:
# Load the trainer
trainer = joblib.load('my-awesome-setfit-model.joblib')
# Use the model and predict
trainer.model.predict(["i loved the spiderman movie!", "pineapple on pizza is the worst 🤮"])
You can use the sklearn wrapper:
Train the model
from setfit.modeling import SKLearnWrapper
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
clf = SKLearnWrapper(model, LogisticRegression())
sentences = ["good", "bad", "very good"]
labels = [1, 0, 1]
clf.fit(sentences, labels)
pred1 = clf.predict(["gooood"])
Save the model
path = "model1"
clf.save(path)
Load the model
clf = SKLearnWrapper(None, None)
clf.load(path)
Test
pred2 = clf.predict(["gooood"])
assert pred1 == pred2
Related
I have created a neural network model and created an ensemble learning model which is the voting model. I have combined a Neural network with random forest,and xgboost. Now I saved the model and try to load it to another Jupiter notebook but I get this error AttributeError: Can't get attribute 'create_model' on <module 'main'>
Here is the code to create the models and it in 1st notebook
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
import numpy
# Function to create model, required for KerasClassifier
def create_model(input_shape=66):
#x_shape= data_x.shape
#input_dim=x_shape[1]
# create model
model = Sequential()
model.add(Dense(12, input_dim=66, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1,activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
seed = 7
numpy.random.seed(seed)
Kc_model = KerasClassifier(
create_model, # Pass in function
input_shape=66, # Pass in the dimensions to above function
epochs=100,
batch_size=32,
verbose=False)
Kc_model._estimator_type = "classifier"
Kc_model.fit(x_train, y_train, epochs=100,batch_size=10)
rf = RandomForestClassifier(max_depth=15, random_state=0)
rf.fit(x_train,y_train)
rf_y_pred = rf.predict(x_test)
#Model Score
print("The accuracy score for Random Forest Classifier is")
print("Accuracy:{}%".format(round(metrics.accuracy_score(y_test, rf_y_pred)*100)))
print("Training:{}%".format(round(rf.score(x_train, y_train)*100)))
print("Test set: {}%".format(round(rf.score(x_test, y_test)*100)))
xgboost_model = XGBClassifier()
xgboost_model.fit(x_train, y_train)
xgboost_y_pred = xgboost_model.predict(x_test)
print("The accuracy score for Voting XGB Classifier is")
print("Accuracy:{}%".format(round(metrics.accuracy_score(y_test, xgboost_y_pred)*100)))
print("Training:{}%".format(round(xgboost_model.score(x_train, y_train)*100)))
print("Test set: {}%".format(round(xgboost_model.score(x_test, y_test)*100)))
from keras.wrappers.scikit_learn import KerasClassifier
import scikeras
from tensorflow import keras
voting = VotingClassifier(
estimators = [('rf',rf),('xgboost_model',xgboost_model),('Kc_model',Kc_model) ],
voting='soft')
#reshaping=y_test.reshape(2712,1)
voting_model =voting.fit(x_train, y_train)
voting_pred = voting_model.predict(x_test)
#Model Score
print("The accuracy score for Voting Classifier is")
print("Training:{}%".format(round(voting_model.score(x_train, y_train)*100)))
print("Test set: {}%".format(round(voting_model.score(x_test, y_test)*100)))
import pickle
# save
with open('voting_model.pkl','wb') as f:
pickle.dump(Kc_model,f)
In the second notebook that I try to load the model , I get an error as you can see below
import pickle
import pandas as pd
with open('voting_model.pkl', 'rb') as f:
Kc_model = pickle.load(f)
The reason this happens is that the keras.wrappers.scikit_learn.KerasClassifier wrapper cannot be pickled. The model building function is not saved. Instead, you should pickle the fitted model:
import pickle
# save
with open('voting_model.pkl','wb') as f:
pickle.dump(Kc_model.model, f)
Now, you can load your model and use it as you wish.
with open('voting_model.pkl', 'rb') as f:
model = pickle.load(f)
# Predict something.
model.predict(X_test)
However, if you need a KerasClassifier instance after loading then you should re-wrap it. Then, you also need to save the classes_ attribute. Finally, now the build function would return the loaded pickle:
# Save this as well.
with open('voting_model_classes.pkl', 'wb') as f:
pickle.dump(Kc_model.classes_, f)
import pickle
from keras.wrappers.scikit_learn import KerasClassifier
def load_model():
with open('voting_model.pkl', 'rb') as f:
return pickle.load(f)
def load_classes():
with open('voting_model_classes.pkl', 'rb') as f:
return pickle.load(f)
Kc_model = KerasClassifier(
load_model,
epochs=100,
batch_size=32,
verbose=False)
Kc_model._estimator_type = "classifier"
# We need to manually call it because it will only be called once the classifier is re-fitted.
Kc_model.model = load_model()
Kc_model.classes_ = load_classes()
# Now you can use Kc_model as KerasClassifier.
The error is expected: the model building function gets pickled by name, and that name doesn't exist in your new notebook.
You could try SciKeras which has an initialize method (docs) which you can call to restore stuff like classes_ if you choose to serialize your Keras model using SavedModel directly (SciKeras's KerasClassifier will gladly accept a model instance).
I had Saved model using
tf.keras.experimental.export_saved_model(model, export_path)
This model have custom layers and loss function.
Loading model using
import tensorflow as tf
import tensorflow_hub as hub
import keras
class training_model:
def __init__(self):
path_bce="D:\\nsfw\\training_model\\models\\bce_20210120_153631"
path2="D:\\nsfw\\training_model\\models\\soft-f1_20210120_153631"
self.graph = tf.Graph()
with self.graph.as_default():
self.session = tf.Session()
with self.session.as_default() :
self.reloaded =tf.keras.experimental.load_from_saved_model(path2, custom_objects={'KerasLayer':hub.KerasLayer})
training_model=training_model()
img = keras.preprocessing.image.load_img(
"0drqz7883ox51.jpg", target_size=(224, 224)
)
img_array = keras.preprocessing.image.img_to_array(img)
img_array = tf.expand_dims(img_array, 0) # Create a batch
with training_model.graph.as_default():
with training_model.session.as_default():
print(training_model.reloaded.predict(img_array,steps=1))
It is working fine if i remove Graph and Session but I want to serve this model with API.
You can try something like this.
with self.graph1.as_default():
self.face_graph = tf.compat.v1.GraphDef()
fid = tf.io.gfile.GFile(self.facenet_model, "rb")
serialized_graph = fid.read()
self.face_graph.ParseFromString(serialized_graph)
tf.import_graph_def(self.face_graph, name="")
self.facenet_sess = tf.compat.v1.Session(graph=self.graph1)
self.images_placeholder = self.graph1.get_tensor_by_name("input:0")
self.embeddings = self.graph1.get_tensor_by_name("embeddings:0")
hub.KerasLayer is a TF2 API. One thing to try might be to switch the prediction part from TF1 style (graph + session) to TF2. Alternatively, you could try TensorFlow Serving as an alternative to custom inference logic.
Problem Description
I'm using tensorflow Estimator API, and have encountered a weird phenomenon.
I'm passing the exact same input_fn to both training and evaluation, and for some reason the images which are provided to the network are not identical.
They seem similar, but after taking a closer look, it seems that evaluation images are ok, but train images are somewhat distorted.
After loading them both, I noticed that for some reason the training images go through some kind of ReLu. I affirmed it with this code, which operates on mat_eval and mat_train, which are tensors that input_fn provides in evaluation and train mode:
special_relu = lambda mat: ((mat - 0.5) / 0.5) * ((mat - 0.5) / 0.5 > 0)
np.allclose(mat_train, special_relu(mat_eval))
>>> True
What I thought and tried
My initial thought was that it is some form of BatchNormalization. But BatchNormalization is supposed to happen within the network, and not as some preprocess, shouldn't it?
What I recorded (using tf.summary.image) was the features['image'] object, passed to my model_fn. And if I understand correctly, the features object is passed to model_fn by the input_fn called by the Estimator object.
Regardless, I tried to remove the parts in the code which are supposed to call the BatchNormalization. This had no effect. Of course, I might have not done that in the right way, but as I said it I don't really think it is BatchNormalization.
Code
from datetime import datetime
from pathlib import Path
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.python.platform import tf_logging as logging
from dcnn import modeling
from dcnn.dv_constants import BATCH_SIZE, BATCHES_PER_EPOCH
from dcnn.variant_io import get_input_fn, num_variants_in_ds
logging.set_verbosity(logging.INFO)
new_checkpoint_name = lambda: f'./train_dir/' \
f'{datetime.now().strftime("%d-%m %H:%M:%S")}'
if __name__ == '__main__':
model_name = 'small_inception'
start_from_checkpoint = ''
# start_from_checkpoint = '/home/yonatan/Desktop/yonas_code/dcnn/train_dir' \
# '/2111132905/model.ckpt-256'
model_dir = str(Path(start_from_checkpoint).parent) if \
start_from_checkpoint else new_checkpoint_name()
test = False
train = True
predict = False
epochs = 1
train_dataset_name = 'same_example'
val_dataset_name = 'same_example'
test_dataset_name = 'same_example'
predict_dataset_name = 'same_example'
model = modeling.get_model(model_name=model_name)
estimator = model.make_estimator( \
batch_size=BATCH_SIZE,
model_dir=model_dir,
params=dict(batches_per_epoch=BATCHES_PER_EPOCH),
use_tpu=False,
master='',
# The target of the TensorFlow standard server to use. Can be the empty string to run locally using an inprocess server.
start_from_checkpoint=start_from_checkpoint)
if train:
train_input_fn = get_input_fn(train_dataset_name, repeat=True)
val_input_fn = get_input_fn(val_dataset_name, repeat=False)
steps = (epochs * num_variants_in_ds(train_dataset_name)) / \
BATCH_SIZE
train_spec = tf.estimator.TrainSpec(input_fn=val_input_fn,
max_steps=steps)
eval_spec = tf.estimator.EvalSpec(input_fn=val_input_fn,
throttle_secs=1)
metrics = tf.estimator.train_and_evaluate(estimator, train_spec,
eval_spec)
print(metrics)
I have plenty of more code to share, but I tried to be concise. If anyone has any idea why this behavior happens, or needs more information, let me know.
I want to do a neural network training in Tensorflow/Keras but prefer to use python multiprocessing module to maximize use of system resources and save time. What I do is simply like this (I want to run this code on a system without GPU or with one or more GPUs):
import ... (required modules)
from multiprocessing import Pool
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
tf.keras.backend.set_session(sess)
... some tf and non-tf variable initializations...
... some functions to facilitate reading tensorflow datasets in TFRecord format...
... function defining keras model...
# Main worker function
def doWork(args):
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from tensorflow.keras.models import load_model
train_data = read_datasets(...)
val_data = read_datasets(...)
test_data = read_datasets(...)
if (NumGPUs > 1):
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = keras_model(...)
model.compile(...)
else:
model = keras_model(...)
model.compile(...)
model.fit(train_data, epochs=epochs, steps_per_epoch=train_steps, ...)
_, test_acc = model.evaluate(test_data, steps=test_steps)
...log results...
if __name__ == '__main__':
pool = Pool(processes=2)
a1 = <set of parameters for the first run>
a1 = <set of parameters for the second run>
pool.map(doWork, (a1, a2))
I can run this code on different computers and get my results, but some times I face system hangups (especially if I want to abort execution by pressing CTRL+C) or program termination with different errors, and I guess the above is not the right style of combining Tensorflow/Keras and Python multiprocessing. What is the correct style of writing the above code?
Right now we are successfully able to serve models using Tensorflow Serving. We have used following method to export the model and host it with Tensorflow Serving.
------------
For exporting
------------------
from tensorflow.contrib.session_bundle import exporter
K.set_learning_phase(0)
export_path = ... # where to save the exported graph
export_version = ... # version number (integer)
saver = tf.train.Saver(sharded=True)
model_exporter = exporter.Exporter(saver)
signature = exporter.classification_signature(input_tensor=model.input,
scores_tensor=model.output)
model_exporter.init(sess.graph.as_graph_def(),
default_graph_signature=signature)
model_exporter.export(export_path, tf.constant(export_version), sess)
--------------------------------------
For hosting
-----------------------------------------------
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_name=default --model_base_path=/serving/models
However our issue is - we want keras to be integrated with Tensorflow serving. We would like to serve the model through Tensorflow serving using Keras.
The reason we would like to have that is because - in our architecture we follow couple of different ways to train our model like deeplearning4j + Keras ,
Tensorflow + Keras, but for serving we would like to use only one servable engine that's Tensorflow Serving. We don't see any straight forward way to achieve that. Any comments ?
Thank you.
Very recently TensorFlow changed the way it exports the model, so the majority of the tutorials available on web are outdated. I honestly don't know how deeplearning4j works, but I use Keras quite often. I managed to create a simple example that I already posted on this issue in TensorFlow Serving Github.
I'm not sure whether this will help you, but I'd like to share how I did and maybe it will give you some insights. My first trial prior to creating my custom model was to use a trained model available on Keras such as VGG19. I did this as follows.
Model creation
import keras.backend as K
from keras.applications import VGG19
from keras.models import Model
# very important to do this as a first thing
K.set_learning_phase(0)
model = VGG19(include_top=True, weights='imagenet')
# The creation of a new model might be optional depending on the goal
config = model.get_config()
weights = model.get_weights()
new_model = Model.from_config(config)
new_model.set_weights(weights)
Exporting the model
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import utils
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import build_signature_def, predict_signature_def
from tensorflow.contrib.session_bundle import exporter
export_path = 'folder_to_export'
builder = saved_model_builder.SavedModelBuilder(export_path)
signature = predict_signature_def(inputs={'images': new_model.input},
outputs={'scores': new_model.output})
with K.get_session() as sess:
builder.add_meta_graph_and_variables(sess=sess,
tags=[tag_constants.SERVING],
signature_def_map={'predict': signature})
builder.save()
Some side notes
It can vary depending on Keras, TensorFlow, and TensorFlow Serving
version. I used the latest ones.
Beware of the names of the signatures, since they should be used in the client as well.
When creating the client, all preprocessing steps that are needed for the
model (preprocess_input() for example) must be executed. I didn't try
to add such step in the graph itself as Inception client example.
With respect to serving different models within the same server, I think that something similar to the creation of a model_config_file might help you. To do so, you can create a config file similar to this:
model_config_list: {
config: {
name: "my_model_1",
base_path: "/tmp/model_1",
model_platform: "tensorflow"
},
config: {
name: "my_model_2",
base_path: "/tmp/model_2",
model_platform: "tensorflow"
}
}
Finally, you can run the client like this:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --config_file=model_config.conf
try this script i wrote, you can convert keras models into tensorflow frozen graphs, ( i saw that some models give rise to strange behaviours when you export them without freezing the variables).
import sys
from keras.models import load_model
import tensorflow as tf
from keras import backend as K
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import tag_constants
K.set_learning_phase(0)
K.set_image_data_format('channels_last')
INPUT_MODEL = sys.argv[1]
NUMBER_OF_OUTPUTS = 1
OUTPUT_NODE_PREFIX = 'output_node'
OUTPUT_FOLDER= 'frozen'
OUTPUT_GRAPH = 'frozen_model.pb'
OUTPUT_SERVABLE_FOLDER = sys.argv[2]
INPUT_TENSOR = sys.argv[3]
try:
model = load_model(INPUT_MODEL)
except ValueError as err:
print('Please check the input saved model file')
raise err
output = [None]*NUMBER_OF_OUTPUTS
output_node_names = [None]*NUMBER_OF_OUTPUTS
for i in range(NUMBER_OF_OUTPUTS):
output_node_names[i] = OUTPUT_NODE_PREFIX+str(i)
output[i] = tf.identity(model.outputs[i], name=output_node_names[i])
print('Output Tensor names: ', output_node_names)
sess = K.get_session()
try:
frozen_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), output_node_names)
graph_io.write_graph(frozen_graph, OUTPUT_FOLDER, OUTPUT_GRAPH, as_text=False)
print(f'Frozen graph ready for inference/serving at {OUTPUT_FOLDER}/{OUTPUT_GRAPH}')
except:
print('Error Occured')
builder = tf.saved_model.builder.SavedModelBuilder(OUTPUT_SERVABLE_FOLDER)
with tf.gfile.GFile(f'{OUTPUT_FOLDER}/{OUTPUT_GRAPH}', "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sigs = {}
OUTPUT_TENSOR = output_node_names
with tf.Session(graph=tf.Graph()) as sess:
tf.import_graph_def(graph_def, name="")
g = tf.get_default_graph()
inp = g.get_tensor_by_name(INPUT_TENSOR)
out = g.get_tensor_by_name(OUTPUT_TENSOR[0] + ':0')
sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
tf.saved_model.signature_def_utils.predict_signature_def(
{"input": inp}, {"outout": out})
builder.add_meta_graph_and_variables(sess,
[tag_constants.SERVING],
signature_def_map=sigs)
try:
builder.save()
print(f'Model ready for deployment at {OUTPUT_SERVABLE_FOLDER}/saved_model.pb')
print('Prediction signature : ')
print(sigs['serving_default'])
except:
print('Error Occured, please checked frozen graph')
I have recently added this blogpost that explain how to save a Keras model and serve it with Tensorflow Serving.
TL;DR:
Saving an Inception3 pretrained model:
### Load a pretrained inception_v3
inception_model = keras.applications.inception_v3.InceptionV3(weights='imagenet')
# Define a destination path for the model
MODEL_EXPORT_DIR = '/tmp/inception_v3'
MODEL_VERSION = 1
MODEL_EXPORT_PATH = os.path.join(MODEL_EXPORT_DIR, str(MODEL_VERSION))
# We'll need to create an input mapping, and name each of the input tensors.
# In the inception_v3 Keras model, there is only a single input and we'll name it 'image'
input_names = ['image']
name_to_input = {name: t_input for name, t_input in zip(input_names, inception_model.inputs)}
# Save the model to the MODEL_EXPORT_PATH
# Note using 'name_to_input' mapping, the names defined here will also be used for querying the service later
tf.saved_model.simple_save(
keras.backend.get_session(),
MODEL_EXPORT_PATH,
inputs=name_to_input,
outputs={t.name: t for t in inception_model.outputs})
And then starting a TF serving Docker:
Copy the saved model to the hosts' specified directory. (source=/tmp/inception_v3 in this example)
Run the docker:
docker run -d -p 8501:8501 --name keras_inception_v3 --mount type=bind,source=/tmp/inception_v3,target=/models/inception_v3 -e MODEL_NAME=inception_v3 -t tensorflow/serving
Verify that there's network access to the Tensorflow service. In order to get the local docker ip (172.*.*.*) for testing run:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' keras_inception_v3