I have used the retrain.py script provided at the tensorflow github to finetune a pretrained inceptionV3 model on my own dataset. I have the model saved to disk and now I want to use it as the starting point for another round of training in which I retrain all of the convolutional layers. Below is the code I am attempting to use to create the new graph. I thought that once I have the graph loaded into the default graph I could access the variables in it with tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES) and set them all to trainable. However, there doesn't appear to be anything in any of the tf.GraphKeys variables (GLOBAL_VARIABLES, TRAINABLE_VARIABLES, MODEL_VARIABLES etc.). So when I try to create the optimizer I get an error "ValueError: No variables to optimize." What am I doing wrong?
def create_graph(model_path, class_count):
"""Creates a graph from saved GraphDef file and returns a saver."""
with tf.Graph().as_default() as graph:
# Creates graph from saved graph_def.pb.
with tf.gfile.FastGFile(model_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
_ = tf.import_graph_def(graph_def, name='')
#import logits from saved graph
logits_tensor = graph.get_tensor_by_name("final_training_ops/biases/final_biases:0")
ground_truth_input = tf.placeholder(tf.float32,
[None, class_count],
name='GroundTruthInput')
print(tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
#connect logits to the training ops
with tf.name_scope('cross_entropy'):
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
labels=ground_truth_input, logits=logits_tensor)
with tf.name_scope('total'):
cross_entropy_mean = tf.reduce_mean(cross_entropy)
tf.summary.scalar('cross_entropy', cross_entropy_mean)
with tf.name_scope('train'):
optimizer = tf.train.GradientDescentOptimizer(0.001)
train_step = optimizer.minimize(cross_entropy_mean)
return graph, ground_truth_input, cross_entropy_mean, train_step
Related
I followed this tutorial about building an encoder-decoder language translation model and built one for my native language.
Now I want to save it, deploy on cloud ML engine and make predictions with HTTP request.
I couldn't find a clear example on how to save this model,
I am new to ML and found TF save guide v confusing..
Is there a way to save this model using something like
tf.keras.models.save_model
Create the train saver after opening the session and after the training is done save the model:
with tf.Session() as sess:
saver = tf.train.Saver()
# Training of the model
save_path = saver.save(sess, "logs/encoder_decoder")
print(f"Model saved in path {save_path}")
You can save a Keras model in Keras's HDF5 format, see:
https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model
You will want to do something like:
import tf.keras
model = tf.keras.Model(blah blah)
model.save('my_model.h5')
If you migrate to TF 2.0, it's more straightforward to build a model in tf.keras and deploy using the TF SavedModel format. This 2.0 tutorial shows using a pretrained tf.keras model, saving the model in SavedModel format, deploying to the cloud and then doing an HTTP request for a prediction:
https://www.tensorflow.org/beta/guide/saved_model
I know I am a little late but was having the same problem (see How do I save an encoder-decoder model with TensorFlow? for more details) and figured out a solution. It's a little hacky, but it works!
Step 1 - Saving your model
Save your tokenizer (if applicable). Then individually save the weights of the model you used to train your data (naming your layers helps here).
# Save the tokenizer
with open('tokenizer.pickle', 'wb') as handle:
pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)
# save the weights individually
for layer in model.layers:
weights = layer.get_weights()
if weights != []:
np.savez(f'{layer.name}.npz', weights)
Step 2 - reloading the weights
You will want to reload the tokenizer (as applicable) then load the weights you just saved. The loaded weights are in an npz format so can't be used directly, but the very short documentation will tell you everything you need to know about this file type https://numpy.org/doc/stable/reference/generated/numpy.savez.html
# load the tokenizer
with open('tokenizer.pickle', 'rb') as handle:
tokenizer = pickle.load(handle)
# load the weights
w_encoder_embeddings = np.load('encoder_embeddings.npz', allow_pickle=True)
w_decoder_embeddings = np.load('decoder_embeddings.npz', allow_pickle=True)
w_encoder_lstm = np.load('encoder_lstm.npz', allow_pickle=True)
w_decoder_lstm = np.load('decoder_lstm.npz', allow_pickle=True)
w_dense = np.load('dense.npz', allow_pickle=True)
Step 3 - Recreate the your training model and apply the weights
You'll want to re-run the code you used to create your model. In my case this was:
encoder_inputs = Input(shape=(None,), name="encoder_inputs")
encoder_embeddings = Embedding(vocab_size, embedding_size, mask_zero=True, name="encoder_embeddings")(encoder_inputs)
# Encoder lstm
encoder_lstm = LSTM(512, return_state=True, name="encoder_lstm")
encoder_outputs, state_h, state_c = encoder_lstm(encoder_embeddings)
# discard `encoder_outputs` and only keep the states.
encoder_states = [state_h, state_c]
# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None,), name="decoder_inputs")
# target word embeddings
decoder_embeddings = Embedding(vocab_size, embedding_size, mask_zero=True, name="decoder_embeddings")
training_decoder_embeddings = decoder_embeddings(decoder_inputs)
# decoder lstm
decoder_lstm = LSTM(512, return_sequences=True, return_state=True, name="decoder_lstm")
decoder_outputs, _, _ = decoder_lstm(training_decoder_embeddings,
initial_state=encoder_states)
decoder_dense = TimeDistributed(Dense(vocab_size, activation='softmax'), name="dense")
decoder_outputs = decoder_dense(decoder_outputs)
# While training, model takes input and traget words and outputs target strings
loaded_model = Model([encoder_inputs, decoder_inputs], decoder_outputs, name="training_model")
Now you can apply your saved weights to these layers! It takes a little bit of investigation which weight goes to which layer, but this is made a lot easier by naming your layers and inspecting your model layers with model.layers.
# set the weights of the model
loaded_model.layers[2].set_weights(w_encoder_embeddings['arr_0'])
loaded_model.layers[3].set_weights(w_decoder_embeddings['arr_0'])
loaded_model.layers[4].set_weights(w_encoder_lstm['arr_0'])
loaded_model.layers[5].set_weights(w_decoder_lstm['arr_0'])
loaded_model.layers[6].set_weights(w_dense['arr_0'])
Step 4 - Create the inference model
Finally, you can now create your inference model based on this training model! Again in my case this was:
encoder_model = Model(encoder_inputs, encoder_states)
# Redefine the decoder model with decoder will be getting below inputs from encoder while in prediction
decoder_state_input_h = Input(shape=(512,))
decoder_state_input_c = Input(shape=(512,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
inference_decoder_embeddings = decoder_embeddings(decoder_inputs)
decoder_outputs2, state_h2, state_c2 = decoder_lstm(inference_decoder_embeddings, initial_state=decoder_states_inputs)
decoder_states2 = [state_h2, state_c2]
decoder_outputs2 = decoder_dense(decoder_outputs2)
# sampling model will take encoder states and decoder_input(seed initially) and output the predictions(french word index) We dont care about decoder_states2
decoder_model = Model(
[decoder_inputs] + decoder_states_inputs,
[decoder_outputs2] + decoder_states2)
And voilà! You can now make inferences using the previously trained model!
I am restoring a pre-trained model in TensorFlow and would like to know if my code works as desired. Here is the code below. It looks like it does not make use of the pre-trained model because the loss from the code below is way higher than training error, e.g., 200 something. Please let me know how I can modify it. Thank you in advance.
def main(_):
with tf.Graph().as_default() as g:
x, img, rows, cols = load_image(FLAGS.input, FLAGS.image_size)
with slim.arg_scope(resnet_v2.resnet_arg_scope(weight_decay=0.0005)):
logits, endpoints = resnet_v2.resnet_v2_152(x, nb_classes, is_training=False)
predictions = tf.argmax(logits, 1)
saver = tf.train.Saver()
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
# Load a pretrained model.
print("\nLoading a model...")
saver.restore(sess, checkpoint_path)
print("\nFeedforwarding...")
pred, loss_out, output, grads_val = sess.run([predictions, loss, conv_layer, norm_grads])
I would really appreciate it if I could get some help in saving and restoring LSTMs.
I have this LSTM layer -
# LSTM cell
cell = tf.contrib.rnn.LSTMCell(n_hidden)
output, current_state = tf.nn.dynamic_rnn(cell, word_vectors, dtype=tf.float32)
outputs = tf.transpose(output, [1, 0, 2])
last = tf.gather(outputs, int(outputs.get_shape()[0]) - 1)
# Saver function
saver = tf.train.Saver()
saver.save(sess, 'test-model')
The saver saves the model and allows me to save and restore the weights and biases of the LSTM. However, I need to restore this LSTM layer and feed it a new set of inputs.
To restore the entire model, I'm doing:
with tf.Session() as sess:
saver = tf.train.import_meta_graph('test-model.meta')
saver.restore(sess, tf.train.latest_checkpoint('./'))
Is it possible for me to initialize an LSTM cell with the pre-trained weights and biases?
If not, how do I restore this LSTM layer?
Thank you very much!
You are already loading the model, and so the weights of the model. All you need to do is use get_tensor_by_name to get any tensor from the graph and use it for inference.
Example:
with tf.Session() as sess:
saver = tf.train.import_meta_graph('test-model.meta')
saver.restore(sess, tf.train.latest_checkpoint('./'))
# Get the tensors by their variable name
word_vec = = detection_graph.get_tensor_by_name('word_vec:0')
output_tensor = detection_graph.get_tensor_by_name('outputs:0')
sess.run(output_tensor, feed_dict={word_vec: ...})
In the above example word_vec and outputs are names assigned to the tensors during creation of the graph. Make sure you assign names, so that they can be called by their name.
img_batch, label_batch = tf.train.shuffle_batch([img, label],
batch_size=128,capacity=2000,
min_after_dequeue=1000)
logits = cifar10.inference(img_batch)
saver = tf.train.Saver()
sess = tf.Session()
init = tf.initialize_all_variables()
sess.run(init)
saver.restore(sess,"/Tensorflow/cata.tfmodel")
tf.train.start_queue_runners(sess=sess)
I have trained the model "cata.tfmodel" in another program and restored the sess in this program, but I don't know how to use this model to classify one picture and output the classification.
The type of my dataset is tfrecords and read it by queue.
I'm confused that if I covert one picture into tfrecords, it will contain image and label, what to do next?
logits = cifar10.inference(img_batch)
saver = tf.train.Saver(tf.all_variables())
Besides, If I delete logits , saver will throw ValueError: No variables to save
I am new to TensorFlow. I have trained the inception_v3 model successfully with my training data; now I want to predict the output of several images, but the number of them is different from the batch_size in training. I did it as follows:
from tensorflow.contrib.slim.nets import inception_v3 as inception
checkpoint_dir =os.path.join('runs', configure_name, 'checkpoints')
checkpoint_file = tf.train.latest_checkpoint(checkpoint_dir)
graph = tf.Graph()
with graph.as_default():
session_conf = tf.ConfigProto(
allow_soft_placement=True,
log_device_placement=False)
sess = tf.Session(config=session_conf)
with sess.as_default():
# Load the saved meta graph and restore variables
saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
saver.restore(sess, checkpoint_file)
x = tf.placeholder(tf.float32, [batch_size,input_size,input_size,num_channels], name='images')
_, end_points = inception.inception_v3(x,num_classes=num_classes, is_training=False)
outputs = end_points['Predictions']
scores = sess.run(outputs, feed_dict={x: x_eval})
predictions = np.argmax(scores,axis=1)
It gave me the errors as follows:
FailedPreconditionError: Attempting to use uninitialized value InceptionV3/Conv2d_1a_3x3/weights_1
It seems that the model parameters in "outputs" are not fed in successfully, but I do not know how to do it. Any ideas? Thanks.
Here you explicitly set the first dim of input placeholder x batch_size, so each time you need to feed a numpy.array type tensor with the same dim or your program will go wrong.
A solution can be setting the first dim of any placeholder(input and label) None so that this dim can be any int or different during training and validation
UPDATE:
if you have already trained your model with a fixed first dim placeholder input(and label), you can change it when restore this graph with tf.train.import_meta_graph(grapg_def=your_graph_def, input_map={'your_train_input_placedholer_name':new_placeholder})
here new_placeholder is a placeholder you newly create with unfixed first dim.