I am restoring a pre-trained model in TensorFlow and would like to know if my code works as desired. Here is the code below. It looks like it does not make use of the pre-trained model because the loss from the code below is way higher than training error, e.g., 200 something. Please let me know how I can modify it. Thank you in advance.
def main(_):
with tf.Graph().as_default() as g:
x, img, rows, cols = load_image(FLAGS.input, FLAGS.image_size)
with slim.arg_scope(resnet_v2.resnet_arg_scope(weight_decay=0.0005)):
logits, endpoints = resnet_v2.resnet_v2_152(x, nb_classes, is_training=False)
predictions = tf.argmax(logits, 1)
saver = tf.train.Saver()
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
# Load a pretrained model.
print("\nLoading a model...")
saver.restore(sess, checkpoint_path)
print("\nFeedforwarding...")
pred, loss_out, output, grads_val = sess.run([predictions, loss, conv_layer, norm_grads])
Related
Currently I have resnet_v2_50.ckpt from tensorflow's open sourced pretrained model. I am trying to serve this model in Go cause my backend for my web app is going to be in Go. If I were to create my own model and train it, and then save it. I have no trouble with serving it in Go but I am trying to use pre-trained model to save time on my end.
Here's a simple example of how I save my model
mnist = input_data.read_data_sets(DATA_DIR, one_hot=True)
# Recall that each image is 28x28
x = tf.placeholder(tf.float32, [None, 784], name='imageinput')
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.add(tf.matmul(x, W), b)
labels = tf.placeholder(tf.float32, [None, 10])
cross_entropy_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=y))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy_loss)
with tf.Session() as sess:
with tf.device("/cpu:0"):
sess.run(tf.global_variables_initializer())
for i in range(1000):
batch_x, batch_label = mnist.train.next_batch(100)
loss, _ = sess.run([cross_entropy_loss, train_step], feed_dict={x: batch_x, labels: batch_label})
print '%d: %f' % (i + 1, loss)
infer = tf.argmax(y, axis=1, name='infer')
truth = tf.argmax(labels, axis=1)
correct_prediction = tf.equal(infer, truth)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print sess.run(accuracy, feed_dict={x: mnist.test.images, labels: mnist.test.labels})
print 'Time to save the graph!'
builder = tf.saved_model.builder.SavedModelBuilder('mnist_model')
builder.add_meta_graph_and_variables(sess, ['serve'])
builder.save()
I can load it using tensorflow in Go
model, err := tf.LoadSavedModel("./tf_mnist_py/mnist_model", []string{"serve"}, nil)
if err != nil {
fmt.Printf("Error loading saved model: %s\n", err.Error())
return
}
defer model.Session.Close()
But now when it comes to pretrained model, I am dealing with ckpt files. One solution I have is to load it up in Python and then save it as protobuf.
from tensorflow.python.tools import inspect_checkpoint as ckpt
ckpt.print_tensors_in_checkpoint_file('./resnet50/resnet_v2_50.ckpt',
tensor_name='',
all_tensors=False,
all_tensor_names=False)
tf.reset_default_graph()
saver = tf.train.Saver()
with tf.Session() as sess:
saver.restore(sess, './resnet50/resnet_v2_50.ckpt')
print 'Model is restored'
print 'Time to save the graph!'
builder = tf.saved_model.builder.SavedModelBuilder('resnet_50_model')
builder.add_meta_graph_and_variables(sess, ['serve'])
builder.save()
However, this gives me an error saying that ValueError: No variables to save. I can fix it by declaring a variable
v1 = tf.get_variable('total_loss/ExponentialMovingAverage', shape=[])
But here comes my question, does that mean I must declare EVERY variable in the ResNet50 and have tensorflow load the values from ckpt file into those variables and then I perform save? Is there a shortcut to do this?
You should import the variables as well from the .meta file if you have it available with the pretrained model like
saver = tf.train.import_meta_graph('./resnet50/resnet_v2_50.meta')
Detailed tutorial here.
If you don't have the .meta available, but you do have the network generating code, like with resnet_v2_50 in tensorflow/models/blob/master/research/slim/nets/resnet_v2.py then you should import that file and run the resnet_v2_50 function, which will define all the variables for you. Then restore the checkpoint.
I have used the retrain.py script provided at the tensorflow github to finetune a pretrained inceptionV3 model on my own dataset. I have the model saved to disk and now I want to use it as the starting point for another round of training in which I retrain all of the convolutional layers. Below is the code I am attempting to use to create the new graph. I thought that once I have the graph loaded into the default graph I could access the variables in it with tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES) and set them all to trainable. However, there doesn't appear to be anything in any of the tf.GraphKeys variables (GLOBAL_VARIABLES, TRAINABLE_VARIABLES, MODEL_VARIABLES etc.). So when I try to create the optimizer I get an error "ValueError: No variables to optimize." What am I doing wrong?
def create_graph(model_path, class_count):
"""Creates a graph from saved GraphDef file and returns a saver."""
with tf.Graph().as_default() as graph:
# Creates graph from saved graph_def.pb.
with tf.gfile.FastGFile(model_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
_ = tf.import_graph_def(graph_def, name='')
#import logits from saved graph
logits_tensor = graph.get_tensor_by_name("final_training_ops/biases/final_biases:0")
ground_truth_input = tf.placeholder(tf.float32,
[None, class_count],
name='GroundTruthInput')
print(tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
#connect logits to the training ops
with tf.name_scope('cross_entropy'):
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
labels=ground_truth_input, logits=logits_tensor)
with tf.name_scope('total'):
cross_entropy_mean = tf.reduce_mean(cross_entropy)
tf.summary.scalar('cross_entropy', cross_entropy_mean)
with tf.name_scope('train'):
optimizer = tf.train.GradientDescentOptimizer(0.001)
train_step = optimizer.minimize(cross_entropy_mean)
return graph, ground_truth_input, cross_entropy_mean, train_step
I would really appreciate it if I could get some help in saving and restoring LSTMs.
I have this LSTM layer -
# LSTM cell
cell = tf.contrib.rnn.LSTMCell(n_hidden)
output, current_state = tf.nn.dynamic_rnn(cell, word_vectors, dtype=tf.float32)
outputs = tf.transpose(output, [1, 0, 2])
last = tf.gather(outputs, int(outputs.get_shape()[0]) - 1)
# Saver function
saver = tf.train.Saver()
saver.save(sess, 'test-model')
The saver saves the model and allows me to save and restore the weights and biases of the LSTM. However, I need to restore this LSTM layer and feed it a new set of inputs.
To restore the entire model, I'm doing:
with tf.Session() as sess:
saver = tf.train.import_meta_graph('test-model.meta')
saver.restore(sess, tf.train.latest_checkpoint('./'))
Is it possible for me to initialize an LSTM cell with the pre-trained weights and biases?
If not, how do I restore this LSTM layer?
Thank you very much!
You are already loading the model, and so the weights of the model. All you need to do is use get_tensor_by_name to get any tensor from the graph and use it for inference.
Example:
with tf.Session() as sess:
saver = tf.train.import_meta_graph('test-model.meta')
saver.restore(sess, tf.train.latest_checkpoint('./'))
# Get the tensors by their variable name
word_vec = = detection_graph.get_tensor_by_name('word_vec:0')
output_tensor = detection_graph.get_tensor_by_name('outputs:0')
sess.run(output_tensor, feed_dict={word_vec: ...})
In the above example word_vec and outputs are names assigned to the tensors during creation of the graph. Make sure you assign names, so that they can be called by their name.
img_batch, label_batch = tf.train.shuffle_batch([img, label],
batch_size=128,capacity=2000,
min_after_dequeue=1000)
logits = cifar10.inference(img_batch)
saver = tf.train.Saver()
sess = tf.Session()
init = tf.initialize_all_variables()
sess.run(init)
saver.restore(sess,"/Tensorflow/cata.tfmodel")
tf.train.start_queue_runners(sess=sess)
I have trained the model "cata.tfmodel" in another program and restored the sess in this program, but I don't know how to use this model to classify one picture and output the classification.
The type of my dataset is tfrecords and read it by queue.
I'm confused that if I covert one picture into tfrecords, it will contain image and label, what to do next?
logits = cifar10.inference(img_batch)
saver = tf.train.Saver(tf.all_variables())
Besides, If I delete logits , saver will throw ValueError: No variables to save
Trying to run the Inceptionv3 Tensorflow model with the architecture and the checkpoint provided by Google here.
My issue is that my script crashes on saver.restore(sess, "./inception_v3.ckpt") with the following error:
tensorflow.python.framework.errors.NotFoundError: Tensor name "InceptionV3/Mixed_5b/Branch_1/Conv2d_0b_5x5/biases" not found in checkpoint files ./inception_v3.ckpt
Here is my code:
import tensorflow as tf
import inception_v3
with tf.Session() as sess:
image = tf.read_file('./file.jpg')
# code to decode, crop, convert jpeg
eval_inputs = tf.pack([image])
logits, _ = inception_v3.inception_v3(eval_inputs, num_classes=1001, is_training=False)
sess.run(tf.initialize_all_variables())
saver = tf.train.Saver()
saver.restore(sess, "./inception_v3.ckpt")
I get the same errors with the other checkpoint/model combinations so this must be an issue with my code. Not sure what I am doing wrong though.
Thank you
Indeed the checkpoint file does not contain this tensor. Can you file a bug on github?
You need to call inception_v3() within the arg_scope() returned by inception_v3_arg_scope() like this:
import tensorflow as tf
import tensorflow.contrib.slim as slim
from nets.inception_v3 import inception_v3, inception_v3_arg_scope
height = 299
width = 299
channels = 3
# Create graph
X = tf.placeholder(tf.float32, shape=[None, height, width, channels])
with slim.arg_scope(inception_v3_arg_scope()):
logits, end_points = inception_v3(X, num_classes=1001,
is_training=False)
predictions = end_points["Predictions"]
saver = tf.train.Saver()
X_test = ... # your images, shape [batch_size, 299, 299, 3]
# Execute graph
with tf.Session() as sess:
saver.restore(sess, "./inception_v3.ckpt")
predictions_val = predictions.eval(feed_dict={X: X_test})
predicted_classes = np.argmax(predictions_val, axis=1)
I recommend clearly separating the construction phase and the execution phase. Just tested on a random photo on the web, and it worked fine. :)