I am saving a tensor flow graph and variables using :
builder = tf.saved_model.builder.SavedModelBuilder(export_dir)
builder.add_meta_graph_and_variables(sess, ["nn"])
builder.save(as_text=False)
I would like the smallest save file with which I can run classification with. Specifically I am looking to exclude Adam variables that were created during training from the save file.
When I iterate on the values returned by tf.all_variables() prior to saving, I get variables I expect like:
tf.Variable 'mymodel/fully_connected/weights:0' shape=(128, 100) dtype=float32_ref
But also two Adam copies of the same:
tf.Variable 'mymodel/train/mymodel/fully_connected/weights/Adam:0' shape=(128, 100) dtype=float32_ref
tf.Variable 'mymodel/train/mymodel/fully_connected/weights/Adam_1:0' shape=(128, 100) dtype=float32_ref
This triples the size of my checkpoint file and I am constrained by the target system. The save file will only be used for classification, not training, so I don't need Adam optimizer variables.
Any suggestions on how I can most easily not have these written to the save file?
Any other suggestions for reducing save file size while still being able to run classification are appreciated.
I ended up getting things working using the following code.
I saved the graph, weights, biases, etc. like this:
with tf.Session(graph=tf.Graph()) as sess:
// Checkpoint of trained model that has Adam optimizer variables
tf.saved_model.loader.load(sess, ["audio_nn"], FLAGS.checkpoint_dir)
# Dump out checkpoint without Adam optimizer variables
saver = tf.train.Saver(tf.model_variables())
saver.save(sess, 'my-model')
Fortunately tf.model_variables() returned only the variables needed for the model and not the Adam optimizer variables.
And restored for classification like this:
with tf.Session(graph=tf.Graph()) as sess:
imported_meta = tf.train.import_meta_graph("my-model.meta")
imported_meta.restore(sess, "my-model")
features_tensor = sess.graph.get_tensor_by_name( params.INPUT_TENSOR_NAME)
prediction_tensor = sess.graph.get_tensor_by_name( 'mymodel/prediction:0')
... prediction code ...
The checkpoint file is now about one third the size as before.
I was having the same issue, but my tf.model_variables() was empty. I was able to save only the necessary variables using your same idea except substituting tf.model_variables() with tf.trainable_variables(). This nicely excludes the ADAM variables. My model saving code looks as follows:
saver = tf.train.Saver(tf.trainable_variables())
saver.save(session, 'model.ckpt')
You can specify the variables in the saver:
saver = tf.train.Saver(...variables...)
saver.save(sess, 'my-model', global_step=step)
Reference: https://www.tensorflow.org/api_docs/python/tf/train/Saver
Related
So I'm training a model on a machine with GPU. Of course I save it in the end of the training:
a = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
saver = tf.train.Saver(a)
saver.save(sess, save_path)
Now I have one file, but every time I restore the model from the same file I get different numbers in the matrices, and different predictions for the same examples.
I restore the model like this:
saver = tf.train.import_meta_graph('{}.meta'.format(save_path))
sess.run(tf.global_variables_initializer())
saver.restore(sess, save_path)
What is happening here?
When you call sess.run(tf.global_variables_initializer()) after importing the frozen graph, you probably reinitialise some variables that you should not.
Instead, you should initialise only the uninitialised variables. One way to do it would be (credit to this answer)
uninitialized_vars = []
for var in tf.all_variables():
try:
sess.run(var)
except tf.errors.FailedPreconditionError:
uninitialized_vars.append(var)
init_new_vars_op = tf.initialize_variables(uninitialized_vars)
How does one save the weights of a single neural network in a tensorflow graph so that it can be loaded in a different program into a network with the same architecture?
My training code requires 3 other neural networks for the training process alone. If I were to use saver.save(sess, 'my-model)', wouldn't it save all the variables in the tensorflow graph? This doesn't seem correct for my use case.
Maybe this comes from my misunderstanding of how tensorflow should work. Am I approaching this problem correctly?
The best approach would be to use tensorflow variables scope. Say you have model_1, model_2, and model_3 and you only want to save model_1:
First, define the models in your training code:
with tf.variable_scope('model_1'):
model one declaration here
...
with tf.variable_scope('model_2'):
model one declaration here
...
with tf.variable_scope('model_3'):
model one declaration here
...
Next, define saver over the variables of model_1:
model_1_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope="model_1")
saver = tf.train.Saver(model_1_variables)
While training you can save a checkpoint just like you mentioned:
saver.save(sess, 'my-model')
After your training is done and you want to restore the weights in your evaluation code, make sure you define model_1 and saver the same way:
with tf.variable_scope('model_1'):
model one declaration here
...
model_1_variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope="model_1")
saver = tf.train.Saver(model_1_variables)
sess = tf.Session()
saver.restore(sess, 'my-model')`
How can I finetune a pretrained model in tensorflow on a new dataset? In Caffe I can simply rename the last layer and set some parameters for random initialization. Is something similar possible in tensorflow?
Say I have a checkpoint file (deeplab_resnet.ckpt) and some code that sets up the computational graph in which I can modify the last layer such that it has the same number of ouputs as the new dataset has classes.
Then I try to start the session like this:
sess = tf.Session(config=config)
init = tf.initialize_all_variables()
sess.run(init)
trainable = tf.trainable_variables()
saver = tf.train.Saver(var_list=trainable, max_to_keep=40)
saver.restore(sess, 'ckpt_path/deeplab_resnet.ckpt')
However this gives me an error when calling the saver.restore function since it expects the exact same graph structure as the the one it was saved from.
How can I only load all weights except for the last layer from the 'ckpt_path/deeplab_resnet.ckpt' file?
I also tried changing the Classification layer name but no luck there either...
I'm using the tensorflow-deeplab-resnet model
You can specify the names of the variables that you want to restore.
So, you can get a list of all of the variables in the model and filter out the variables of the last layer:
all_vars = tf.all_variables()
var_to_restore = [v for v in all_vars if not v.name.startswith('xxx')]
saver = tf.train.Saver(var_to_restore)
See the documentation for the details.
Alternatively, you can try to load the whole model an create a new "branch" out of the layer before the last and use it in the cost function during the training.
I am a bit of a beginner with tensorflow so please excuse if this is a stupid question and the answer is obvious.
I have created a Tensorflow graph where starting with placeholders for X and y I have optimized some tensors which represent my model. Part of the graph is something where a vector of predictions can be calculated, e.g. for linear regression something like
y_model = tf.add(tf.mul(X,w),d)
y_vals = sess.run(y_model,feed_dict={....})
After training has been completed I have acceptable values for w and d and now I want to save my model for later. Then, in a different python session I want to restore the model so that I can again run
## Starting brand new python session
import tensorflow as tf
## somehow restor the graph and the values here: how????
## so that I can run this:
y_vals = sess.run(y_model,feed_dict={....})
for some different data and get back the y-values.
I want this to work in a way where the graph for calculating the y-values from the placeholders is also stored and restored - as long as the placeholders get fed the correct data, this should work transparently without the user (the one who applies the model) needing to know what the graph looks like).
As far as I understand tf.train.Saver().save(..) only saves the variables but I also want to save the graph. I think that tf.train.export_meta_graph could be relevant here but I do not understand how to use it correctly, the documentation is a bit cryptic to me and the examples do not even use export_meta_graph anywhere.
From the docs, try this:
# Create some variables.
v1 = tf.Variable(..., name="v1")
v2 = tf.Variable(..., name="v2")
...
# Add an op to initialize the variables.
init_op = tf.global_variables_initializer()
# Add ops to save and restore all the variables.
saver = tf.train.Saver()
# Later, launch the model, initialize the variables, do some work, save the
# variables to disk.
with tf.Session() as sess:
sess.run(init_op)
# Do some work with the model.
..
# Save the variables to disk.
save_path = saver.save(sess, "/tmp/model.ckpt")
print("Model saved in file: %s" % save_path)
You can specify the path.
And if you want to restore the model, try:
with tf.Session() as sess:
saver = tf.train.import_meta_graph('/tmp/model.ckpt.meta')
saver.restore(sess, "/tmp/model.ckpt")
Saving Graph in Tensorflow:
import tensorflow as tf
# Create some placeholder variables
x_pl = tf.placeholder(..., name="x")
y_pl = tf.placeholder(..., name="y")
# Add some operation to the Graph
add_op = tf.add(x, y)
with tf.Session() as sess:
# Add variable initializer
init = tf.global_variables_initializer()
# Add ops to save variables to checkpoints
# Unless var_list is specified Saver will save ALL named variables
# in Graph
# Optionally set maximum of 3 latest models to be saved
saver = tf.train.Saver(max_to_keep=3)
# Run variable initializer
sess.run(init)
for i in range(no_steps):
# Feed placeholders with some data and run operation
sess.run(add_op, feed_dict={x_pl: i+1, y_pl: i+5})
saver.save(sess, "path/to/checkpoint/model.ckpt", global_step=i)
This will save the following files:
1) Meta Graph
.meta file:
MetaGraphDef protocol buffer representation of MetaGraph which saves the complete Tf Graph structure i.e. the GraphDef that describes the dataflow and all metadata associated with it e.g. all variables, operations, collections, etc.
importing the graph structure will recreate the Graph and all its variables, then the corresponding values for these variables can be restored from the checkpoint file
if you don't want to restore the Graph however you can reconstruct all of the information in the MetaGraphDef by re-executing the Python code that builds the model n.b. you must recreate the EXACT SAME variables first before restoring their values from the checkpoint
since Meta Graph file is not always needed, you can switch off writing the file in saver.save using write_meta_graph=False
2) Checkpoint files
.data file:
binary file containing VALUES of all saved variables outlined in tf.train.Saver() (default is all variables)
.index file:
immutable table describing all tensors and their metadata checkpoint file:
keeps a record of latest checkpoint files saved
Restoring Graph in Tensorflow:
import tensorflow as tf
latest_checkpoint = tf.train.latest_checkpoint("path/to/checkpoint")
# Load latest checkpoint Graph via import_meta_graph:
# - construct protocol buffer from file content
# - add all nodes to current graph and recreate collections
# - return Saver
saver = tf.train.import_meta_graph(latest_checkpoint + '.meta')
# Start session
with tf.Session() as sess:
# Restore previously trained variables from disk
print("Restoring Model: {}".format("path/to/checkpoint"))
saver.restore(sess, latest_checkpoint)
# Retrieve protobuf graph definition
graph = tf.get_default_graph()
print("Restored Operations from MetaGraph:")
for op in graph.get_operations():
print(op.name)
# Access restored placeholder variables
x_pl = graph.get_tensor_by_name("x_pl:0")
y_pl = graph.get_tensor_by_name("y_pl:0")
# Access restored operation to re run
accuracy_op = graph.get_tensor_by_name("accuracy_op:0")
This is just a quick example with the basics, for a working implementation see here.
In order to save the graph, you need to freeze the graph.
Here is the python script for freezing the graph : https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py
Here is a code snippet for freezing graph:
from tensorflow.python.tools import freeze_graph
freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
input_binary, checkpoint_path, output_node
restore_op_name, filename_tensor_name,
output_frozen_graph_name, True, "")
where output node corresponds to output tensor variable.
output = tf.nn.softmax(outer_layer_name,name="output")
I am a beginner in TensorFlow, currently training a CNN.
I am using Saver in order to save the parameters used by the model, but I am having concerns whether this would itself store all the Variables used by the model, and is sufficient to restore the values to re-run the program for performing classification/testing on the trained network.
Let us look at the famous example MNIST given by TensorFlow.
In the example, we have bunch of Convolutional blocks, all of which have weight, and bias variables that gets initialised when the program is run.
W_conv1 = init_weight([5,5,1,32])
b_conv1 = init_bias([32])
After having processed several layers, we create a session, and initialise all the variables added to the graph.
sess = tf.Session()
sess.run(tf.initialize_all_variables())
saver = tf.train.Saver()
Here, is it possible to comment the saver.save code, and replace it by saver.restore(sess,file_path) after the training, in order to restore the weight, bias, etc., parameters back to the graph? Is this how it should be ?
for i in range(1000):
...
if i%500 == 0:
saver.save(sess,"model%d.cpkt"%(i))
I am currently training on large dataset, so terminating, and restarting the training is a waste of time, and resources so I request someone to please clarify before the I start the training.
If you want to save the final result only once, you can do this:
with tf.Session() as sess:
for i in range(1000):
...
path = saver.save(sess, "model.ckpt") # out of the loop
print "Saved:", path
In other programs, you can load the model using the path returned from saver.save for prediction or something. You can see some examples at https://github.com/sugyan/tensorflow-mnist.
Based on the explanation in here and Sung Kim solution I wrote a very simple model exactly for this problem. Basically in this way you need to create an object from the same class and restore its variables from the saver. You can find an example of this solution here.