Does it exist a parameter that specifies a tf.Variable as non-trainable, so that the variable is not included in tf.trainable_variables()?
You can mark variables as "non-trainable" on definition:
v = tf.Variable(tf.zeros([1]), trainable=False)
From the linked documentation (circa TensorFlow v0.11):
trainable: If True, the default, also adds the variable to the graph collection GraphKeys.TRAINABLE_VARIABLES. This collection is used as the default list of variables to use by the Optimizer classes.
There are also ways to change this condition with APIs such as tf.get_variable([v]).
You can create non-trainable variables in two different ways:
tf.Variable(a, trainable=False)
tf.get_variable("a", a, trainable=False)
There is no easy way to change the variable from trainable to non-trainable and otherwise. Also there is no easy way to check whether the variable is trainable (you need to check whether the name of your variable is in the list of tf.trainable_variables()
Related
I have a piece of code (not mine) that defines a non-trainable variable that is used to define another property of the layer, which looks something like
initial_weight_val = 1.0
w = my_layer.add_weight(name=layer.name + '/my_weight', shape=(),
initializer=tf.initializers.constant(initial_weight_val),
trainable=False)
# Use w to set another parameter of the layer.
my_layer.the_parameter = some_function(w)
Please, do not tell me what a non-trainable variable is (Of course, I know what it is?), which is also discussed in What is the definition of a non-trainable parameter?.
However, given that w will not be changed (I think), I don't get why someone would define such a variable, rather than simply using the Python variable initial_weight_val directly, especially when using TensorFlow 2.0 (which is my case and the only case I am interested in). Of course, one possibility would be that this variable could become trainable, in case one needs it to be trainable later, but why should one anticipate this, anyway?
Can I safely use initial_weight_val to define the_parameter, i.e. pass initial_weight_val to some_function rather than w?
I am concerned with this issue because I cannot save a model with a variable, because I get the error "variable is not JSON serializable" (Keras and TF are so buggy, btw!), so I was trying to understand the equivalence between user-defined non-trainable variables and Python variables.
You must make sure that this value doesn't change at all, and that it's a single value.
Then yes, you can use a Python var (if a python var is compatible with the function that uses this w).
In which case, you'd put that initial_weight_val both in the __init__ and in the get_config methods of the layer in order for it to be properly saved.
Now, if the function only accepts tensors, but you're still sure that this value will not change at all, then you can on call make w = tf.constant(self.initial_weight_val). You still have the value in __init__ and in get_config as a python var.
Finally, if this value, although non-trainable is changing, or if it's a tensor with many elements, then you'd better let it be a weight. (Non-trainable means "non trainable by backpropagation", but still allowed to be updated here and there).
There should be absolutely no problem for saving loading this weight if you defined it correctly, which should be inside build, with self.add_weight(....), as shown in https://keras.io/layers/writing-your-own-keras-layers/ .
A cool Keras example that uses non-trainable but updatable weights is the BatchNormalization layer. The mean and std of the batches are updated every pass, but not via backpropagation (thus trainable=False)
I'm building a stacked convolutional autoencoder with Tensorflow Core(no API pure Tensorflow). I want to add non trainable layers between encoder and decoder. Does anybody know how to add non trainable layers in tensorflow graph. The tensorboard graph picture is attached, the ops which appears in the blue marked box are the ones that I want to make non trainable, or one can say I do not want gradient computation on them.
TF Version: 1.15
I've tried out tf.stop_gradient() method but this method prevents the contribution of all the input before it. Tensorboard Graph
You have two options:
When you define the weights variable with tf.Variable or tf.get_variable, pass trainable=False. This will stop the variable from being added to the trainable variables collection (accessible through tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)), which is used by default as the list of variables to train by the optimizer.
When you define the optimization step with minimize or compute_gradients, pass a var_list argument with the list of variables that you want to train. The optimizer will then ignore the trainable variables collection and will only affect the listed variables.
In the following code I am unable to understand the need of using tf.Variable? I get the same value whether I use tf.Variable or omit it.
`initial = tf.Variable(tf.truncated_normal(shape=[1,10,1], mean=0,
stddev=0.1,seed=123))`
As I answered in your another post, I will post it again, In tensorflow, anything that is created using tf.Variable(), will get updated during training in back-propagation, for example, a weight matrix.
Ideally, by default, every tf.Variable() becomes trainable unless you specify it non-trainable explicitly.
If you do this initial = tf.truncated_normal([5,10], mean=0, stddev=0.1), then tensorflow will not know that it's a trainable variable and hence it will not be trained. It will stay constant throughout the training.
I am trying to fine tune the last few layers in the tensorflow/slim resnet-v2-50 model for a dataset that I have.
I am struggling to find the names of the layers that I can train. In a tensorflow model, is there a way to find the names of the layers which are train-able? Is there a way to get these names an ordered way so that I can select a few last layers to train? Is there a way to get this information from tensorboard?
Just type
print(tf.trainable_variables())
This will print all the trainable variables.
When you want to train or optimize only certain layers of a pre-trained network, this is what you need to know.
TensorFlow's minimize method takes an optional argument var_list, a list of variables to be adjusted through back-propagation.
If you don't specify var_list, any TF variable in the graph could be adjusted by the optimizer. When you specify some variables in var_list, TF holds all other variables constant.
Here's an example of a script which jonbruner and his collaborator have used.
tvars = tf.trainable_variables()
g_vars = [var for var in tvars if 'g_' in var.name]
g_trainer = tf.train.AdamOptimizer(0.0001).minimize(g_loss, var_list=g_vars)
This finds all the variables they defined earlier that have "g_" in the variable name, puts them into a list, and runs the ADAM optimizer on them.
You can find the related answers here on Quora
I created a trainable variable in a scope. Later, I entered the same scope, set the scope to reuse_variables, and used get_variable to retrieve the same variable. However, I cannot set the variable's trainable property to False. My get_variable line is like:
weight_var = tf.get_variable('weights', trainable = False)
But the variable 'weights' is still in the output of tf.trainable_variables.
Can I set a shared variable's trainable flag to False by using get_variable?
The reason I want to do this is that I'm trying to reuse the low-level filters pre-trained from VGG net in my model, and I want to build the graph like before, retrieve the weights variable, and assign VGG filter values to the weight variable, and then keep them fixed during the following training step.
After looking at the documentation and the code, I was not able to find a way to remove a Variable from the TRAINABLE_VARIABLES.
Here is what happens:
The first time tf.get_variable('weights', trainable=True) is called, the variable is added to the list of TRAINABLE_VARIABLES.
The second time you call tf.get_variable('weights', trainable=False), you get the same variable but the argument trainable=False has no effect as the variable is already present in the list of TRAINABLE_VARIABLES (and there is no way to remove it from there)
First solution
When calling the minimize method of the optimizer (see doc.), you can pass a var_list=[...] as argument with the variables you want to optimizer.
For instance, if you want to freeze all the layers of VGG except the last two, you can pass the weights of the last two layers in var_list.
Second solution
You can use a tf.train.Saver() to save variables and restore them later (see this tutorial).
First you train your entire VGG model with all trainable variables. You save them in a checkpoint file by calling saver.save(sess, "/path/to/dir/model.ckpt").
Then (in another file) you train the second version with non trainable variables. You load the variables previously stored with saver.restore(sess, "/path/to/dir/model.ckpt").
Optionally, you can decide to save only some of the variables in your checkpoint file. See the doc for more info.
When you want to train or optimize only certain layers of a pre-trained network, this is what you need to know.
TensorFlow's minimize method takes an optional argument var_list, a list of variables to be adjusted through back-propagation.
If you don't specify var_list, any TF variable in the graph could be adjusted by the optimizer. When you specify some variables in var_list, TF holds all other variables constant.
Here's an example of a script which jonbruner and his collaborator have used.
tvars = tf.trainable_variables()
g_vars = [var for var in tvars if 'g_' in var.name]
g_trainer = tf.train.AdamOptimizer(0.0001).minimize(g_loss, var_list=g_vars)
This finds all the variables they defined earlier that have "g_" in the variable name, puts them into a list, and runs the ADAM optimizer on them.
You can find the related answers here on Quora
In order to remove a variable from the list of trainable variables, you can first access the collection through:
trainable_collection = tf.get_collection_ref(tf.GraphKeys.TRAINABLE_VARIABLES)
There, trainable_collection contains a reference to the collection of trainable variables. If you pop elements from this list, doing for example trainable_collection.pop(0), you are going to remove the corresponding variable from the trainable variables, and thus this variable will not be trained.
Although this works with pop, I am still struggling to find a way to correctly use remove with the correct argument, so we don't depend on the index of the variables.
EDIT: Given that you have the name of the variables in the graph (you can obtain that by inspecting the graph protobuf or, what is easier, using Tensorboard), you can use it to loop through the list of trainable variables and then remove the variables from the trainable collection.
Example: say that I want the variables with names "batch_normalization/gamma:0" and "batch_normalization/beta:0" NOT to be trained, but they are already added to the TRAINABLE_VARIABLES collection. What I can do is:
`
#gets a reference to the list containing the trainable variables
trainable_collection = tf.get_collection_ref(tf.GraphKeys.TRAINABLE_VARIABLES)
variables_to_remove = list()
for vari in trainable_collection:
#uses the attribute 'name' of the variable
if vari.name=="batch_normalization/gamma:0" or vari.name=="batch_normalization/beta:0":
variables_to_remove.append(vari)
for rem in variables_to_remove:
trainable_collection.remove(rem)
`
This will successfully remove the two variables from the collection, and they will not be trained anymore.
You can use tf.get_collection_ref to get the reference of collection rather than tf.get_collection