Tensorflow: Convert constant tensor from pre-trained Vgg model to variable - tensorflow

My question is how can I convert a constant tensor loaded from a pre-trained Vgg16 model to a tf.Variable tensor? The motivation is that I need to compute the gradient of a specific loss with respect to the Conv4_3 layers' kernel, however, the kernel were seems set to a tf.Constant type and it is not accepted by tf.Optimizer.compute_gradients method.
F = vgg.graph.get_tensor_by_name('pretrained_vgg16/conv4_3/filter:0')
G = optimizer.compute_gradients(losses, var_list=[F])
# TypeError: Argument is not a tf.Variable: Tensor("pretrained_vgg16/conv4_3/filter:0", shape=(3, 3, 512, 512), dtype=float32)
What I have tried is to use tf.assign method to update the kernel to a variable type tensor with initial value set to be the original kernel, but it gives a TypeError: Input 'ref' of 'Assign' Op requires l-value input
F = tf.assign(F, tf.Variable(F, trainable=False))
So, how can I achieve that? Many thanks in advance!
Update: I download the pretrained model according to Pretrained Vgg16 Tensorflow model and then I loaded the model by:
with open('vgg16.tfmodel', mode='rb') as f:
fileContent = f.read()
graph_def = tf.GraphDef()
graph_def.ParseFromString(fileContent)
# Map input tensor
inputs = tf.placeholder("float", [1, 224, 224, 3], name='inputs')
tf.import_graph_def(graph_def, input_map={ "images": inputs }, name='pretrained_vgg16')
graph = tf.get_default_graph()
All the code above is defined in a class named vgg.

The reason why you did not get variables from the pre-trained model could be explained in this answer. Briefly, tf.import_graph_def just restore the structure of a graph, without the variables.
A solution to this is to build the model yourself, with same variable name to the pre-trained model. Then load pre-trained model and assign every variable with specific parameter.
I recommend this vgg model.

Related

Multi-channel inputs with Tensorflow Objection Detection API V2

I'd like to construct a network in the Tensorflow V2 object detection API using 5-channel images. However, I am stuck on how to modify the weights of the first convolutional layer using the Tensorflow 2.2 framework.
I have downloaded the pre-trained RetinaNet from the V2 Model Zoo. I then tried the following to modify the weights in the first layer of the checkpoint and save them back:
tf_path = tf.train.latest_checkpoint('./RetinaNet/checkpoint/')
init_vars = tf.train.list_variables(tf_path)
tf_vars = {}
for name, shape in init_vars:
array = tf.train.load_variable(tf_path, name)
try:
if shape[2]==3:#look for a layer who's 3rd input dimension is 3 i.e. the 1st convolutional layer
array=np.concatenate((array,array[:,:,:2,:]),axis=2)
array=array.astype('float32')
tf_vars[name]=tf.Variable(array)
else:
tf_vars[name]=tf.Variable(array)
except:
tf_vars[name]=tf.Variable(array)
saver = tf.compat.v1.train.Saver(var_list=tf_vars)
sess = tf.compat.v1.Session()
saver.save(sess, './RetinaNet/checkpoint/ckpt-0')
I loaded the model back in to make sure the 1st convolutional layer had been changed - all looks ok.
But when I go to train the model, I get the following error:
Model was constructed with shape (None, None, None, 3) for input Tensor("input_1:0", shape=(None, None, None, 3), dtype=float32), but it was called on an input with incompatible shape (64, 128, 128, 5)
Which leads me to believe my method of modifying the weights is not so "ok" after all. Would anyone have some tips on how to modify these weights correctly?
Thanks
This now works but the solution is very hacky... it also means not training from the pretrained weights from the model zoo - so you need to comment everything to do with the fine_tune_checkpoint in the config file.
Then, go to .\Lib\site-packages\official\vision\image_classification\efficientnet and change the number of input channels and number of classes in efficientnet_model.py and efficientnet_config.py.

Issue with feeding value into placeholder tensor for sess.run()

I want to get the value of an intermediate tensor in a convolutional neural network for a specific input. I know how to do this in keras and even though I have trained a model using keras, I'm going to move towards constructing and training the model using only tensorflow. Therefore, I want to move away from something like K.function(input_layer, output_layer) which is fairly simple, and instead use tensorflow. I believe I should use placeholder values, like the following approach:
with tf.compat.v1.Session(graph=tf.Graph()) as sess:
loaded_model = tf.keras.models.load_model(filepath)
graph = tf.compat.v1.get_default_graph()
images = tf.compat.v1.placeholder(tf.float32, shape=(None, 28, 28, 1)) # To specify input at MNIST images
output_tensor = graph.get_tensor_by_name(tensor_name) # tensor_name is 'dense_1/MatMul:0'
output = sess.run([output_tensor], feed_dict={images: x_test[0:1]}) # x_test[0:1] is of shape (1, 28, 28, 1)
print(output)
However, I get the following error message for the sess.run() line: Invalid argument: You must feed a value for placeholder tensor 'conv2d_2_input' with dtype float and shape [?,28,28,1]. I am unsure why I get this message because the image used for feed_dict is of type float and is what I believe to be the correct shape. Any help would be suggested.
You must use the input tensor from the Keras model, not make your own new placeholder, which would be disconnected from the rest of the model:
with tf.Graph().as_default(), tf.compat.v1.Session() as sess:
# Load model
loaded_model = tf.keras.models.load_model(filepath)
# Take model input tensor
images = loaded_model.input
# Take output of the second layer (index 1)
output_tensor = loaded_model.layers[1].output
# Evaluate
output = sess.run(output_tensor, feed_dict={images: x_test[0:1]})
print(output)

Saving model weights in Keras: what is model weights?

I created a deep learning model for image recognition by Keras, and I saved the model weights by model.save_weights('weights.h5'). Also, I loaded it and used the weights again.
I know that model.save_weights() saves the model weights. My question is what is the model weights? Is it the filters weights?
Model weights are all the parameters (including trainable and non-trainable) of the model which are in turn all the parameters used in the layers of the model. And yes, for a convolution layer that would be the filter weights as well as the biases.
Actually, you can see them for each layer: try model.layers[layer_index].get_weights() and you would get the weights of that layer. When you call save_weights() actually it is the output of get_weights() called on each of the layers that is stored in the file.
For example for a convolution layer, get_weights() method would return a list with two elements which corresponds to filter weights and the biases. Here is an example:
model = Sequential()
model.add(Conv2D(5, (3,3), input_shape=(100, 100, 3)))
filters, biases = model.layers[0].get_weights()
>>> filters.shape
(3, 3, 3, 5) <--- 5 filters of shape (3, 3, 3)
>>> biases.shape
(5,) <--- one bias parameter for each filter
>>> filters[:, :, :, 0] # get the first filter's weights
array([[[-0.26788074, -0.20213448, 0.06233829],
[ 0.08651951, 0.21303588, 0.08127764],
[ 0.04672694, -0.24589485, -0.12873489]],
[[ 0.10841686, 0.24839908, -0.07466605],
[-0.26903206, -0.0341135 , 0.15083215],
[-0.07382561, -0.00576964, -0.25354072]],
[[-0.02937067, 0.22315139, -0.12964793],
[ 0.23371089, 0.19973844, -0.00728002],
[-0.2748396 , -0.02097657, 0.22772402]]], dtype=float32)

How do I resolve an InvalidArgumentError in Classifier model?

New to TensorFlow, so apologies for newbie question.
Following this tutorial but instead of using image data I am using numerical data.
Load the dataset:
train_dataset_url = "xxx.csv"
train_dataset_fp = tf.keras.utils.get_file(
fname=os.path.basename(train_dataset_url),
origin=train_dataset_url)
Make training dataset:
batch_size = 32
train_dataset = tf.contrib.data.make_csv_dataset(
train_dataset_fp,
batch_size,
column_names=column_names,
label_name=label_name,
num_epochs=1)
Train classified model using:
model = tf.keras.Sequential([
tf.keras.layers.Dense(10, activation=tf.nn.relu, input_shape=(1,)),
tf.keras.layers.Dense(10, activation=tf.nn.relu),
tf.keras.layers.Dense(4)
])
But when I "test" the model with the same inputs:
predictions = model(features)
I receive the error:
InvalidArgumentError: cannot compute MatMul as input #0(zero-based) was expected to be a float tensor but is a int32 tensor [Op:MatMul]
It's possible I have missed something fundamental. I feel like I need to specify a type somewhere.
The data which you feed in the model is a numpy array according to my assumption . The error states that the model requires a tensor with dtype=float32 or float64. You are providing a int32 numpy array. So, wherever you create a numpy array, just mention the dtype as float32.

Keras: difference of InputLayer and Input

I made a model using Keras with Tensorflow. I use Inputlayer with these lines of code:
img1 = tf.placeholder(tf.float32, shape=(None, img_width, img_heigh, img_ch))
first_input = InputLayer(input_tensor=img1, input_shape=(img_width, img_heigh, img_ch))
first_dense = Conv2D(16, 3, 3, activation='relu', border_mode='same', name='1st_conv1')(first_input)
But I get this error:
ValueError: Layer 1st_conv1 was called with an input that isn't a symbolic tensor. Received type: <class 'keras.engine.topology.InputLayer'>. Full input: [<keras.engine.topology.InputLayer object at 0x00000000112170F0>]. All inputs to the layer should be tensors.
When I use Input like this, it works fine:
first_input = Input(tensor=img1, shape=(224, 224, 3), name='1st_input')
first_dense = Conv2D(16, 3, 3, activation='relu', border_mode='same', name='1st_conv1')(first_input)
What is the difference between Inputlayer and Input?
InputLayer is a layer.
Input is a tensor.
You can only call layers passing tensors to them.
The idea is:
outputTensor = SomeLayer(inputTensor)
So, only Input can be passed because it's a tensor.
Honestly, I have no idea about the reason for the existence of InputLayer. Maybe it's supposed to be used internally. I never used it, and it seems I'll never need it.
According to tensorflow website, "It is generally recommend to use the functional layer API via Input, (which creates an InputLayer) without directly using InputLayer."
Know more at this page here
Input: Used for creating a functional model
inp=tf.keras.Input(shape=[?,?,?])
x=layers.Conv2D(.....)(inp)
Input Layer: used for creating a sequential model
x=tf.keras.Sequential()
x.add(tf.keras.layers.InputLayer(shape=[?,?,?]))
And the other difference is that
When using InputLayer with the Keras Sequential model, it can be skipped by moving the input_shape parameter to the first layer after the InputLayer.
That is in sequential model you can skip the InputLayer and specify the shape directly in the first layer.
i.e From this
model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=(4,)),
tf.keras.layers.Dense(8)])
To this
model = tf.keras.Sequential([
tf.keras.layers.Dense(8, input_shape=(4,))])
To define it in simple words:
keras.layers.Input is used to instantiate a Keras Tensor. In this case, your data is probably not a tf tensor, maybe an np array.
On the other hand, keras.layers.InputLayer is a layer where your data is already defined as one of the tf tensor types, i.e., can be a ragged tensor or constant or other types.
I hope this helps!