How to add each layer of resnet50 to an empty model? - tensorflow

question description:
I want to transfer resnet50 to a binary classification task. I use keras.applications Resnet50(include_top=false),so it still need last layer(dense layer) which i can set class number.First I use sequential code to combine resnet50 and dense layer. It make the model become only two layers. The first layer is resnet50 (Functional),the second layer is dense (Dense).
The operation hide all the layers in resnet50 and make it become a whole layer. But I need resnet's all layer to generate grad-cam picture. So, I try to add each layer to an empty model and add the last dense layer to save all layers. But I encounter problems.
here is my code:
resnet50 = keras.applications.ResNet50(include_top = False,
pooling = 'avg',
weights = 'imagenet')
resnet50.summary()
for layer in resnet50.layers[0:-10]:
layer.trainable = False
# resnet50_new = keras.models.Sequential([
# resnet50,
# keras.layers.Dense(number_class, activation = 'softmax'),
# ])
model= keras.models.Sequential()
for layer in resnet50.layers:
print(layer)
model.add(tf.keras.layers.Lambada(layer))
model.add(keras.layers.Dense(number_class, activation = 'softmax'))
resnet_new=model
resnet50_new.compile(loss="categorical_crossentropy",
optimizer="sgd", metrics=[tf.keras.metrics.AUC(),'accuracy'])
resnet50_new.summary()
error:
Error happen in add layer,here is error information:
for layer in resnet50.layers:
print(layer)
model.add(tf.keras.layers.Lambada(layer))
ValueError: Exception encountered when calling layer "conv2_block1_add" (type Add).
A merge layer should be called on a list of inputs. Received: inputs=Tensor("Placeholder:0", shape=(None, None, None, 256), dtype=float32) (not a list of tensors)
Call arguments received:
• inputs=tf.Tensor(shape=(None, None, None, 256), dtype=float32)
my understanding:
layer conv2_block1_add is a merging mayer, it can not be added by model.add((layer)) code.
but, i do not know how to manage it.
ps: today,the website can not upload image,so i can not show more pictures details,my question has been closed before,so i post a new one.I hope supervisor will not close my question.I have tried my best to clarify the question.Thank you for your help!

Related

How does Keras determine the weights in the first layer of pre-trained models?

Keras pre-trained models (VGG, ResNet, DenseNet, etc.) have weights established after training on ImageNet with input shape (224, 224, 3). However, Keras allows us to specify any other input shape (width and height should be no smaller than 32). How does Keras determine the initial weights of the first hidden layer when the input shape is other than (224, 224, 3)?
It depends on parameter include_top.
Example:
import tensorflow as tf
model = tf.keras.applications.VGG16(include_top = True, input_shape=(299, 299, 3))
model.summary()
This will throw an error because when you pass include_top = True whole VGG16 architecture will be loaded including Dense layers.
As Dense layers care about the shape, it will throw an error. Because of the operation that Dense layers employ, shapes must be defined and matched with the input shape.
-- Source Code --
Second Example:
import tensorflow as tf
model = tf.keras.applications.VGG16(include_top = False, input_shape=(299, 299, 3))
model.summary()
This time, model only has convolutional layers because include_top = False. Convolutional layers are just sliding filters on the image. So input shape is not a problem for normal convolutions.
When you pass an input_shape, Keras creates an Input layer for that shape. Then creates the model, after that loads the weights.
-- Source Code --
The only constraint here is that, since these models are trained on RGB images, the new images should also have 3 channels.

Multi-channel inputs with Tensorflow Objection Detection API V2

I'd like to construct a network in the Tensorflow V2 object detection API using 5-channel images. However, I am stuck on how to modify the weights of the first convolutional layer using the Tensorflow 2.2 framework.
I have downloaded the pre-trained RetinaNet from the V2 Model Zoo. I then tried the following to modify the weights in the first layer of the checkpoint and save them back:
tf_path = tf.train.latest_checkpoint('./RetinaNet/checkpoint/')
init_vars = tf.train.list_variables(tf_path)
tf_vars = {}
for name, shape in init_vars:
array = tf.train.load_variable(tf_path, name)
try:
if shape[2]==3:#look for a layer who's 3rd input dimension is 3 i.e. the 1st convolutional layer
array=np.concatenate((array,array[:,:,:2,:]),axis=2)
array=array.astype('float32')
tf_vars[name]=tf.Variable(array)
else:
tf_vars[name]=tf.Variable(array)
except:
tf_vars[name]=tf.Variable(array)
saver = tf.compat.v1.train.Saver(var_list=tf_vars)
sess = tf.compat.v1.Session()
saver.save(sess, './RetinaNet/checkpoint/ckpt-0')
I loaded the model back in to make sure the 1st convolutional layer had been changed - all looks ok.
But when I go to train the model, I get the following error:
Model was constructed with shape (None, None, None, 3) for input Tensor("input_1:0", shape=(None, None, None, 3), dtype=float32), but it was called on an input with incompatible shape (64, 128, 128, 5)
Which leads me to believe my method of modifying the weights is not so "ok" after all. Would anyone have some tips on how to modify these weights correctly?
Thanks
This now works but the solution is very hacky... it also means not training from the pretrained weights from the model zoo - so you need to comment everything to do with the fine_tune_checkpoint in the config file.
Then, go to .\Lib\site-packages\official\vision\image_classification\efficientnet and change the number of input channels and number of classes in efficientnet_model.py and efficientnet_config.py.

How to use embedding models in tensorflow hub with LSTM layer?

I'm learning tensorflow 2 working through the text classification with TF hub tutorial. It used an embedding module from TF hub. I was wondering if I could modify the model to include a LSTM layer. Here's what I've tried:
train_data, validation_data, test_data = tfds.load(
name="imdb_reviews",
split=('train[:60%]', 'train[60%:]', 'test'),
as_supervised=True)
embedding = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"
hub_layer = hub.KerasLayer(embedding, input_shape=[],
dtype=tf.string, trainable=True)
model = tf.keras.Sequential()
model.add(hub_layer)
model.add(tf.keras.layers.Embedding(10000, 50))
model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(1))
model.summary()
model.compile(optimizer='adam',
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_data.shuffle(10000).batch(512),
epochs=10,
validation_data=validation_data.batch(512),
verbose=1)
results = model.evaluate(test_data.batch(512), verbose=2)
for name, value in zip(model.metrics_names, results):
print("%s: %.3f" % (name, value))
I don't know how to get the vocabulary size from the hub_layer. So I just put 10000 there. When run it, it throws this exception:
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[480,1] = -6 is not in [0, 10000)
[[node sequential/embedding/embedding_lookup (defined at .../learning/tensorflow/text_classify.py:36) ]] [Op:__inference_train_function_36284]
Errors may have originated from an input operation.
Input Source operations connected to node sequential/embedding/embedding_lookup:
sequential/embedding/embedding_lookup/34017 (defined at Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112)
Function call stack:
train_function
I stuck here. My questions are:
how should I use the embedding module from TF hub to feed an LSTM layer? it looks like embedding lookup has some issues with the setting.
how do I get the vocabulary size from the hub layer?
Thanks
Finally figured out the way to link pre-trained embeddings to LSTM or other layers. Just post the steps here in case anyone feels helpful.
Embedding layer has to be the first layer in the model. (hub_layer is the same as Embedding layer.) The not very intuitive part is that any text input to the hub layer will be converted to only one vector of shape [embedding_dim]. You need to do sentence splitting and tokenization to make sure whatever input to the model is a sequence in the form of array of arrays. e.g., "Let us prepare the data." should be converted to [["let"],["us"],["prepare"], ["the"], ["data"]]. You will also need to pad the sequences if you are using batch mode.
In addition, you will need to convert your target tokens to int if your training labels are strings. The input to the model is array of strings with shape [batch, seq_length], the hub embedding layer converts it to [batch, seq_length, embed_dim]. (If you add a LSTM or other RNN layer, the output from the layer is [batch, seq_length, rnn_units]. ) The output dense layer will output index of text instead of actual text. The index of text is stored in the downloaded tfhub directory as "tokens.txt". You can load the file and convert text to the corresponding index. Otherwise you cannot compute the loss.

Build layers with fixed weights in TensorFlow

I want to build a fully-connected (dense) layer for a regression task. I usually do it with TF2, using Keras API like:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=2, activation='sigmoid', input_shape=(1, )))
model.add(tf.keras.layers.Dense(units=2, activation='linear'))
model.compile(optimizer='adam', loss='mae')
model.fit(inp_data, out_data, epochs=1000)
Now I want to build a custom layer. The layer is composed of, say 10 units, in which 8 units have predefined, fixed, untrainable weights and biases and 2 units have randomly-chosen weights and biases, to be trained by the network. Has anyone any idea how can I define it in Tensorflow?
Keras layers may receive a trainable parameter, True by default, to indicate whether you want them to be trained. Non-trainable layers will just keep the value they are given by the initializer. If I understand correctly, you want to have one layer which is only partially trainable. That is not possible as such with existing layers. Maybe you could do it with a custom layer class, but you can have an equivalent behavior by using two simple layers and then concatenating them (as long as your activation works element-wise, and even it it doesn't, like in a softmax layer, you could apply that activation after the concatenation). This is how it could work:
inputs = tf.keras.Input(shape=(1,))
# This is the trainable part of the layer
layer_train = tf.keras.layers.Dense(units=8, activation='sigmoid')(inputs)
# This is the non-trainable part
layer_const = tf.keras.layers.Dense(units=2, activation='sigmoid', trainable=False)(inputs)
# Merge both parts
layer = tf.keras.layers.Concatenate()([layer_train, layer_const])
# Make model
model = tf.keras.Model(inputs=inputs, outputs=layer)
# ...

TF-slim layers count

Would the code below represent one or two layers? I'm confused because isn't there also supposed to be an input layer in a neural net?
input_layer = slim.fully_connected(input, 6000, activation_fn=tf.nn.relu)
output = slim.fully_connected(input_layer, num_output)
Does that contain a hidden layer? I'm just trying to be able to visualize the net. Thanks in advance!
You have a neural network with one hidden layer. In your code, input corresponds to the 'Input' layer in the above image. input_layer is what the image calls 'Hidden'. output is what the image calls 'Output'.
Remember that the "input layer" of a neural network isn't a traditional fully-connected layer since it's just raw data without an activation. It's a bit of a misnomer. Those neurons in the picture above in the input layer are not the same as the neurons in the hidden layer or output layer.
From tensorflow-slim:
Furthermore, TF-Slim's slim.stack operator allows a caller to repeatedly apply the same operation with different arguments to create a stack or tower of layers. slim.stack also creates a new tf.variable_scope for each operation created. For example, a simple way to create a Multi-Layer Perceptron (MLP):
# Verbose way:
x = slim.fully_connected(x, 32, scope='fc/fc_1')
x = slim.fully_connected(x, 64, scope='fc/fc_2')
x = slim.fully_connected(x, 128, scope='fc/fc_3')
# Equivalent, TF-Slim way using slim.stack:
slim.stack(x, slim.fully_connected, [32, 64, 128], scope='fc')
So the network mentioned here is a [32, 64,128] network - a layer with a hidden size of 64.