I'd like to construct a network in the Tensorflow V2 object detection API using 5-channel images. However, I am stuck on how to modify the weights of the first convolutional layer using the Tensorflow 2.2 framework.
I have downloaded the pre-trained RetinaNet from the V2 Model Zoo. I then tried the following to modify the weights in the first layer of the checkpoint and save them back:
tf_path = tf.train.latest_checkpoint('./RetinaNet/checkpoint/')
init_vars = tf.train.list_variables(tf_path)
tf_vars = {}
for name, shape in init_vars:
array = tf.train.load_variable(tf_path, name)
try:
if shape[2]==3:#look for a layer who's 3rd input dimension is 3 i.e. the 1st convolutional layer
array=np.concatenate((array,array[:,:,:2,:]),axis=2)
array=array.astype('float32')
tf_vars[name]=tf.Variable(array)
else:
tf_vars[name]=tf.Variable(array)
except:
tf_vars[name]=tf.Variable(array)
saver = tf.compat.v1.train.Saver(var_list=tf_vars)
sess = tf.compat.v1.Session()
saver.save(sess, './RetinaNet/checkpoint/ckpt-0')
I loaded the model back in to make sure the 1st convolutional layer had been changed - all looks ok.
But when I go to train the model, I get the following error:
Model was constructed with shape (None, None, None, 3) for input Tensor("input_1:0", shape=(None, None, None, 3), dtype=float32), but it was called on an input with incompatible shape (64, 128, 128, 5)
Which leads me to believe my method of modifying the weights is not so "ok" after all. Would anyone have some tips on how to modify these weights correctly?
Thanks
This now works but the solution is very hacky... it also means not training from the pretrained weights from the model zoo - so you need to comment everything to do with the fine_tune_checkpoint in the config file.
Then, go to .\Lib\site-packages\official\vision\image_classification\efficientnet and change the number of input channels and number of classes in efficientnet_model.py and efficientnet_config.py.
Related
question description:
I want to transfer resnet50 to a binary classification task. I use keras.applications Resnet50(include_top=false),so it still need last layer(dense layer) which i can set class number.First I use sequential code to combine resnet50 and dense layer. It make the model become only two layers. The first layer is resnet50 (Functional),the second layer is dense (Dense).
The operation hide all the layers in resnet50 and make it become a whole layer. But I need resnet's all layer to generate grad-cam picture. So, I try to add each layer to an empty model and add the last dense layer to save all layers. But I encounter problems.
here is my code:
resnet50 = keras.applications.ResNet50(include_top = False,
pooling = 'avg',
weights = 'imagenet')
resnet50.summary()
for layer in resnet50.layers[0:-10]:
layer.trainable = False
# resnet50_new = keras.models.Sequential([
# resnet50,
# keras.layers.Dense(number_class, activation = 'softmax'),
# ])
model= keras.models.Sequential()
for layer in resnet50.layers:
print(layer)
model.add(tf.keras.layers.Lambada(layer))
model.add(keras.layers.Dense(number_class, activation = 'softmax'))
resnet_new=model
resnet50_new.compile(loss="categorical_crossentropy",
optimizer="sgd", metrics=[tf.keras.metrics.AUC(),'accuracy'])
resnet50_new.summary()
error:
Error happen in add layer，here is error information:
for layer in resnet50.layers:
print(layer)
model.add(tf.keras.layers.Lambada(layer))
ValueError: Exception encountered when calling layer "conv2_block1_add" (type Add).
A merge layer should be called on a list of inputs. Received: inputs=Tensor("Placeholder:0", shape=(None, None, None, 256), dtype=float32) (not a list of tensors)
Call arguments received:
• inputs=tf.Tensor(shape=(None, None, None, 256), dtype=float32)
my understanding:
layer conv2_block1_add is a merging mayer, it can not be added by model.add((layer)) code.
but, i do not know how to manage it.
ps: today,the website can not upload image,so i can not show more pictures details,my question has been closed before,so i post a new one.I hope supervisor will not close my question.I have tried my best to clarify the question.Thank you for your help!
I want to get the value of an intermediate tensor in a convolutional neural network for a specific input. I know how to do this in keras and even though I have trained a model using keras, I'm going to move towards constructing and training the model using only tensorflow. Therefore, I want to move away from something like K.function(input_layer, output_layer) which is fairly simple, and instead use tensorflow. I believe I should use placeholder values, like the following approach:
with tf.compat.v1.Session(graph=tf.Graph()) as sess:
loaded_model = tf.keras.models.load_model(filepath)
graph = tf.compat.v1.get_default_graph()
images = tf.compat.v1.placeholder(tf.float32, shape=(None, 28, 28, 1)) # To specify input at MNIST images
output_tensor = graph.get_tensor_by_name(tensor_name) # tensor_name is 'dense_1/MatMul:0'
output = sess.run([output_tensor], feed_dict={images: x_test[0:1]}) # x_test[0:1] is of shape (1, 28, 28, 1)
print(output)
However, I get the following error message for the sess.run() line: Invalid argument: You must feed a value for placeholder tensor 'conv2d_2_input' with dtype float and shape [?,28,28,1]. I am unsure why I get this message because the image used for feed_dict is of type float and is what I believe to be the correct shape. Any help would be suggested.
You must use the input tensor from the Keras model, not make your own new placeholder, which would be disconnected from the rest of the model:
with tf.Graph().as_default(), tf.compat.v1.Session() as sess:
# Load model
loaded_model = tf.keras.models.load_model(filepath)
# Take model input tensor
images = loaded_model.input
# Take output of the second layer (index 1)
output_tensor = loaded_model.layers[1].output
# Evaluate
output = sess.run(output_tensor, feed_dict={images: x_test[0:1]})
print(output)
I'm learning tensorflow 2 working through the text classification with TF hub tutorial. It used an embedding module from TF hub. I was wondering if I could modify the model to include a LSTM layer. Here's what I've tried:
train_data, validation_data, test_data = tfds.load(
name="imdb_reviews",
split=('train[:60%]', 'train[60%:]', 'test'),
as_supervised=True)
embedding = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"
hub_layer = hub.KerasLayer(embedding, input_shape=[],
dtype=tf.string, trainable=True)
model = tf.keras.Sequential()
model.add(hub_layer)
model.add(tf.keras.layers.Embedding(10000, 50))
model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(1))
model.summary()
model.compile(optimizer='adam',
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_data.shuffle(10000).batch(512),
epochs=10,
validation_data=validation_data.batch(512),
verbose=1)
results = model.evaluate(test_data.batch(512), verbose=2)
for name, value in zip(model.metrics_names, results):
print("%s: %.3f" % (name, value))
I don't know how to get the vocabulary size from the hub_layer. So I just put 10000 there. When run it, it throws this exception:
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[480,1] = -6 is not in [0, 10000)
[[node sequential/embedding/embedding_lookup (defined at .../learning/tensorflow/text_classify.py:36) ]] [Op:__inference_train_function_36284]
Errors may have originated from an input operation.
Input Source operations connected to node sequential/embedding/embedding_lookup:
sequential/embedding/embedding_lookup/34017 (defined at Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:112)
Function call stack:
train_function
I stuck here. My questions are:
how should I use the embedding module from TF hub to feed an LSTM layer? it looks like embedding lookup has some issues with the setting.
how do I get the vocabulary size from the hub layer?
Thanks
Finally figured out the way to link pre-trained embeddings to LSTM or other layers. Just post the steps here in case anyone feels helpful.
Embedding layer has to be the first layer in the model. (hub_layer is the same as Embedding layer.) The not very intuitive part is that any text input to the hub layer will be converted to only one vector of shape [embedding_dim]. You need to do sentence splitting and tokenization to make sure whatever input to the model is a sequence in the form of array of arrays. e.g., "Let us prepare the data." should be converted to [["let"],["us"],["prepare"], ["the"], ["data"]]. You will also need to pad the sequences if you are using batch mode.
In addition, you will need to convert your target tokens to int if your training labels are strings. The input to the model is array of strings with shape [batch, seq_length], the hub embedding layer converts it to [batch, seq_length, embed_dim]. (If you add a LSTM or other RNN layer, the output from the layer is [batch, seq_length, rnn_units]. ) The output dense layer will output index of text instead of actual text. The index of text is stored in the downloaded tfhub directory as "tokens.txt". You can load the file and convert text to the corresponding index. Otherwise you cannot compute the loss.
I want to use a pre-trained BagNet (https://github.com/wielandbrendel/bag-of-local-features-models) to extract features. The net has fixed input height and width, the input is (None,3,224,224). Now, I want to build a new model with fexible input sizes. I tried approaches with model.layers.pop()[0] to remove the first layer and replace it with a flexible input but I get errors:
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input0_6:0", shape=(?, 3, 224, 224), dtype=float32) at layer "input0". The following previous layers were accessed without issue: []
keras_model = bagnets.keras.bagnet8()
keras_model.layers.pop()[0]
x = Input(batch_shape=(None, 3, None, None))
newModel = Model(x, keras_model.output)
How could I ressolve this issue or what are other options?
My question is how can I convert a constant tensor loaded from a pre-trained Vgg16 model to a tf.Variable tensor? The motivation is that I need to compute the gradient of a specific loss with respect to the Conv4_3 layers' kernel, however, the kernel were seems set to a tf.Constant type and it is not accepted by tf.Optimizer.compute_gradients method.
F = vgg.graph.get_tensor_by_name('pretrained_vgg16/conv4_3/filter:0')
G = optimizer.compute_gradients(losses, var_list=[F])
# TypeError: Argument is not a tf.Variable: Tensor("pretrained_vgg16/conv4_3/filter:0", shape=(3, 3, 512, 512), dtype=float32)
What I have tried is to use tf.assign method to update the kernel to a variable type tensor with initial value set to be the original kernel, but it gives a TypeError: Input 'ref' of 'Assign' Op requires l-value input
F = tf.assign(F, tf.Variable(F, trainable=False))
So, how can I achieve that? Many thanks in advance!
Update: I download the pretrained model according to Pretrained Vgg16 Tensorflow model and then I loaded the model by:
with open('vgg16.tfmodel', mode='rb') as f:
fileContent = f.read()
graph_def = tf.GraphDef()
graph_def.ParseFromString(fileContent)
# Map input tensor
inputs = tf.placeholder("float", [1, 224, 224, 3], name='inputs')
tf.import_graph_def(graph_def, input_map={ "images": inputs }, name='pretrained_vgg16')
graph = tf.get_default_graph()
All the code above is defined in a class named vgg.
The reason why you did not get variables from the pre-trained model could be explained in this answer. Briefly, tf.import_graph_def just restore the structure of a graph, without the variables.
A solution to this is to build the model yourself, with same variable name to the pre-trained model. Then load pre-trained model and assign every variable with specific parameter.
I recommend this vgg model.