I have a U-net network with VGG16 encoder architecture with pre-trained imagenet weights. Since my input images are grayscale, I added in a convolutional layer with depth 3 prior to sending the input to the U-net model.
Now, I'm trying to get the output of an intermediate layer within the U-net network. I create an intermediate model whose output is the output of the layer that I'm interested in. Here is my code:
base_model = sm.Unet('vgg16', encoder_weights='imagenet', classes=1, activation='sigmoid')
inp = Input(shape=(448, 224, 1))
l1 = Conv2D(3, (1,1))(inp)
out = base_model(l1)
model = Model(inp, out)
model.summary()
intermediate_layer_model = Model(inputs=model.layers[0].input,
outputs=model.get_layer('model_1').get_layer('center_block2_relu').output)
Here is the output:
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) (None, 448, 224, 1) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 448, 224, 3) 6
_________________________________________________________________
model_1 (Model) multiple 23752273
=================================================================
Total params: 23,752,279
Trainable params: 23,748,247
Non-trainable params: 4,032
_________________________________________________________________
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(?, ?, ?, 3), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
It seems to me that there is an issue with the U-net model having an input layer (input_1) and I'm not supplying this information during the construction of intermediate_layer_model. However, I expect that the intermediate model to take only the grayscale images as input and not require an additional 3-channel input.
Any help would be appreciated.
Related
I am a newbie to all this so please be kind to this question :)
What I am trying to do is train a Mobilenet classifier using the transfer learning technique and then implement the Gradcam technique to understand what my model is looking into.
I created a model
input_layer = tf.keras.layers.Input(shape=IMG_SHAPE)
x = preprocess_input(input_layer)
y = base_model(x)
y = tf.keras.layers.GlobalAveragePooling2D()(y)
y = tf.keras.layers.Dropout(0.2)(y)
outputs = tf.keras.layers.Dense(5)(y)
model = tf.keras.Model(inputs=input_layer, outputs=outputs)
model.summary()
model summary:
Model: "functional_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
tf_op_layer_RealDiv_1 (Tenso [(None, 224, 224, 3)] 0
_________________________________________________________________
tf_op_layer_Sub_1 (TensorFlo [(None, 224, 224, 3)] 0
_________________________________________________________________
mobilenetv2_1.00_224 (Functi (None, 7, 7, 1280) 2257984
_________________________________________________________________
global_average_pooling2d_1 ( (None, 1280) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 1280) 0
_________________________________________________________________
dense_1 (Dense) (None, 5) 6405
=================================================================
Total params: 2,264,389
Trainable params: 6,405
Non-trainable params: 2,257,984
_________________________________________________________________
passed it to grad cam algorithm but the grad cam algorithm is not able to find the last convolutional layer
Plausible solution:
If instead of having an encapsulated 'mobilenetv2_1.00_224' layer if I can have unwrapped layers of mobilenet added in the model the grad cam algorithm will be able to find that last layer
Problem
I am not able to create the model where I can have data augmentation and pre_processing layer added to mobilenet unwrapped layers.
Thanks in advance
Regards
Ankit
#skruff see if this helps
def make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=None):
# First, we create a model that maps the input image to the activations
# of the last conv layer as well as the output predictions
grad_model = tf.keras.models.Model(
[model.inputs], [model.get_layer(last_conv_layer_name).output, model.output]
)
# Then, we compute the gradient of the top predicted class for our input image
# with respect to the activations of the last conv layer
with tf.GradientTape() as tape:
last_conv_layer_output, preds = grad_model(img_array)
if pred_index is None:
pred_index = tf.argmax(preds[0])
class_channel = preds[:, pred_index]
# This is the gradient of the output neuron (top predicted or chosen)
# with regard to the output feature map of the last conv layer
grads = tape.gradient(class_channel, last_conv_layer_output)
# This is a vector where each entry is the mean intensity of the gradient
# over a specific feature map channel
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))
# We multiply each channel in the feature map array
# by "how important this channel is" with regard to the top predicted class
# then sum all the channels to obtain the heatmap class activation
last_conv_layer_output = last_conv_layer_output[0]
heatmap = last_conv_layer_output # pooled_grads[..., tf.newaxis]
heatmap = tf.squeeze(heatmap)
# For visualization purpose, we will also normalize the heatmap between 0 & 1
heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)
return heatmap.numpy()
I'm using tensorflow 2.6 keras for transfer learning. Currently I take MobileNetV2. I take input, apply some preprocessing using Lambda layer, then feed this preprocessed input to MobileNetV2, then add Dense layer and train this thing. Training, inference etc actually work as expected.
However, the summary of the model looks as follows:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input (InputLayer) [(None, 201, 189, 1)] 0
_________________________________________________________________
lambda (Lambda) (None, 201, 189, None) 0
_________________________________________________________________
lambda_1 (Lambda) (None, 201, 189, None) 0
_________________________________________________________________
mobilenetv2_1.00_224 (Functi (None, 7, 6, 1280) 2257984
_________________________________________________________________
flatten (Flatten) (None, 53760) 0
_________________________________________________________________
output (Dense) (None, 2) 107522
=================================================================
Total params: 2,365,506
Trainable params: 2,331,394
Non-trainable params: 34,112
So the MobileNetV2 structure is hidden and shown as one layer of type tensorflow.python.keras.engine.functional.Functional. If I print summary of this layer, I get all the internal layers of the model. I have a script for automatic GradCam visualizations which looks for the last Conv layer of the model. If the model is constructed by hand using Lambda, Conv2D, Dense layers, then everyhting works fine. If I use pretrained model, then currently it fails, because the Conv layer is hidden inside of this Functional layer.
How do I construct my modified MobileNetV2 model with my additional layers so that the full structure of the model is shown?
This is how I approximately construct my final model:
input = Input(shape=params.image_shape, name="input")
flow = input
flow = input_correction(flow, params) #some Lambda layers
keras_model = MobileNetV2(
input_shape=image_shape,
weights='imagenet',
include_top=False)
keras_model_output=keras_model(flow)
keras_model_input=input
keras_model_output = Flatten()(keras_model_output)
output = Dense(units=len(params.classes),
activation=tf.nn.softmax,
name="output")(keras_model_output)
model = Model(inputs=keras_model_input, outputs=output)
model.compile(...)
In default, summary doesnt show nested models. Just include expand_nested argument in the summary.
model.summary(expand_nested=True)
I am trying to use grad-CAM (I'm following this https://www.pyimagesearch.com/2020/03/09/grad-cam-visualize-class-activation-maps-with-keras-tensorflow-and-deep-learning/ from PyImageSearch) on a CNN I'm using transfer learning on.
In particular, I am using a simple CNN for a regression problem. I used MobileNetV2 with an Average Pooling layer and a Dense layer with one unit on top, as shown below:
base_model = MobileNetV2(include_top=False, input_shape=(224, 224, 3), weights='imagenet')
base_model.trainable = False
inputs = keras.Input(shape=(224, 224, 3))
x = base_model(inputs)
x = keras.layers.GlobalAveragePooling2D()(x)
outputs = keras.layers.Dense(1, activation="linear")(x)
model = keras.Model(inputs, outputs)
and the summary is:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
mobilenetv2_1.00_224 (Model) (None, 7, 7, 1280) 2257984
_________________________________________________________________
global_average_pooling2d (Gl (None, 1280) 0
_________________________________________________________________
dense (Dense) (None, 1) 1281
=================================================================
Total params: 2,259,265
Trainable params: 1,281
Non-trainable params: 2,257,984
_________________________________________________________________
I initialize the CAM object with:
pred = 0.35
cam = GradCAM(model, pred, layerName='input_2')
where pred is the predicted output on which I want to inspect the CAM and I also specify the layer name in order to refer to the input layer. Then I compute the heatmap on a sample image "img":
heatmap = cam.compute_heatmap(img)
Now, let's focus on a part of the implementation of the function compute_heatmap from PyImageSearch:
# record operations for automatic differentiation
with tf.GradientTape() as tape:
# cast the image tensor to a float-32 data type, pass the
# image through the gradient model, and grab the loss
# associated with the specific class index
inputs = tf.cast(image, tf.float32)
(convOutputs, predictions) = gradModel(inputs)
# loss = predictions[:, self.classIdx] # original from PyImageSearch
loss = predictions[:] # modified by me as I have only 1 output unit
# use automatic differentiation to compute the gradients
grads = tape.gradient(loss, convOutputs)
The problem here is that the gradient grads is None.
I thought that maybe the problem could lie in the network structure (all goes fine when reproducing the example of the classification task from the website), but I can't figure out where is the problem with this network used for regression!
Could you please help me?
i want to save my trained model to .h5 without last two layers, in order to transfer learning using my custom model in the furture, just like MobileNetV2 include_top=False, can someone help me, thanks!
base_model = tf.keras.applications.mobilenet_v2.MobileNetV2(
alpha=1.0,
input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
model = tf.keras.Sequential([
base_model,
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(255, activation=tf.nn.softmax)
])
trained model like this:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
mobilenetv2_1.00_224 (Model) (None, 2, 2, 1280) 2257984
_________________________________________________________________
global_average_pooling2d (Gl (None, 1280) 0
_________________________________________________________________
dense (Dense) (None, 205) 262605
=================================================================
Total params: 2,520,589
Trainable params: 2,486,477
Non-trainable params: 34,112
_________________________________________________________________
when i try to using it for transfer learning
keras_model = loadModel(keras_model_path)
keras_model.summary()
input = keras_model.input
hidden = tf.keras.layers.GlobalMaxPooling2D()(keras_model.layers[-3].output)
out = tf.keras.layers.Dense(128, activation=tf.nn.softmax)(hidden)
model2 = tf.keras.Model(input, out)
model2.summary()
an error occurs
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(?, 64, 64, 3), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
i want to save my trained model to .h5 without last two layers,
why don't you save the full model with model.save() and when you reload it for transfer learning, just remove the layers using:
model.layers.pop()
You can also remove the layers before saving the model but I wouldn't do that
I am trying to develop a siamese network for simple face verification (and recognition in the second stage). I have a network in place that I managed to train but I am a bit puzzled when it comes to how to save and restore the model + making predictions with the trained model. Hoping that maybe an experienced person in the domain can help to make progress..
Here is how I create my siamese network, to begin with...
model = ResNet50(weights='imagenet') # get the original ResNet50 model
model.layers.pop() # Remove the last layer
for layer in model.layers:
layer.trainable = False # do not train any of original layers
x = model.get_layer('flatten_1').output
model_out = Dense(128, activation='relu', name='model_out')(x)
model_out = Lambda(lambda x: K.l2_normalize(x,axis=-1))(model_out)
new_model = Model(inputs=model.input, outputs=model_out)
# At this point, a new layer (with 128 units) added and normalization applied.
# Now create siamese network on top of this
anchor_in = Input(shape=(224, 224, 3))
positive_in = Input(shape=(224, 224, 3))
negative_in = Input(shape=(224, 224, 3))
anchor_out = new_model(anchor_in)
positive_out = new_model(positive_in)
negative_out = new_model(negative_in)
merged_vector = concatenate([anchor_out, positive_out, negative_out], axis=-1)
# Define the trainable model
siamese_model = Model(inputs=[anchor_in, positive_in, negative_in],
outputs=merged_vector)
siamese_model.compile(optimizer=Adam(lr=.0001),
loss=triplet_loss,
metrics=[dist_between_anchor_positive,
dist_between_anchor_negative])
And I train the siamese_model. When I train it, if I interpret results right, it is not really training the underlying model, it just trains the new siamese network (essentially, just the last layer is trained).
But this model has 3 input streams. After the training, I need to save this model in a way so that it just takes 1 or 2 inputs so that I can perform predictions by calculating the distance between 2 given images. How do I save this model and reuse it now?
Thank you in advance!
ADDENDUM:
In case you wonder, here is the summary of siamese model.
siamese_model.summary()
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_2 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
input_3 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
input_4 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
model_1 (Model) (None, 128) 23849984 input_2[0][0]
input_3[0][0]
input_4[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 384) 0 model_1[1][0]
model_1[2][0]
model_1[3][0]
==================================================================================================
Total params: 23,849,984
Trainable params: 262,272
Non-trainable params: 23,587,712
__________________________________________________________________________________________________
You can use below code to save your model
siamese_model.save_weights(MODEL_WEIGHTS_FILE)
And then to load your model you need to use
siamese_model.load_weights(MODEL_WEIGHTS_FILE)
Thanks