I have the sample following model
from tensorflow.keras import models
from tensorflow.keras import layers
sample_model = models.Sequential()
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))
sample_model.compile(loss="binary_crossentropy",
optimizer="adam", metrics = ["accuracy"])
IP for the model:
sam_x = np.random.rand(10,4)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)
The confusion is the fit should have thrown an error of shape mismatch as the expected_input_shape for the 2nd Dense Layer is given as (None,44) but the output for the 1st Dense Layer (which is the input of the 2nd Dense Layer) will be of shape (None,32). But it ran successfully.
I don't understand why there was no error. Any clarifications will be helpful
The input_shape keyword argument has an effect only on the first layer of a Sequential. The shape of the input of the other layers will be derived from their previous layer.
That behaviour is hinted in the doc of tf.keras.layers.InputShape:
When using InputLayer with Keras Sequential model, it can be skipped by moving the input_shape parameter to the first layer after the InputLayer.
And in the Sequential Model guide.
The behaviour can be confirmed by looking at the source of the Sequential.add method:
if not self._layers:
if isinstance(layer, input_layer.InputLayer):
# Case where the user passes an Input or InputLayer layer via `add`.
set_inputs = True
else:
batch_shape, dtype = training_utils.get_input_shape_and_dtype(layer)
if batch_shape:
# Instantiate an input layer.
x = input_layer.Input(
batch_shape=batch_shape, dtype=dtype, name=layer.name + '_input')
# This will build the current layer
# and create the node connecting the current layer
# to the input layer we just created.
layer(x)
set_inputs = True
If there is no layers yet in the model, then an Input will be added to the model with the shape derived from the first layer of the model. This is done only if no layer is present yet in the model.
That shape is either fully known (if input_shape has been passed to the first layer of the model) or will be fully known once the model is built (for example, with a call to model.build(input_shape)).
The thing is after checking the input shape of the model from the first layer, it won't check or deal with other declared input shape inside that same model. For example, if you write your model the following way
sample_model.add(layers.Dense(32, input_shape=(4,)))
sample_model.add(layers.Dense(16, input_shape = (44,)))
sample_model.add(layers.Dense(8, input_shape = (32,)))
The program will always check the first declared input shape layer and discard the rest. So, if you start your first layer with input_shape = (44,), you need to pass exact feature numbers to your model as input such as:
sam_x = np.random.rand(10,44)
sam_y = np.array([0,1,1,0,1,0,0,1,0,1,])
sample_model.fit(sam_x,sam_y)
Additionally, if you look at the Functional API, unlike the Sequential model, you must create and define a standalone Input layer that specifies the shape of input data. It's not learnable but simply a spec layer. It's a kind of gateway of the input data for the model. That means even if we define input_shape inside the other layers, they all will be discarded. For example:
nputs = keras.Input(shape=(4,))
dense = layers.Dense(64, input_shape=(8,)) # dicard input_shape
x = dense(inputs)
x = layers.Dense(64, input_shape=(16,))(x) # dicard input_shape
outputs = layers.Dense(10)(x)
model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
Here is a more complex example with Conv2D and MNIST.
encoder_input = keras.Input(shape=(28, 28, 1),)
x = layers.Conv2D(16, 3, activation="relu", input_shape=[32,32,3])(encoder_input)
x = layers.Conv2D(32, 3, activation="relu", input_shape=[64,64,3])(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu", input_shape=[224,321,3])(x)
x = layers.Conv2D(16, 3, activation="relu", input_shape=[420,32,3])(x)
x = layers.GlobalMaxPooling2D()(x)
out = layers.Dense(10, activation='softmax')(x)
encoder = keras.Model(encoder_input, out, name="encoder")
encoder.summary()
Model: "encoder"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_15 (InputLayer) [(None, 28, 28, 1)] 0
_________________________________________________________________
conv2d_8 (Conv2D) (None, 26, 26, 16) 160
_________________________________________________________________
conv2d_9 (Conv2D) (None, 24, 24, 32) 4640
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 8, 8, 32) 0
_________________________________________________________________
conv2d_10 (Conv2D) (None, 6, 6, 32) 9248
_________________________________________________________________
conv2d_11 (Conv2D) (None, 4, 4, 16) 4624
_________________________________________________________________
global_max_pooling2d_2 (Glob (None, 16) 0
_________________________________________________________________
dense_56 (Dense) (None, 10) 170
=================================================================
Total params: 18,842
Trainable params: 18,842
Non-trainable params: 0
def pre_process(image, label):
return (image / 256)[...,None].astype('float32'),
tf.keras.utils.to_categorical(label, num_classes=10)
(x, y), (_, _) = tf.keras.datasets.mnist.load_data('mnist')
encoder.compile(
loss = tf.keras.losses.CategoricalCrossentropy(),
metrics = tf.keras.metrics.CategoricalAccuracy(),
optimizer = tf.keras.optimizers.Adam())
encoder.fit(x, y, batch_size=256)
4s 14ms/step - loss: 1.4303 - categorical_accuracy: 0.5279
I think Keras will create (or preserves to create) an additional Input Layer - but as the second dense layer is added using model.add() it will automatically be connected to the layer before, and thus the extra input layer stays unconnected and is not part of the model.
(I agree that it would be nice of Keras to hint at unconnected layers, I sometimes created unconnected layers when using the functional API and changed the inputs. Keras doesn't remind me that I had jumped several layers, I just wondered why the summary() was so short...)
Related
I am a newbie to all this so please be kind to this question :)
What I am trying to do is train a Mobilenet classifier using the transfer learning technique and then implement the Gradcam technique to understand what my model is looking into.
I created a model
input_layer = tf.keras.layers.Input(shape=IMG_SHAPE)
x = preprocess_input(input_layer)
y = base_model(x)
y = tf.keras.layers.GlobalAveragePooling2D()(y)
y = tf.keras.layers.Dropout(0.2)(y)
outputs = tf.keras.layers.Dense(5)(y)
model = tf.keras.Model(inputs=input_layer, outputs=outputs)
model.summary()
model summary:
Model: "functional_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
tf_op_layer_RealDiv_1 (Tenso [(None, 224, 224, 3)] 0
_________________________________________________________________
tf_op_layer_Sub_1 (TensorFlo [(None, 224, 224, 3)] 0
_________________________________________________________________
mobilenetv2_1.00_224 (Functi (None, 7, 7, 1280) 2257984
_________________________________________________________________
global_average_pooling2d_1 ( (None, 1280) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 1280) 0
_________________________________________________________________
dense_1 (Dense) (None, 5) 6405
=================================================================
Total params: 2,264,389
Trainable params: 6,405
Non-trainable params: 2,257,984
_________________________________________________________________
passed it to grad cam algorithm but the grad cam algorithm is not able to find the last convolutional layer
Plausible solution:
If instead of having an encapsulated 'mobilenetv2_1.00_224' layer if I can have unwrapped layers of mobilenet added in the model the grad cam algorithm will be able to find that last layer
Problem
I am not able to create the model where I can have data augmentation and pre_processing layer added to mobilenet unwrapped layers.
Thanks in advance
Regards
Ankit
#skruff see if this helps
def make_gradcam_heatmap(img_array, model, last_conv_layer_name, pred_index=None):
# First, we create a model that maps the input image to the activations
# of the last conv layer as well as the output predictions
grad_model = tf.keras.models.Model(
[model.inputs], [model.get_layer(last_conv_layer_name).output, model.output]
)
# Then, we compute the gradient of the top predicted class for our input image
# with respect to the activations of the last conv layer
with tf.GradientTape() as tape:
last_conv_layer_output, preds = grad_model(img_array)
if pred_index is None:
pred_index = tf.argmax(preds[0])
class_channel = preds[:, pred_index]
# This is the gradient of the output neuron (top predicted or chosen)
# with regard to the output feature map of the last conv layer
grads = tape.gradient(class_channel, last_conv_layer_output)
# This is a vector where each entry is the mean intensity of the gradient
# over a specific feature map channel
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))
# We multiply each channel in the feature map array
# by "how important this channel is" with regard to the top predicted class
# then sum all the channels to obtain the heatmap class activation
last_conv_layer_output = last_conv_layer_output[0]
heatmap = last_conv_layer_output # pooled_grads[..., tf.newaxis]
heatmap = tf.squeeze(heatmap)
# For visualization purpose, we will also normalize the heatmap between 0 & 1
heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)
return heatmap.numpy()
I am trying to use grad-CAM (I'm following this https://www.pyimagesearch.com/2020/03/09/grad-cam-visualize-class-activation-maps-with-keras-tensorflow-and-deep-learning/ from PyImageSearch) on a CNN I'm using transfer learning on.
In particular, I am using a simple CNN for a regression problem. I used MobileNetV2 with an Average Pooling layer and a Dense layer with one unit on top, as shown below:
base_model = MobileNetV2(include_top=False, input_shape=(224, 224, 3), weights='imagenet')
base_model.trainable = False
inputs = keras.Input(shape=(224, 224, 3))
x = base_model(inputs)
x = keras.layers.GlobalAveragePooling2D()(x)
outputs = keras.layers.Dense(1, activation="linear")(x)
model = keras.Model(inputs, outputs)
and the summary is:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
mobilenetv2_1.00_224 (Model) (None, 7, 7, 1280) 2257984
_________________________________________________________________
global_average_pooling2d (Gl (None, 1280) 0
_________________________________________________________________
dense (Dense) (None, 1) 1281
=================================================================
Total params: 2,259,265
Trainable params: 1,281
Non-trainable params: 2,257,984
_________________________________________________________________
I initialize the CAM object with:
pred = 0.35
cam = GradCAM(model, pred, layerName='input_2')
where pred is the predicted output on which I want to inspect the CAM and I also specify the layer name in order to refer to the input layer. Then I compute the heatmap on a sample image "img":
heatmap = cam.compute_heatmap(img)
Now, let's focus on a part of the implementation of the function compute_heatmap from PyImageSearch:
# record operations for automatic differentiation
with tf.GradientTape() as tape:
# cast the image tensor to a float-32 data type, pass the
# image through the gradient model, and grab the loss
# associated with the specific class index
inputs = tf.cast(image, tf.float32)
(convOutputs, predictions) = gradModel(inputs)
# loss = predictions[:, self.classIdx] # original from PyImageSearch
loss = predictions[:] # modified by me as I have only 1 output unit
# use automatic differentiation to compute the gradients
grads = tape.gradient(loss, convOutputs)
The problem here is that the gradient grads is None.
I thought that maybe the problem could lie in the network structure (all goes fine when reproducing the example of the classification task from the website), but I can't figure out where is the problem with this network used for regression!
Could you please help me?
import keras
import numpy as numpy
class ActivationLogger(keras.callbacks.Callback):
def set_model(self,model):
self.model = model //inform the callback of what model we will be calling
layer_outputs = [layer.output for layer in model.layers]
self.activations_model = keras.models.Model(model.input,layer_outputs)//returns activation of every layer
def on_epoch_end(self,epoch,logs = None):
if self.validation_data is None:
raise RuntimeError("Requires validation_data")
validation_sample = self.validation_data[0][0:1]
activations = self.activations_model.predict(validation_sample) #computes activation of every epoch
f = open('activations_at_epoch_' + str(epoch) + '.npz', 'w')
np.savez(f, activations)
f.close()
While I was reading this code to create custom callbacks,I couldn't understand few lines of code.I know what are callbacks. What I understood from the above code is that we inherit the super class keras.callbacks.Callback and in the set_model fucntion, we inform the callback of what model it will be calling. I am not able to understand the below line, why does keras.models.Model take model.input?
self.activations_model = keras.models.Model(model.input,
layer_outputs)
and the line activations = self.activations_model.predict(validation_sample)
The further lines just save the numpy arrays to the drive. Also is the callback created,called on every epoch?
Let's say i have an simple model
model = Sequential()
model.add(Dense(32, input_shape=(784, 1), activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(4, activation='softmax'))
cb = ActivationLogger()
cb.set_model(model)
Now let me go through line by line of function set_model:
self.model = model
self.model.summary() = Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 32) 25120
_________________________________________________________________
dense_1 (Dense) (None, 16) 528
_________________________________________________________________
dropout (Dropout) (None, 16) 0
_________________________________________________________________
dense_2 (Dense) (None, 4) 68
=================================================================
Total params: 25,716
Trainable params: 25,716
Non-trainable params: 0
second line:
layer_outputs = [layer.output for layer in model.layers]
print(layer_outputs) = [<tf.Tensor 'dense/Relu:0' shape=(None, 32) dtype=float32>, <tf.Tensor 'dense_1/Relu:0' shape=(None, 16) dtype=float32>, <tf.Tensor 'dropout/cond/Identity:0' shape=(None, 16) dtype=float32>, <tf.Tensor 'dense_2/Softmax:0' shape=(None, 4) dtype=float32>]
layer_outputs contains all the tensors or the layers of the models
and the
third line:
self.activations_model = keras.models.Model(model.input,layer_outputs)
Now in this line, it creates a model with input shape corresponding to original model(model.input = it gives the input tensor or layer of a model. you can also checkout the output shape of a model using model.output)
so self.activation_model is model with one input shape((784, ) in this case) and output at every layer
so when you feed any input through this model it will give you a list of outputs correspond to every layer
Normally output will be a numpy array of shape (none, 4) (taking main Sequential model)
but self.activation will give you a list a numpy arrays. So the line
activations = self.activations_model.predict(validation_sample)
activation just contains the predictions of self.activation_model which a nothing but a list of numpy arrays
[(none, 32)(output of first layer), (None, 16)(output of 2nd), (none, 16)(dropout lyr), (none, 4)(final)
i would suggest you to read about keras Model Function api which is used to make models with many input and outputs
Iam trying to understand the time distributed layer in keras/tensorflow.
As far as I have understood it is a kind of wrapper, making it possible to in example process a sequence of images.
Now Iam wondering how would design a time distributed network without using the time distributed layer.
In example if I would have a sequence of 3 images, each having 1 channel and a pixel dimension of 256x256px, that should first be processed by a CNN and then by LSTM cells.
My input to the time distributed layer would then be (N,3,256,256,1), where N is the batch size.
The CNN would then have 3 outputs, which are fed to the LSTM cell.
Now, without using the time distributed layers, would it be possible to accomplish the same by setting up a network with 3 different inputs and 3 similar CNNs? The outputs of the 3 CNNs could then be flattened and concatenated.
Is that any different from the time distributed approach?
Thanks in advance,
M
I created a prototype for you. I used the least number of layers and arbitrary units/kernels/filters, change them as you like. It creates a cnn model first that takes inputs of size (256,256,1). It uses the same cnn model 3 times (for your three images in the sequence) to extract features. It stacks all the features using Lambda layer to put it back in a sequence. The sequence then goes through LSTM layer. I have chosen for the LSTM to return a single feature vector per example, but if you want the output to be a sequence as well, you could change it to say return_sequences=True. You could also add final additional layers to adapt it to your needs.
from tensorflow.keras.layers import Input, LSTM, Conv2D, Flatten, Lambda
from tensorflow.keras import Model
import tensorflow.keras.backend as K
def create_cnn_model():
inp = Input(shape=(256,256,1))
x = Conv2D(filters=16, kernel_size=5, strides=2)(inp)
x = Flatten()(x)
model = Model(inputs=inp, outputs=x, name='cnn_Model')
return model
def combined_model():
cnn_model = create_cnn_model()
inp_1 = Input(shape=(256,256,1))
inp_2 = Input(shape=(256,256,1))
inp_3 = Input(shape=(256,256,1))
out_1 = cnn_model(inp_1)
out_2 = cnn_model(inp_2)
out_3 = cnn_model(inp_3)
lstm_inp = [out_1, out_2, out_3]
lstm_inp = Lambda(lambda x: K.stack(x, axis=-2))(lstm_inp)
x = LSTM(units=32, return_sequences=False)(lstm_inp)
model = Model(inputs=[inp_1, inp_2, inp_3], outputs=x)
return model
Now create the model as such:
model = combined_model()
Check the summary:
model.summary()
which will print:
Model: "model_14"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_53 (InputLayer) [(None, 256, 256, 1) 0
__________________________________________________________________________________________________
input_54 (InputLayer) [(None, 256, 256, 1) 0
__________________________________________________________________________________________________
input_55 (InputLayer) [(None, 256, 256, 1) 0
__________________________________________________________________________________________________
cnn_Model (Model) (None, 254016) 416 input_53[0][0]
input_54[0][0]
input_55[0][0]
__________________________________________________________________________________________________
lambda_3 (Lambda) (None, 3, 254016) 0 cnn_Model[1][0]
cnn_Model[2][0]
cnn_Model[3][0]
__________________________________________________________________________________________________
lstm_13 (LSTM) (None, 32) 32518272 lambda_3[0][0]
==================================================================================================
Total params: 32,518,688
Trainable params: 32,518,688
Non-trainable params: 0
The inner cnn model summary could be printed:
model.get_layer('cnn_Model').summary()
which currently prints:
Model: "cnn_Model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_52 (InputLayer) [(None, 256, 256, 1)] 0
_________________________________________________________________
conv2d_10 (Conv2D) (None, 126, 126, 16) 416
_________________________________________________________________
flatten_6 (Flatten) (None, 254016) 0
=================================================================
Total params: 416
Trainable params: 416
Non-trainable params: 0
_________________________
Your model expects a list as input. The list should have a length of 3 (since there are 3 images in a sequence). Each element of the list should be a numpy array of shape (batch_size, 256, 256, 1). I have worked a dummy example below with a batch size of 1:
import numpy as np
a = np.zeros((256,256,1)) # first image filled with zeros
b = np.zeros((256,256,1)) # second image filled with zeros
c = np.zeros((256,256,1)) # third image filled with zeros
a = np.expand_dims(a, 0) # adding batch dimension to make it (1, 256, 256, 1)
b = np.expand_dims(b, 0) # same here
c = np.expand_dims(c, 0) # same here
model.compile(loss='mse', optimizer='adam')
# train your model with model.fit(....)
e = model.predict([a,b,c]) # a,b and c have shape of (1, 256, 256, 1) where the first 1 is the batch size
I am trying to develop a siamese network for simple face verification (and recognition in the second stage). I have a network in place that I managed to train but I am a bit puzzled when it comes to how to save and restore the model + making predictions with the trained model. Hoping that maybe an experienced person in the domain can help to make progress..
Here is how I create my siamese network, to begin with...
model = ResNet50(weights='imagenet') # get the original ResNet50 model
model.layers.pop() # Remove the last layer
for layer in model.layers:
layer.trainable = False # do not train any of original layers
x = model.get_layer('flatten_1').output
model_out = Dense(128, activation='relu', name='model_out')(x)
model_out = Lambda(lambda x: K.l2_normalize(x,axis=-1))(model_out)
new_model = Model(inputs=model.input, outputs=model_out)
# At this point, a new layer (with 128 units) added and normalization applied.
# Now create siamese network on top of this
anchor_in = Input(shape=(224, 224, 3))
positive_in = Input(shape=(224, 224, 3))
negative_in = Input(shape=(224, 224, 3))
anchor_out = new_model(anchor_in)
positive_out = new_model(positive_in)
negative_out = new_model(negative_in)
merged_vector = concatenate([anchor_out, positive_out, negative_out], axis=-1)
# Define the trainable model
siamese_model = Model(inputs=[anchor_in, positive_in, negative_in],
outputs=merged_vector)
siamese_model.compile(optimizer=Adam(lr=.0001),
loss=triplet_loss,
metrics=[dist_between_anchor_positive,
dist_between_anchor_negative])
And I train the siamese_model. When I train it, if I interpret results right, it is not really training the underlying model, it just trains the new siamese network (essentially, just the last layer is trained).
But this model has 3 input streams. After the training, I need to save this model in a way so that it just takes 1 or 2 inputs so that I can perform predictions by calculating the distance between 2 given images. How do I save this model and reuse it now?
Thank you in advance!
ADDENDUM:
In case you wonder, here is the summary of siamese model.
siamese_model.summary()
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_2 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
input_3 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
input_4 (InputLayer) (None, 224, 224, 3) 0
__________________________________________________________________________________________________
model_1 (Model) (None, 128) 23849984 input_2[0][0]
input_3[0][0]
input_4[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 384) 0 model_1[1][0]
model_1[2][0]
model_1[3][0]
==================================================================================================
Total params: 23,849,984
Trainable params: 262,272
Non-trainable params: 23,587,712
__________________________________________________________________________________________________
You can use below code to save your model
siamese_model.save_weights(MODEL_WEIGHTS_FILE)
And then to load your model you need to use
siamese_model.load_weights(MODEL_WEIGHTS_FILE)
Thanks