Related
I'm training a U-net type model with a minor variation in the architecture which is the Atrous Spatial Pyramid pooling (ASPP) layer at the bottleneck after the encoder. I profiled the model during one forward pass and used tensorboard to check the tracer_view to see which part of the model has the highest latency.
Profiler Tracer View with ASPP layer
This revealed that there's a lot of idle GPU time at ASPP computation. I double checked it by removing the ASPP layer and the just connected the encoder to the decoder. In this experiment, the idle time that was previously there disappeared.
Profiler Tracer View without ASPP layer
I understand that the second model example would be a bit smaller than the former.
This is how my model looks like with ASPP layer. And to I just commented those ASPP layers out to profile the model without ASPP layers.
With ASPP
def get_custom_deeplab(image_size: tuple, num_classes: int):
"""
This model uses a vanilla CNN backbone. This model also uses upsampling2d in place of conv2d transpose
"""
input_layer = keras.Input(shape=(image_size[0], image_size[1], 3))
conv1 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(input_layer)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(128, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(pool1)
conv2 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(128, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(pool2)
conv3 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = Conv2D(128, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(pool3)
conv4 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
#######ASPP layers
out_1 = Conv2D(256, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), dilation_rate=1, padding='same')(pool4)
out_6 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), dilation_rate=6, padding='same')(pool4)
out_12 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), dilation_rate=10, padding='same')(pool4)
out_14 = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), dilation_rate=14, padding='same')(pool4)
x = layers.Concatenate(axis=-1)([out_1, out_6, out_12, out_14])
########ASPP's output
x = Conv2D(256, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), dilation_rate=1, padding='same')(x)
x = layers.UpSampling2D(
(2,2),interpolation="bilinear",
)(x)
skip_connection_1 = pool3
x = layers.Concatenate(axis=-1)([x,skip_connection_1])
x = Conv2D(128, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(x)
x = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(x)
x = layers.UpSampling2D(
(2,2),interpolation="bilinear",
)(x)
skip_connection_2 = pool2
x = layers.Concatenate(axis=-1)([x,skip_connection_2])
x = Conv2D(128, (1, 1), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(x)
x = Conv2D(256, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(x)
x = layers.UpSampling2D(
(2,2),interpolation="bilinear",
)(x)
x = Conv2D(64, (3, 3), activation='relu', kernel_initializer='lecun_uniform', kernel_constraint=max_norm(3), padding='same')(x)
x = layers.UpSampling2D(
(2,2),interpolation="bilinear",
)(x)
x = Conv2D(
num_classes,
kernel_size=1,
padding="same",
use_bias=True,
kernel_initializer=keras.initializers.HeNormal(),
)(x)
return tf.keras.Model(inputs=input_layer,outputs=x)
But, I would like to know if there's any workaround to mitigate the problem of GPU idle time when the model has layers like ASPP?
model = Sequential()
model.add(Conv2D(128, (3, 3), activation='relu', input_shape=(64, 64, 3), padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128, activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(5, activation = 'softmax'))
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=[tf.keras.metrics.Recall()])
This code works fine for metrics=['accuracy']), but it shows ValueError: Shapes (None, 1) and (None, 5) are incompatible for metrics=[tf.keras.metrics.Recall()])
Please help me. Thanks in advance.
Recall makes sense only for binary classification. Your final layer has 5 nodes, which essentially means you have 5 classes. You should change recall to another metric. Documentation should help you choose an appropriate metric for your model. Categorical accuracy should be good enough to get started.
I was building a simple CNN for a binary image classification task and found something confusing regarding the use of the BatchNormalization layer in Tensorflow. There are 320 images in the training set with evenly divided negative and positive cases. There are 80 images in the validation set with evenly divided negative and positive cases. I set the batch_size for both training and validation set to 32.
Here is the architecture of my original model.
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dropout, Dense
model = Sequential([
Input(shape=(256, 256, 3]),
Conv2D(64, (3, 3), padding=“same”, activation=“relu”),
Conv2D(64, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(128, (3, 3), padding=“same”, activation=“relu”),
Conv2D(128, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Flatten(),
Dense(128, activation=“relu”),
Dropout(0.5),
Dense(64, activation=“relu”),
Dense(1, activation=“sigmoid”)
])
Both training and validation accuracies increased from somewhere between 0.5 and 0.6 and eventually converged somewhere between 0.95, which is good. However, with everything else unchanged, if I introduced BatchNormalization layers between Conv2D layers like the following:
model = Sequential([
Input(shape=(256, 256, 3]),
Conv2D(64, (3, 3), padding=“same”),
BatchNormalization(),
ReLU(),
Conv2D(64, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(128, (3, 3), padding=“same”),
BatchNormalization(),
ReLU(),
Conv2D(128, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(256, (3, 3), padding=“same”),
BatchNormalization(),
ReLU(),
Conv2D(256, (3, 3), padding=“same”),
BatchNormalization(),
ReLU(),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Flatten(),
Dense(128, activation=“relu”),
Dropout(0.5),
Dense(64, activation=“relu”),
Dense(1, activation=“sigmoid”)
])
The training accuracy still behaved the same way as before but validation accuracy remained constant at 0.5. I also tried several other implementations of BatchNormalization layers and had the same observation.
model = Sequential([
Input(shape=(256, 256, 3]),
Conv2D(64, (3, 3), padding=“same”, activation=“relu”),
BatchNormalization(),
Conv2D(64, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(128, (3, 3), padding=“same”, activation=“relu”),
BatchNormalization(),
Conv2D(128, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
BatchNormalization(),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
BatchNormalization(),
Conv2D(256, (3, 3), padding=“same”, activation=“relu”),
MaxPooling2D((2, 2)),
Flatten(),
Dense(128, activation=“relu”),
Dropout(0.5),
Dense(64, activation=“relu”),
Dense(1, activation=“sigmoid”)
])
I am rather new with deep learning and from what I have read recently on different tutorials about BatchNormalization, it should usually (though not always) be a performance booster. However, from what I have observed, BatchNormalization was ruining my validation performance. There are people online saying that there are bugs in Keras’s implementation of BatchNormalization. I had tried to reduce the layers of BatchNormalization and found that with even a single layer of BatchNormalization introduced, the validation accuracy will be constant at 0.5. Do you have any ideas why?
I have this model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(64, (3, 3), padding='same',
input_shape=(96, 96, 1),
kernel_regularizer = tf.keras.regularizers.l2(0.01)),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(128, (5, 5), padding='same'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(512, (3, 3), padding='same'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(512, (3, 3), padding='same'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1024),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1024),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.PReLU(alpha_initializer='zeros'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(7, activation='softmax')
])
model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-7),
loss="categorical_crossentropy",
metrics =['accuracy'])
the model works, trained well and everything, but when I try to predict I get a weird shape, as you can see in the image below
why do I have a none, my shape is (96,96,1), how do I fix this?
There is nothing to fix, the None in the batch dimension implies that this dimension is variable sized. This makes sense since the model can be trained using any batch size.
I'm pretty new to maching learning and when I was looking at a tutorial for a convolutional neural network I wanted to experiment on my own on how to increase accuracy. However, when I tried to add another convolutional and pooling layer to my model it displayed an error message. This is before I added the layer:
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(62))
And this is after:
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(62))
This is the error message it gave me:
ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node conv2d_36/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true](max_pooling2d_26/MaxPool, conv2d_36/Conv2D/ReadVariableOp)' with input shapes: [?,1,1,64], [3,3,64,64]. site:stackoverflow.com
This is because you reduce the dimensionality too much inside your network. use padding='same' in your convolutional layer to avoid this dimensionality error
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', padding='same',
input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(62))
model.summary()