Related
I am using a modified U-Net architecture to perform auto-segmentation on a dataset of biomedical images. Although I have achieved some decent results, I have noticed that the training of the model is highly dependent on the seed that I set at the beginning. Using the same seeds and performing multiple runs of my code with those seeds results in very repeatable results. However, with the exact same code, hyperparameters, and training/test set images, the results vary wildly with different seeds. My loss function is the dice coefficient loss (the primary outcome metric that I am concerned with is the dice coefficient) and with some seeds my loss will level off at about 0.95 and only go down around 0.01 over the course of many many epochs and with other seeds my loss won't start to level off until about 0.10. The only difference is the seed. Since the total range for dice coefficient loss is only 0-1 these values represent extremely different results.
As additional information, this phenomenon has occurred for different training set sizes ranging from a few hundred images to a few thousand images. I have double checked and do not believe there to be an issues with my data. Additionally, my dataset is highly unbalanced (only about 3% of my pixels are the region I am trying to segment).
Things I have already tried:
Using alternative loss functions such as binary cross entropy, focal loss, tversky loss, and combined binary cross entropy and dice loss
Adjusting hyperparameters: learning rate (I am using adam optimizer), batch size, filter sizes, model depth
Trying different kernel initializers
Different activations functions (relu vs leaky relu)
Gradient clipping
Batch normalization
Dropout
Any suggestions of how I can solve this issue would be greatly appreciated. This issue has stalled my progress significantly and as I add to my training set the issue seems to exacerbated further by causing me to have to test quite a few seed options before finding one that allows my model to train correctly.
Below is my code starting after I import my images and modules, crop the images and masks, and put them into arrays:
Please note that in my full code setting the seeds and hyperparameters goes at the top.
from numpy.random import seed
seed(3)
from tensorflow import set_random_seed
set_random_seed(4)
# Define Parameters
batch_size = 16
batch_size_test = 1
filter_size = 8
kernel_dimension = 5
learning_rate = 1e-4
num_epochs = 25
# these are functions for pairing the image to its respective mask
def get_dataset(images, mask, batch_size):
dataset_input = tf.data.Dataset.from_tensor_slices(tf.constant(images, dtype=tf.float32)) #converts to tf type
dataset_mask = tf.data.Dataset.from_tensor_slices(tf.constant(mask, dtype=tf.float32)) #converts to tf type
dataset_input = dataset_input.map(lambda x: tf.image.per_image_standardization(x)) #standardizes the image
dataset_input = dataset_input.map(lambda x: tf.image.adjust_contrast(x,1.2)) #adds some contrast
dataset = tf.data.Dataset.zip((dataset_input, dataset_mask)) #pairs the images to the masks into one tf array
dataset = dataset.shuffle(len(images)).repeat() #randomly shuffles dataset and repeats the dataset
dataset = dataset.batch(batch_size).prefetch(batch_size) # set the batch size
print('image shape: ', dataset.output_shapes[0])
print('label shape: ', dataset.output_shapes[1])
print('types: ', dataset.output_types)
print()
print(dataset)
return dataset
def get_dataset_noshuffle(images, mask, batch_size):
dataset_input = tf.data.Dataset.from_tensor_slices(tf.constant(images, dtype=tf.float32))
dataset_mask = tf.data.Dataset.from_tensor_slices(tf.constant(mask, dtype=tf.float32))
dataset_input = dataset_input.map(lambda x: tf.image.per_image_standardization(x))
dataset_input = dataset_input.map(lambda x: tf.image.adjust_contrast(x,1.2))
dataset = tf.data.Dataset.zip((dataset_input, dataset_mask))
dataset = dataset.batch(batch_size).prefetch(batch_size)
print('image shape: ', dataset.output_shapes[0])
print('label shape: ', dataset.output_shapes[1])
print('types: ', dataset.output_types)
print()
print(dataset)
return dataset
X_train, X_test, y_train, y_test = train_test_split(images, mask, test_size=0.0001, random_state=42)
X_test = testimages # if you want to use a separate set of images that you imported earlier then use this
y_test = testmask # and this
# use the get_dataset function to pair the X_train with y_train and X_test with y_test. adjust batch size as needed
train_dataset = get_dataset(X_train, y_train, batch_size)
test_dataset = get_dataset(X_test, y_test, batch_size_test)
test_dataset_noshuffle = get_dataset_noshuffle(X_test, y_test, batch_size_test)
def dice_coef(y_true, y_pred):
smooth = 1.
y_true_f = tf.keras.backend.flatten(y_true)
y_pred_f = tf.keras.backend.flatten(y_pred)
intersection = tf.keras.backend.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (tf.keras.backend.sum(y_true_f) + tf.keras.backend.sum(y_pred_f) + smooth)
def dice_coef_loss(y_true, y_pred):
return 1. - dice_coef(y_true, y_pred)
def unet(pretrained_weights = None,input_size = (size,size,1), df=filter_size, kernel_size = kernel_dimension):
inputs = Input(input_size)
conv1 = Conv2D(df, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
conv1 = BatchNormalization()(conv1)
conv1 = Conv2D(df, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
conv1 = BatchNormalization()(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(df*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
conv2 = BatchNormalization()(conv2)
conv2 = Conv2D(df*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
conv2 = BatchNormalization()(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
conv3 = Conv2D(df*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
conv3 = BatchNormalization()(conv3)
conv3 = Conv2D(df*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
conv3 = BatchNormalization()(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
conv4 = Conv2D(df*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
conv4 = BatchNormalization()(conv4)
conv4 = Conv2D(df*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
conv4 = BatchNormalization()(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
conv5 = Conv2D(df*2*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)
conv5 = BatchNormalization()(conv5)
conv5 = Conv2D(df*2*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)
conv5 = BatchNormalization()(conv5)
up6 = Conv2D(df*2*2*2, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv5))
merge6 = concatenate([conv4,up6], axis = 3)
conv6 = Conv2D(df*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
conv6 = BatchNormalization()(conv6)
conv6 = Conv2D(df*2*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)
conv6 = BatchNormalization()(conv6)
up7 = Conv2D(df*2*2, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))
merge7 = concatenate([conv3,up7], axis = 3)
conv7 = Conv2D(df*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
conv7 = BatchNormalization()(conv7)
conv7 = Conv2D(df*2*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)
conv7 = BatchNormalization()(conv7)
up8 = Conv2D(df*2, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))
merge8 = concatenate([conv2,up8], axis = 3)
conv8 = Conv2D(df*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
conv8 = BatchNormalization()(conv8)
conv8 = Conv2D(df*2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)
conv8 = BatchNormalization()(conv8)
up9 = Conv2D(df, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))
merge9 = concatenate([conv1,up9], axis = 3)
conv9 = Conv2D(df, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)
conv9 = BatchNormalization()(conv9)
conv9 = Conv2D(df, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
conv9 = BatchNormalization()(conv9)
conv9 = Conv2D(2, kernel_size, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)
conv10 = Conv2D(1, (1,1), activation = 'sigmoid')(conv9)
model = Model(inputs, conv10)
model.compile(optimizer = Adam(lr = learning_rate), loss = dice_coef_loss, metrics = ['accuracy','binary_accuracy', 'mae',
tf.keras.metrics.Precision(name='precision'), dice_coef, jacard_coef,
tf.keras.metrics.FalseNegatives(thresholds=0.5, name='FN', dtype=None),
tf.keras.metrics.FalsePositives(thresholds=0.5, name='FP', dtype=None),
tf.keras.metrics.TrueNegatives(thresholds=0.5, name='TN', dtype=None),
tf.keras.metrics.TruePositives(thresholds=0.5, name='TP', dtype=None)])
model.summary() #if you want to printout all the parameters and model summary
if(pretrained_weights):
model.load_weights(pretrained_weights)
return model
model = unet(pretrained_weights=None, df=filter_size, input_size=(size, size, 1), kernel_size=kernel_dimension)
steps_epoch = np.int16(np.ceil(len(X_train)/batch_size)) # determines your steps per epoch
steps_val = np.int16(np.ceil(len(X_test)/batch_size_test)) # determines your steps for the test set
model_checkpoint = ModelCheckpoint(weights_name, monitor='val_loss',verbose=1, save_best_only=True)
#This is the actual training part of the code
history = model.fit(train_dataset, validation_data=test_dataset, steps_per_epoch=steps_epoch,
validation_steps=steps_val, batch_size=batch_size, epochs=num_epochs, verbose=1, callbacks=[model_checkpoint])
This problem is sometimes noticed when you do not have enough training data. Get more training data. If you cannot get additional data, you can increase the data through augmentation techniques. Generally with enough data points the model converges to a minima - if not stuck in local or saddle. Another approach is to use a pre-trained model and fine tune on it. As I see you are initializing the model from scratch -
model = unet(pretrained_weights=None, df=filter_size, input_size=(size, size, 1), kernel_size=kernel_dimension)
I have created an model (down below). And after training, I want to get the output tensor from the bottleneck layers of this model.
So I am trying to create a model of the extracted layers and use this model for predicting.
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.models import Sequential
from keras.layers import Dense, Activation
nstrides = (1,1)
inputs = layers.Input(imshape)
conv01 = layers.Conv2D(32, 4, activation = 'relu',
strides = nstrides, padding="same")(inputs)
conv1 = layers.Conv2D(32, 4, activation = 'relu',
strides = nstrides, padding="same")(conv01)
pool1 = layers.MaxPooling2D(pool_size=(2, 2))(conv1)
.
.
#block4
conv04 = layers.Conv2D(256, 4, activation = 'relu',
strides = nstrides, padding="same")(pool3)
conv4 = layers.Conv2D(256, 4, activation = 'relu',
strides = nstrides, padding="same")(conv04)
pool4 = layers.MaxPooling2D(pool_size=(2, 2))(conv4)
#bottlneck
conv05 = layers.Conv2D(512, 4, activation = 'relu',
strides = nstrides, padding="same")(pool4)
conv5 = layers.Conv2D(512, 4, activation = 'relu',
strides = nstrides, padding="same")(conv05)
upconv5 = layers.Conv2DTranspose(256, kernel_size=(2, 2),
strides = (2,2))(conv5)
#upblock 1
conc6 = layers.concatenate([upconv5, conv4])
conv06 = layers.Conv2D(256, 4, activation = 'relu',
strides = nstrides, padding="same")(conc6)
conv6 = layers.Conv2D(256, 4, activation = 'relu',
strides = nstrides, padding="same")(conv06)
up7 = layers.Conv2DTranspose(126, kernel_size=(2, 2),
strides = (2,2))(conv7)
.
.
.
#combine the model together
model = Model(inputs, outputs)
First, in order to locate the desired layer which would be the new output tensor, you can first do
for i, layer in enumerate(model.layers):
print(i, layer.name)
...
...
12 max_pooling2d_15
13 conv2d_65
14 conv2d_66
15 conv2d_transpose_12
16 concatenate_12
17 conv2d_67
...
...
Here, the layer index from 13 to 15 is from the bottleneck layer of your model. If you want to get the output tensor from this bottleneck layer, you can do:
new_model = Model(model.input,
model.get_layer(index=15).output)
# or,
new_model = Model(model.input,
model.get_layer(name='conv2d_transpose_12').output)
Both are the same, the first one is by index and the second one is by layer name.
I am working on a hand character recognition model. I created a CNN+BiLSTM+CTC Loss model. But getting error when I run model.fit(). Please help me fix this error.
My Model
# input with shape of height=32 and width=128
inputs = Input(shape=(32,128,1))
# convolution layer with kernel size (3,3)
conv_1 = Conv2D(64, (3,3), activation = 'relu', padding='same')(inputs)
# poolig layer with kernel size (2,2)
pool_1 = MaxPooling2D(pool_size=(2, 2), strides=2)(conv_1)
conv_2 = Conv2D(128, (3,3), activation = 'relu', padding='same')(pool_1)
pool_2 = MaxPooling2D(pool_size=(2, 2), strides=2)(conv_2)
conv_3 = Conv2D(256, (3,3), activation = 'relu', padding='same')(pool_2)
conv_4 = Conv2D(256, (3,3), activation = 'relu', padding='same')(conv_3)
# poolig layer with kernel size (2,1)
pool_4 = MaxPooling2D(pool_size=(2, 1))(conv_4)
conv_5 = Conv2D(512, (3,3), activation = 'relu', padding='same')(pool_4)
# Batch normalization layer
batch_norm_5 = BatchNormalization()(conv_5)
conv_6 = Conv2D(512, (3,3), activation = 'relu', padding='same')(batch_norm_5)
batch_norm_6 = BatchNormalization()(conv_6)
pool_6 = MaxPooling2D(pool_size=(2, 1))(batch_norm_6)
conv_7 = Conv2D(512, (2,2), activation = 'relu')(pool_6)
squeezed = Lambda(lambda x: K.squeeze(x, 1))(conv_7)
# bidirectional LSTM layers with units=128
blstm_1 = Bidirectional(LSTM(128, return_sequences=True, dropout = 0.2))(squeezed)
blstm_2 = Bidirectional(LSTM(128, return_sequences=True, dropout = 0.2))(blstm_1)
outputs = Dense(len(char_dict)+1, activation = 'softmax')(blstm_2)
act_model = Model(inputs, outputs)
Define a CTC loss model that takes the outputs of previous model as inputs
labels = Input(name='the_labels', shape=[max_length], dtype='float32')
input_length = Input(name='input_length', shape=[1], dtype='int64')
label_length = Input(name='label_length', shape=[1], dtype='int64')
def ctc_lambda_func(args):
y_pred, labels, input_length, label_length = args
return K.ctc_batch_cost(labels, y_pred, input_length, label_length)
loss_out = Lambda(ctc_lambda_func, output_shape=(1,), name='ctc')([outputs, labels, input_length,
label_length])
model = Model(inputs=[inputs, labels, input_length, label_length], outputs=loss_out)
model.compile(loss={'ctc': lambda y_true, y_pred: y_pred}, optimizer = 'adam')
model.fit(x=[input_array,
output_array,
train_input_length,
train_label_length],
y=np.zeros(input_array.shape[0]),
batch_size=256,
epochs = 100,
validation_data = ([test_input_array, test_output_array, valid_input_length,
valid_label_length], [np.zeros(test_input_array.shape[0])]),
verbose = 1,
callbacks = callbacks_list)
The error I am getting is
ValueError: Shape (None, 17) must have rank 1
Below is the code i am using for training some gestures. the directory for training data is as follows
'E:\build\set_1\training\palm\seq_01','E:\build\set_1\training\palm\seq_02' and so on.
The error i am follwong is on the last lines. I have tried both of the two lines as provided but they are giving error as Invalid Argument error. I am running this code on jupyter notebook.
import tensorflow as tf
from tensorflow import keras
from keras_preprocessing.image import ImageDataGenerator
path = 'E:\build\set_1\training'
training_datagen = ImageDataGenerator(rescale = 1./255)
TRAINING_DIR = 'E:/build/set_1/training/'
train_generator = training_datagen.flow_from_directory(
TRAINING_DIR,
target_size = (150,150),
class_mode= 'categorical',
batch_size=64
)
VALIDATION_DIR = "E:/build/set_1/test/"
validation_datagen = ImageDataGenerator(rescale = 1./255)
validation_generator = training_datagen.flow_from_directory(
VALIDATION_DIR,
target_size=(150,150),
class_mode='categorical',
batch_size=64
)
model = tf.keras.models.Sequential([
# Note the input shape is the desired size of the image 150x150 with 3 bytes color
# This is the first convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(150, 150, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
# The second convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The third convolution
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The fourth convolution
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# Flatten the results to feed into a DNN
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
# 512 neuron hidden layer
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(3, activation='softmax')
])
model.summary()
model.compile(loss='categorical_crossentropy',optimizer = 'rmsprop',
metrics= ['accuracy'])
history = model.fit_generator(train_generator,steps_per_epoch = train_generator.samples//train_generator.batch_size,epochs = 30,validation_data = validation_generator,validation_steps=validation_generator.samples//validation_generator.batch_size)
history = model.fit(train_generator, epochs=25, validation_data = validation_generator, verbose = 1)
I am working on the protein analysis project. We receive the images* of proteins with 4 filters (Red, Green, Blue and Yellow). Every of those RGBY channels contains unique data as different cellular structures are visible with different filters.
The idea is to use a pre-trained network e.g. VGG19 and extend the number of channels from default 3 to 4. Something like this:
(My appologies, I am not allowed to add images directly before 10 reputation, please press the "Run code snippet" button to visualize):
<img src="https://i.stack.imgur.com/TZKka.png" alt="Italian Trulli">
Picture: VGG model with RGB extended to RGBY
The Y channel should be the copy of the existing pretrained channel. Then it is possible to make use of the pretrained weights.
Does anyone have an idea of how such extension of a pretrained network can be achieved?
*
Author of the collage - Allunia from Kaggle, "Protein Atlas - Exploration and Baseline" kernel.
Use the layer.get_weights() and layer.set_weights() functions of Keras api.
Create a template structure for 4-layers VGG (set input shape=(width, height, 4)). Then load the weights from 3-channel RGB model into 4-channel as RGBB.
Below is the code that does the procedure. In case of sequential VGG, the only layer that needs to be modified is the first Convolution layer. The structure of the subsequent layers is independent on the number of channels.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from keras.applications.vgg19 import VGG19
from keras.models import Model
vgg19 = VGG19(weights='imagenet')
vgg19.summary() # To check which layers will be omitted in 'pretrained' model
# Load part of the VGG without the top layers into 'pretrained' model
pretrained = Model(inputs=vgg19.input, outputs=vgg19.get_layer('block5_pool').output)
pretrained.summary()
#%% Prepare model template with 4 input channels
config = pretrained.get_config() # run config['layers'][i] for reference
# to restore layer-by layer structure
from keras.layers import Input, Conv2D, MaxPooling2D
from keras import optimizers
# For training from scratch change kernel_initializer to e.g.'VarianceScaling'
inputs = Input(shape=(224, 224, 4), name='input_17')
# block 1
x = Conv2D(64, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block1_conv1')(inputs)
x = Conv2D(64, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block1_conv2')(x)
x = MaxPooling2D(pool_size=(2, 2), name='block1_pool')(x)
# block 2
x = Conv2D(128, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block2_conv1')(x)
x = Conv2D(128, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block2_conv2')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2,2), name='block2_pool')(x)
# block 3
x = Conv2D(256, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block3_conv1')(x)
x = Conv2D(256, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block3_conv2')(x)
x = Conv2D(256, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block3_conv3')(x)
x = Conv2D(256, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block3_conv4')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2,2), name='block3_pool')(x)
# block 4
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block4_conv1')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block4_conv2')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block4_conv3')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block4_conv4')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2,2), name='block4_pool')(x)
# block 5
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block5_conv1')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block5_conv2')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block5_conv3')(x)
x = Conv2D(512, (3,3), padding='same', activation='relu', kernel_initializer='zeros', name='block5_conv4')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2,2), name='block5_pool')(x)
vgg_template = Model(inputs=inputs, outputs=x)
vgg_template.compile(optimizer=optimizers.RMSprop(lr=2e-4),
loss='categorical_crossentropy',
metrics=['acc'])
#%% Rewrite the weight loading/modification function
import numpy as np
layers_to_modify = ['block1_conv1'] # Turns out the only layer that changes
# shape due to 4th channel is the first
# convolution layer.
for layer in pretrained.layers: # pretrained Model and template have the same
# layers, so it doesn't matter which to
# iterate over.
if layer.get_weights() != []: # Skip input, pooling and no weights layers
target_layer = vgg_template.get_layer(name=layer.name)
if layer.name in layers_to_modify:
kernels = layer.get_weights()[0]
biases = layer.get_weights()[1]
kernels_extra_channel = np.concatenate((kernels,
kernels[:,:,-1:,:]),
axis=-2) # For channels_last
target_layer.set_weights([kernels_extra_channel, biases])
else:
target_layer.set_weights(layer.get_weights())
#%% Save 4 channel model populated with weights for futher use
vgg_template.save('vgg19_modified_clear.hdf5')
Beyond the RGBY case, the following snippet works generally by copying or removing the layer's weights and/or biases vectors dimensions as needed. Please refer to numpy documentation on what numpy.resize does: in the case of the original question it copies the B-channel weights onto the Y-channel (or more generally onto any higher dimensionality).
import numpy as np
import tensorflow as tf
...
model = ... # your RGBY model is here
pretrained_model = tf.keras.models.load_model(...) # pretrained RGB model
# the following assumes that the layers match with the two models and
# only the shapes of weights and/or biases are different
for pretrained_layer, layer in zip(pretrained_model.layers, model.layers):
pretrained = pretrained_layer.get_weights()
target = layer.get_weights()
if len(pretrained) == 0: # skip input, pooling and other no weights layers
continue
try:
# set the pretrained weights as is whenever possible
layer.set_weights(pretrained)
except:
# numpy.resize to the rescue whenever there is a shape mismatch
for idx, (l1, l2) in enumerate(zip(pretrained, target)):
target[idx] = np.resize(l1, l2.shape)
layer.set_weights(target)