TL;DR - ML model loss, when retrained with new data, reaches NaN quickly. All of the "standard" solutions don't work.
Hello,
Recently, I (successfully) trained a CNN/dense-layered model to be able to classify spectrograms (image representations of audio.) I wanted to try training this model again with new data and made sure that it was the correct dimensions, etc.
However, for some reason, the BinaryCrossentropy loss function steadily declines until around 1.000 and suddenly becomes "NaN" within the first epoch. I have tried lowering the learning rate to 1e-8, am using ReLu throughout and sigmoid for the last layer, but nothing seems to be working. Even simplifying the network to only dense layers, this problem still happens. While I have manually normalized my data, I am pretty confident I did it right so that all of my data falls between [0, 1]. There might be a hole here, but I think that is unlikely.
I attached my code for the model architecture here:
input_shape = (125, 128, 1)
model = models.Sequential([
layers.Conv2D(16, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(0.001), input_shape=input_shape),
layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
layers.BatchNormalization(),
layers.Conv2D(16, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
layers.BatchNormalization(),
layers.Conv2D(16, (3, 3), activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
layers.BatchNormalization(),
layers.Conv2D(16, (2, 2), activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
layers.BatchNormalization(),
layers.Conv2D(16, (2, 2), activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
layers.BatchNormalization(),
layers.Conv2D(16, (2, 2), activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
layers.BatchNormalization(),
layers.Dropout(0.3),
layers.Flatten(),
layers.Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
layers.Dropout(0.5),
layers.Dense(1, activation='sigmoid')
])
Interestingly though, I tried using this new data to fine-tune a VGG16 model, and it worked! (there is no loss NaN problem.) I've attached that code here, but I genuinely have no idea where/if there is any difference causing the problem:
base_model = keras.applications.VGG16(
weights="imagenet",
input_shape=(125, 128, 3),
include_top=False,
)
# Freeze the base_model
base_model.trainable = False
# Create new model on top
inputs = keras.Input(shape=(125, 128, 3))
x = inputs
x = base_model(x, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.001))(x)
x = keras.layers.Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001))(x)
x = keras.layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001))(x)
x = keras.layers.Dropout(0.5)(x) # Regularize with dropout
outputs = keras.layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs, outputs)
model.summary()
I think I've been through all of the "book" solutions, and still can't seem to find the source of the problem. Any help would be much appreciated.
Turns out it was an issue with some of my input data (divide by zero error during normalization.) Sorry for all the trouble and thanks for your help.
Remove all kernel_regularizers, BatchNormalization and dropout layer from Convolution layers which are not required. Keep kernel_regularizers and Dropout only in Dense layers in your model definition as well as change the number of kernels in Conv2d layer.
and try again training your model using below code:
model = Sequential([
Rescaling(1./255, input_shape=(img_h,img_w,3)),
Conv2D(16, (3, 3), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
#BatchNormalization(),
Conv2D(16, (3, 3), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
#BatchNormalization(),
Conv2D(32, (3, 3), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
#BatchNormalization(),
Conv2D(32, (2, 2), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
#BatchNormalization(),
Conv2D(64, (2, 2), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((3, 3), strides=(2, 2), padding='same'),
#BatchNormalization(),
Conv2D(64, (2, 2), activation='relu'),#, kernel_regularizer=regularizers.l2(0.001)),
MaxPooling2D((2, 2), strides=(1, 1), padding='same'),
#BatchNormalization(),
#Dropout(0.3),
Flatten(),
Dense(512, activation='relu'), kernel_regularizer=regularizers.l2(0.001)),
Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001)),
Dropout(0.5),
Dense(1, activation='sigmoid')
])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=['accuracy'])
model.fit(...
Output:
Epoch 1/20
63/63 [==============================] - 9s 97ms/step - loss: 1.0032 - accuracy: 0.5035 - val_loss: 0.8219 - val_accuracy: 0.6160
Epoch 2/20
63/63 [==============================] - 6s 88ms/step - loss: 0.7575 - accuracy: 0.5755 - val_loss: 0.7256 - val_accuracy: 0.6120
Epoch 3/20
63/63 [==============================] - 6s 88ms/step - loss: 0.7181 - accuracy: 0.5805 - val_loss: 0.6917 - val_accuracy: 0.6360
Epoch 4/20
63/63 [==============================] - 6s 88ms/step - loss: 0.6749 - accuracy: 0.6190 - val_loss: 0.6671 - val_accuracy: 0.6300
Epoch 5/20
63/63 [==============================] - 6s 95ms/step - loss: 0.6571 - accuracy: 0.6500 - val_loss: 0.6850 - val_accuracy: 0.5980
Epoch 6/20
63/63 [==============================] - 5s 80ms/step - loss: 0.6319 - accuracy: 0.6720 - val_loss: 0.6243 - val_accuracy: 0.6730
Epoch 7/20
63/63 [==============================] - 6s 90ms/step - loss: 0.5923 - accuracy: 0.6935 - val_loss: 0.6144 - val_accuracy: 0.7120
Epoch 8/20
63/63 [==============================] - 6s 89ms/step - loss: 0.5643 - accuracy: 0.7205 - val_loss: 0.6136 - val_accuracy: 0.6700
Epoch 9/20
63/63 [==============================] - 6s 93ms/step - loss: 0.5552 - accuracy: 0.7380 - val_loss: 0.5669 - val_accuracy: 0.7080
Epoch 10/20
63/63 [==============================] - 4s 58ms/step - loss: 0.5423 - accuracy: 0.7400 - val_loss: 0.5819 - val_accuracy: 0.7120
Epoch 11/20
63/63 [==============================] - 4s 57ms/step - loss: 0.4905 - accuracy: 0.7745 - val_loss: 0.6146 - val_accuracy: 0.7020
Epoch 12/20
63/63 [==============================] - 4s 57ms/step - loss: 0.4808 - accuracy: 0.7900 - val_loss: 0.6318 - val_accuracy: 0.7070
Epoch 13/20
63/63 [==============================] - 4s 60ms/step - loss: 0.4602 - accuracy: 0.7990 - val_loss: 0.5707 - val_accuracy: 0.7160
Epoch 14/20
63/63 [==============================] - 4s 61ms/step - loss: 0.4291 - accuracy: 0.8190 - val_loss: 0.6392 - val_accuracy: 0.6910
Epoch 15/20
63/63 [==============================] - 5s 69ms/step - loss: 0.4003 - accuracy: 0.8355 - val_loss: 0.7048 - val_accuracy: 0.7110
Epoch 16/20
63/63 [==============================] - 4s 58ms/step - loss: 0.3658 - accuracy: 0.8430 - val_loss: 0.8027 - val_accuracy: 0.7180
Epoch 17/20
63/63 [==============================] - 4s 58ms/step - loss: 0.3069 - accuracy: 0.8750 - val_loss: 0.9428 - val_accuracy: 0.6970
Epoch 18/20
63/63 [==============================] - 4s 59ms/step - loss: 0.2601 - accuracy: 0.9005 - val_loss: 0.9420 - val_accuracy: 0.7170
Epoch 19/20
63/63 [==============================] - 4s 60ms/step - loss: 0.2061 - accuracy: 0.9230 - val_loss: 0.9134 - val_accuracy: 0.7290
Epoch 20/20
63/63 [==============================] - 4s 62ms/step - loss: 0.1770 - accuracy: 0.9330 - val_loss: 1.0805 - val_accuracy: 0.6930
I am trying to create a callback for extracting the per-epoch outputs in the 1st hidden layer of my model. With what I have written, self.model.layers[0].output outputs a Tensor object but I could not see the actual entries.
Ideally I would like to save these output tensors, and visualise using an epoch vs mean-output plot. This has been implemented in Glorot & Bengio (2010) but the source code is not available.
How shall I edit my code in order to make the model fitting process save the outputs in each epoch? Thanks in advance.
class PerEpochOutputCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print('First layer output of epoch:', epoch+1, self.model.layers[0].output)
model_relu_3= Sequential()
# Use ReLU hidden layer
model_relu_3.add(Dense(3, input_dim= 8, activation= 'relu', kernel_initializer= 'uniform'))
model_relu_3.add(Dense(5, input_dim= 3, activation= 'relu', kernel_initializer= 'uniform'))
model_relu_3.add(Dense(5, input_dim= 5, activation= 'relu', kernel_initializer= 'uniform'))
model_relu_3.add(Dense(1, activation= 'sigmoid', kernel_initializer= 'uniform'))
model_relu_3.compile(loss='binary_crossentropy', optimizer='adam', metrics= ['accuracy'])
# Train model
tr_results = model_relu_3.fit(X, y, validation_split=0.2, epochs=10, batch_size=32,
verbose=2, callbacks=[PerEpochOutputCallback()])
====
Train on 614 samples, validate on 154 samples
Epoch 1/10
First layer output of epoch: 1 Tensor("dense_42/Relu:0", shape=(None, 3), dtype=float32)
614/614 - 0s - loss: 0.6915 - accuracy: 0.6531 - val_loss: 0.6897 - val_accuracy: 0.6429
Epoch 2/10
First layer output of epoch: 2 Tensor("dense_42/Relu:0", shape=(None, 3), dtype=float32)
614/614 - 0s - loss: 0.6874 - accuracy: 0.6531 - val_loss: 0.6853 - val_accuracy: 0.6429
Epoch 3/10
First layer output of epoch: 3 Tensor("dense_42/Relu:0", shape=(None, 3), dtype=float32)
614/614 - 0s - loss: 0.6824 - accuracy: 0.6531 - val_loss: 0.6783 - val_accuracy: 0.6429
I am strugling to understand how the loss (sparse_categorical_crossentropy) can be zero, but the accuracy << 1? Also the "trained" model does not provide good results. How can it be that the loss gives a zero, despite the model not being well trained:
Epoch 1/3
182/182 [==============================] - 496s 3s/step - loss: 0.0000e+00 - accuracy: 0.2682 - val_loss: 0.0000e+00 - val_accuracy: 0.2729
Epoch 2/3
147/182 [=======================>......] - ETA: 1:29 - loss: 0.0000e+00 - accuracy: 0.2645
The model:
number_of_categories = len(class_names)
loss = 'sparse_categorical_crossentropy'
metrics = ['accuracy']
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv2D(30, (5, 5), activation=activation, input_shape=(pheight, pwidth, 3)))
model.add(tf.keras.layers.MaxPooling2D((5, 5)))
model.add(tf.keras.layers.Conv2D(40, (5, 5), activation=activation))
model.add(tf.keras.layers.MaxPooling2D((5, 5)))
model.add(tf.keras.layers.Conv2D(50, (5, 5), activation=activation))
model.add(tf.keras.layers.MaxPooling2D((5, 5)))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(64, activation=activation))
model.add(tf.keras.layers.Dense(number_of_categories, activation='softmax'))
model.compile(optimizer=optimizer, loss=loss, metrics=metrics)
Any thoughts of what is going wrong here are highly welcome ...
And this is the model fit:
history = model.fit(x=training_generator,
validation_data=test_generator,
batch_size=batch_size,
use_multiprocessing=False,
workers=1,
epochs=epochs,
steps_per_epoch=len(training_generator),
max_queue_size=1)
Problem solved: I had accidentally used small float number likes 0.01 instead 1 as ground truth values (y_true). This did cause the wrong behaviour. By scalling it up and rounding to 1 instead it now works.
I am fitting a large CNN network on my training data, validating on 20%. It appears the model performs better on the training than the validation set. What do you suggest so I can improve the model performance.
CNN Architecture:
model = Sequential()
activ = 'relu'
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, 100, 4)))
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ))
#model.add(BatchNormalization(axis = 3))
model.add(MaxPooling2D(pool_size=(1, 2) ))
model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(Conv2D(64, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(MaxPooling2D(pool_size=(1, 2)))
model.add(Conv2D(128, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(Conv2D(128, (1, 3), strides=(1, 1), padding='same', activation=activ ))
model.add(MaxPooling2D(pool_size=(1, 2)))
model.add(Dropout(.5))
model.add(Flatten())
A = model.output_shape
model.add(Dense(int(A[1] * 1/4.), activation=activ))
model.add(Dropout(.5))
model.add(Dense(5, activation='softmax'))
optimizer = Adam(lr=0.003, beta_1=0.9, beta_2=0.999, epsilon=1e-04, decay=0.0)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=100, batch_size=64, shuffle=False,
validation_split=0.2)
However, the validation accuracy doesn't change for the number of epochs.
Epoch 1/100
1065/1065 [==============================] - 14s 13ms/step - loss: 1.4174 - accuracy: 0.5945 - val_loss: 1.4966 - val_accuracy: 0.4417
Epoch 2/100
1065/1065 [==============================] - 14s 13ms/step - loss: 1.1494 - accuracy: 0.6207 - val_loss: 1.4634 - val_accuracy: 0.4417
Epoch 3/100
1065/1065 [==============================] - 19s 18ms/step - loss: 1.1111 - accuracy: 0.6196 - val_loss: 1.4674 - val_accuracy: 0.4417
Epoch 4/100
1065/1065 [==============================] - 15s 14ms/step - loss: 1.1040 - accuracy: 0.6196 - val_loss: 1.4660 - val_accuracy: 0.4417
Epoch 5/100
1065/1065 [==============================] - 18s 17ms/step - loss: 1.1027 - accuracy: 0.6196 - val_loss: 1.4624 - val_accuracy: 0.4417
NOTE: I Adam´s default learning rate 0.001 as well as 0.003 but the output is the same (log).
Your model is working but improving very slowly. I would reduce the dropout value down to .1 initially, then run the model and see if it overfits or not.If it does then slowly increase the dropout rate. Unless your data is already shuffled I would set shuffle=True in model.fit. Also you might try replacing the Flatten layer with a GlobalMaxPooling layer. I also recommend using the EarlyStopping callback which monitors validation and halts training if the loss fails to reduce after 'patience' number of consecutive epochs. Setting restore_best_weights=True will load the weights for the epoch with the lowest validation loss so you don't have to save then reload the weights. Set epochs to a large number to ensure this callback activates. Also use the ReduceLROnPlateau to automatically adjust the learning rate based on validation loss.
The code I use is shown below
es=tf.keras.callbacks.EarlyStopping( monitor="val_loss", patience=3,
verbose=1, restore_best_weights=True)
rlronp=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1,
verbose=1)
callbacks=[es, rlronp]
In model.fit set callbacks=callbacks. Increase the number of epochs you run to say 100 so that the early stopping callback triggers.
I'm trying to implement a JSCC autoencoder using Keras on CIfar-10 dataset. but the values of the output image is always just 1.
I'm new to Keras and I didn't find out how to fix this.
model = Sequential()
model.add(Conv2D(16,(5,5),padding = 'same', strides = 2, input_shape=X_train.shape[1:]))
model.add(Activation('relu'))
model.add(Conv2D(32,(5,5),padding = 'same', strides = 2))
model.add(Activation('relu'))
model.add(Conv2D(32,(5,5),padding = 'same'))
model.add(Activation('relu'))
model.add(Conv2D(32,(5,5),padding = 'same'))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(100))
model.add(Activation('relu'))
model.add(keras.layers.GaussianNoise(0.1))
model.add(Dense(2048))
model.add(Activation('relu'))
model.add(Reshape((8,8,32)))
model.add(Conv2DTranspose(32,(5,5), padding = 'same'))
model.add(Activation('relu'))
model.add(Conv2DTranspose(32,(5,5), padding = 'same'))
model.add(Activation('relu'))
model.add(Conv2DTranspose(32,(5,5), strides = 2 ,padding = 'same'))
model.add(Activation('relu'))
model.add(Conv2DTranspose(3,(5,5), strides = 2 ,padding = 'same'))
model.add(Activation('sigmoid'))
model.compile(loss='mse', optimizer='adam')
model.fit(X_train_norm, X_train_norm,
batch_size=128,
epochs=20,
validation_data=(X_test_norm, X_test_norm),
shuffle=True)
this model compresses the image to a vector with length of 100 and it adds up with a gaussian noise and then it upsamples the vector to original input.
Train on 50000 samples, validate on 10000 samples
Epoch 1/20
50000/50000 [==============================] - 7s 138us/step - loss: 0.0245 - val_loss: 0.0226
Epoch 2/20
50000/50000 [==============================] - 6s 120us/step - loss: 0.0225 - val_loss: 0.0222
Epoch 3/20
50000/50000 [==============================] - 6s 121us/step - loss: 0.0220 - val_loss: 0.0216
Epoch 4/20
50000/50000 [==============================] - 6s 121us/step - loss: 0.0214 - val_loss: 0.0211
Epoch 5/20
50000/50000 [==============================] - 6s 119us/step - loss: 0.0208 - val_loss: 0.0207
...
>>>model.predict(X_train[:32])
array([[[[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.],
...,
You used the normalised data during training but the raw data at prediction.
instead of:
model.predict(X_train[:32])
use:
model.predict(X_train_norm[:32])