What is wrong with my CNN? [closed] - tensorflow

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I really don't understand what is wrong with my model. Sometimes it gives me excellent results, but in other cases results are just absurd. During training, from one moment to another it gives absurd results. I tried model with 3 dropout layers and without them, and get same strange results. Here's my model definition:
batch_size = 1
epochs = 25
model = Sequential()
model.add(Conv1D(32, input_shape=(1040,1), kernel_size=100,padding='same',name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=70, strides=1, padding='same',name='PoolingLayer1'))
#model.add(Dropout(0.10))
model.add(Conv1D(64, kernel_size=70,padding='same',name='ConvLayer2',strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=40, strides=1, padding='same',name='PoolingLayer2'))
#model.add(Dropout(0.10))
model.add(Conv1D(128, kernel_size=40,padding='same',name='ConvLayer3',strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=10, strides=1, padding='same',name='PoolingLayer3'))
#model.add(Dropout(0.10))
model.add(Flatten())
model.add(Dense(1,name='output', activation='linear'))
w = model.get_weights()
model.compile(loss='mse', optimizer=keras.optimizers.Adam(lr=0.001),metrics=['mse'])
Get that kind of results: Results screenshot
What is happening? And also, do you know how I can improve this model to get better results?

Answer 1

Decrease Kernel and Pool Sizes
From just glancing over your architecture I would say that it's worth trying much smaller values for the pooling and conv filters. The network would have to find a pattern while looking at 100 values at the same time. To put this in perspective, when convolutional nets are used in image processing they have found that kernel sizes of 2-4 are best. You can still look at many data-points all at once in the deeper layers because as they pool together they are looking at deeper combinations of data points.
Increase Batch Size
It's very hard for a network to establish a good gradient from a single example. You should be using larger batch sizes, I would start with 32 and move around from there.
Start with the above changes and then try...
Adding another dense layer before your output layer
Batch normalization between layers
A different activation function. Not sure what your use-case is but you may need to look at that too to optimize performance
Try something like this to see if it improves.
from keras import Sequential
from keras.layers import Conv1D, LeakyReLU, MaxPooling1D, Flatten, Dense
import keras
batch_size = 32
epochs = 25
model = Sequential()
model.add(Conv1D(32, input_shape=(1040, 1), kernel_size=2, padding='same', name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=2, strides=1, padding='same', name='PoolingLayer1'))
# model.add(Dropout(0.10))
model.add(Conv1D(64, kernel_size=3, padding='same', name='ConvLayer2', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=3, strides=1, padding='same', name='PoolingLayer2'))
# model.add(Dropout(0.10))
model.add(Conv1D(128, kernel_size=3, padding='same', name='ConvLayer3', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=3, strides=1, padding='same', name='PoolingLayer3'))
# model.add(Dropout(0.10))
model.add(Flatten())
model.add(Dense(1, name='output', activation='linear'))
model.compile(loss='mse', optimizer=keras.optimizers.Adam(lr=0.001), metrics=['mse'])
model.summary()
BatchNormalization Example
from keras.layers import BatchNormalization
model.add(Conv1D(32, input_shape=(1040, 1), kernel_size=2, padding='same', name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(BatchNormalization()) # Try adding this after each activation function except the output layer
model.add(MaxPooling1D(pool_size=2, strides=1, padding='same', name='PoolingLayer1'))
I would also add early stopping and/or model check-pointing to stop the training once the validation loss stops improving and allows you to load the weights for the best validation loss.