I'm currently trying to build a CNN that can detect whether a patient has pnemonia caused by covid or not, and no matter what parameters I change the model accuracy is staying at 49%/50% so its basically useless because it's the same as a coin flip. Here is my code, I thought I would try using the VGG-16 model.
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, GlobalAveragePooling2D
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping
from keras.preprocessing.image import ImageDataGenerator
# Loading in the dataset
traindata = ImageDataGenerator(rescale=1/255)
trainingdata = traindata.flow_from_directory(
directory="Covid-19CT/TrainingData",
target_size=(224,224),
batch_size=100,
class_mode="binary")
testdata = ImageDataGenerator(rescale=1/255)
testingdata = testdata.flow_from_directory(
directory="Covid-19CT/TestingData",
target_size=(224,224),
batch_size=100,
class_mode="binary")
# Initialize the model w/ Sequential & add layers + input and output <- will refer to the VGG 16 model architecture
model = Sequential()
model.add(Conv2D(input_shape=(224,224,3),filters=64,kernel_size=(2,2),padding="same", activation="relu"))
model.add(Conv2D(filters=64, kernel_size=(3,3), padding="same", activation ="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=2))
model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=2))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=2))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=2))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2), strides=2))
model.add(GlobalAveragePooling2D())
model.add(Dense(units=4096, activation="relu"))
model.add(Dense(units=4096, activation="relu"))
model.add(Dense(units=1000, activation="relu"))
model.add(Dense(units=1, activation="softmax"))
# Compile the model
model_optimizer = Adam(lr=0.001)
model.compile(optimizer=model_optimizer, loss=keras.losses.binary_crossentropy, metrics=['accuracy'])
# Add the callbacks
checkpoint = ModelCheckpoint(filepath="Covid-19.hdf5", monitor='val_acc', verbose=1, save_best_only=True, save_weights_only=False, mode='auto')
early = EarlyStopping(monitor='val_acc', min_delta=0, patience=50, verbose=1, mode='auto')
fit = model.fit_generator(steps_per_epoch=25, generator=trainingdata, validation_data=testingdata, validation_steps=10,epochs=10,callbacks=[checkpoint,early])
This always gives:
Epoch 1/10 6/25 [======>.......................] - ETA: 1:22:37 -
loss: 7.5388 - accuracy: 0.5083
<- Well, it just always gives a really poor accuracy...
Additional info:
Some of the images in the data set are JPG others are PNG (Not sure if this is the culprit)
The Dataset has 2072 images for training Covid CTs and 2098 images for training NonCovid CTs
The Dataset has 576 images for testing Covid CTs and 532 images for testing NonCovid CTs
File structure looks like this: Covid19ModelImages -> Training Data & Testing Data - Training Data has 2 subfolders Covid19CT and noncovid19 CT and testing data also has 2 subfolders Covid19CT and noncovid19CT
Also: Am I just being too impatient? I never let it run past the 1st epoch cause I just assume its never going to get better than 50%, could it be that the model will improve more on the next epochs?
If anyone would be willing to help out, or if you need any other additional info to maybe help you gain a better understanding of the problem, please let me know!
Since you are using binary cross entropy, the activation function in the dense layer with 1 unit should be "sigmoid". Since you are not using a GPU you have very long training times per epoch. To see if the model is working correctly you may want to reduce this time. There are few things you could do. Try reducing the image size say to 128 by 128. With 224 X 224 you have 50176 pixels to process versus 16384 for the 128 X 128 image so you reduce the computations by about a factor of 3. Also you have two dense layers with 4096 units. This is also computationally expense. It may also lead to overfitting. Try your model initially without these layers and see how it performs. I am not a fan of early stopping because it is a crutch to avoid dealing with the over fitting issue. If you encounter over fitting add a dropout layer to help avoid it. Finally I recommend you use an adjustable learning rate. The callback ReduceLROnPlateau makes this easy to do. Set it to monitor validation loss. You can set the parameters to reduce the learning rate a factor<1 if the loss fails to decrease after "patience" number of consecutive epochs. I usually use factor=.5 and patience=1. This also enables you to use a larger initial learning rate for faster convergence. Documentation is here. You need to let your model run for several epochs to see if the training loss and validation loss are decreasing.
Related
I have the following CNN topology -
model=Sequential()
#model.add(Lambda(standardize,input_shape=(28,28,1)))
model.add(Conv2D(filters=64, kernel_size = (3,3), activation="relu", input_shape=(32,32,3)))
model.add(Conv2D(filters=64, kernel_size = (3,3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.50))
model.add(Conv2D(filters=128, kernel_size = (3,3), activation="relu"))
model.add(Conv2D(filters=128, kernel_size = (3,3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.50))
model.add(Conv2D(filters=256, kernel_size = (3,3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.50))
model.add(Flatten())
model.add(Dense(512,activation="relu"))
model.add(BatchNormalization())
model.add(Dense(10,activation="softmax"))
This network worked really well on just 500 training images of the MNIST dataset (https://www.tensorflow.org/datasets/catalog/mnist) Input size - (28,28,1)
However, on 500 training images of the SVHN dataset (http://ufldl.stanford.edu/housenumbers/), the model doesn't seem to learn as its validation accuracy maxed out at 0.1959. Input size - (32,32,3)
Another CNN I created which is much simpler than this network however, reaches ~70% validation accuracy on the SVHN dataset.
Im failing to understand why this might be the case - is it cause the CNN doesnt work the same on RGB images? Is it cause the CNN isnt complex enough to extract features of the SVHN dataset?
Let me know if there is anything else I could provide to help you guys out with this problem. :)
Given a pre-trained well-performing auto-encoder. When I train a classifier on encodings (produced by the auto-encoder) the classifier does very poorly. In particular, it does much worse than training a classifier on normal inputs (i.e. unencoded inputs).
However, when I fine-tune the encoder based on classification loss, the classifier does quite well.
Why are encoded representations bad for classification?
Details: I’m working on CIFAR-100 and trying to classify coarse image labels, i.e. 20 classes (but I think I had the same problem when doing classification on CIFAR-10). The classifier has 5 layers and I’m using dropout:
classifier = tf.keras.Sequential([
tf.keras.layers.Dense(512,
activation='relu',
name='classifier_hidden_1'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(256,
activation='relu',
name='classifier_hidden_2'),
tf.keras.layers.Dense(128,
activation='relu',
name='classifier_hidden_3'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(64,
activation='relu',
name='classifier_hidden_4'),
tf.keras.layers.Dense(num_classes,
activation=None,
name='classifier_out'),
], name='classifier')
I want to freeze training of first two layers of following code after 3rd epoch. Total epoch is set to 10.
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
How can I "freeze" Keras layers?
To "freeze" a layer means to exclude it from training, i.e. its weights will never be updated. This is useful in the context of fine-tuning a model or using fixed embeddings for a text input.
You can change the trainable attribute of a layer.
for layer in model.layers[:2]:
layer.trainable = False
For this to take effect, you will need to call compile() on your model after modifying the trainable property. If you don't you will receive a warning "Discrepancy between trainable weights and collected trainable weights" and all your layers will be still trainable. So:
Build and compile the model
Train it for 3 epochs
Freeze layers you want
compile the model again
Train the rest epochs
This should work:
for epoch in range(3):
model.fit(.., epochs=1)
# save the weights of this model
model.save_weights("weight_file.h5")
# freeze the layers you want
for layer in model.layers[:2]:
layer.trainable = False
In order to train further with these weights but first two layers frozen, you need to compile the model again.
model.compile(..)
# train further
for epoch in range(3, 10):
model.fit(..., epochs=1)
I have a CNN that I'm trying to train and I cant figure out why its not learning. It has 32 classes which are different types of clothes of about 1000 images in each folder.
Issue is this is the result at the end of training which takes about 9 hours on my GPU
loss: 3.3403 - acc: 0.0542 - val_loss: 3.3387 - val_acc: 0.0534
If anyone could give me directions on how to get this network to train better I would be grateful.
# dimensions of our images.
img_width, img_height = 228, 228
train_data_dir = 'Clothes/train'
validation_data_dir = 'Clothes/test'
nb_train_samples = 25061
nb_validation_samples = 8360
epochs = 20
batch_size = 64
if K.image_data_format() == 'channels_first':
input_shape = (3, img_width, img_height)
else:
input_shape = (img_width, img_height, 3)
model = Sequential()
model.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='tanh', input_shape=input_shape))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='tanh'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(32, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
[test_datagen = ImageDataGenerator(rescale=1. / 255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode='categorical',
shuffle = True)
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode='categorical',
shuffle = True)
history = model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples // batch_size,
epochs=epochs,
validation_data=validation_generator,
validation_steps=nb_validation_samples // batch_size)
Here is the plot of the training & validation loss
A network may not converge/learn for several reasons, but here is a list of tips that I think is relevant in your case (based on my own experience):
Transfer Learning: The first thing you should know is that it's very hard to train an image classifier from scratch for most problems, you need much more computing power and time for that. I strongly recommend using transfer learning. There are multiple trained architectures available in Keras that you can use as initial weights fr your network (or other methods).
Training step: For the optimizer, I recommend to use Adam first and to vary the learning rate to see how the loss responds. Also, since you are using Convolutional Layers, you should consider adding Batch Normalization Layers, that can speed significantly the training time, and change the Convolutional activations to 'relu', which make them much faster to train.
You could also try decreasing the Dropout values but I don't think that's the main issue here. Also, If you are considering training your network from scratch,
you should start with fewer layers and add more gradually to get a better idea of what's going on.
train/test split: I see that you are using 8360 observations in your test set. Given the size of your training set, I think it's too much. 1000 for example is enough. The more training samples you have, the more satisfying your results will be.
Also, before judging the accuracy of your model, you should start by establishing a baseline model to benchmark your model. The baseline model depends on your problem, but in general I choose a model that predicts the most common class in the dataset. You should also look at another metric 'top_k_accuracy' available in Keras that is interesting when you have a relatively high number of classes to predict. It helps you to see how close your model is to the right prediction.
First, in order to keep your sanity, check carefully for any bugs, and that your data is being sent in as intended
You might want to add a Top K accuracy metric to get a better idea of whether it's close to getting it, or totally wrong.
Here are some tuning things to try:
Change the kernel size to 3 and activation to relu
model.add(Conv2D(filters=64, kernel_size=3, padding='same', activation='relu'))
If you think your model is underfitting then try increasing the number of Conv layers per pooling to start with. But you could also increase the number of filters or the number of conv + pool repetitions.
Adam optimizer might learn a bit faster than RMS prop
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
Probably the biggest improvement would be to get more data. I think your data set is probably too small for the scope of the problem.
You might want to try transfer learning from a pre-trained image recognition network.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I really don't understand what is wrong with my model. Sometimes it gives me excellent results, but in other cases results are just absurd. During training, from one moment to another it gives absurd results. I tried model with 3 dropout layers and without them, and get same strange results. Here's my model definition:
batch_size = 1
epochs = 25
model = Sequential()
model.add(Conv1D(32, input_shape=(1040,1), kernel_size=100,padding='same',name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=70, strides=1, padding='same',name='PoolingLayer1'))
#model.add(Dropout(0.10))
model.add(Conv1D(64, kernel_size=70,padding='same',name='ConvLayer2',strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=40, strides=1, padding='same',name='PoolingLayer2'))
#model.add(Dropout(0.10))
model.add(Conv1D(128, kernel_size=40,padding='same',name='ConvLayer3',strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=10, strides=1, padding='same',name='PoolingLayer3'))
#model.add(Dropout(0.10))
model.add(Flatten())
model.add(Dense(1,name='output', activation='linear'))
w = model.get_weights()
model.compile(loss='mse', optimizer=keras.optimizers.Adam(lr=0.001),metrics=['mse'])
Get that kind of results: Results screenshot
What is happening? And also, do you know how I can improve this model to get better results?
Decrease Kernel and Pool Sizes
From just glancing over your architecture I would say that it's worth trying much smaller values for the pooling and conv filters. The network would have to find a pattern while looking at 100 values at the same time. To put this in perspective, when convolutional nets are used in image processing they have found that kernel sizes of 2-4 are best. You can still look at many data-points all at once in the deeper layers because as they pool together they are looking at deeper combinations of data points.
Increase Batch Size
It's very hard for a network to establish a good gradient from a single example. You should be using larger batch sizes, I would start with 32 and move around from there.
Start with the above changes and then try...
Adding another dense layer before your output layer
Batch normalization between layers
A different activation function. Not sure what your use-case is but you may need to look at that too to optimize performance
Try something like this to see if it improves.
from keras import Sequential
from keras.layers import Conv1D, LeakyReLU, MaxPooling1D, Flatten, Dense
import keras
batch_size = 32
epochs = 25
model = Sequential()
model.add(Conv1D(32, input_shape=(1040, 1), kernel_size=2, padding='same', name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=2, strides=1, padding='same', name='PoolingLayer1'))
# model.add(Dropout(0.10))
model.add(Conv1D(64, kernel_size=3, padding='same', name='ConvLayer2', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=3, strides=1, padding='same', name='PoolingLayer2'))
# model.add(Dropout(0.10))
model.add(Conv1D(128, kernel_size=3, padding='same', name='ConvLayer3', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling1D(pool_size=3, strides=1, padding='same', name='PoolingLayer3'))
# model.add(Dropout(0.10))
model.add(Flatten())
model.add(Dense(1, name='output', activation='linear'))
model.compile(loss='mse', optimizer=keras.optimizers.Adam(lr=0.001), metrics=['mse'])
model.summary()
BatchNormalization Example
from keras.layers import BatchNormalization
model.add(Conv1D(32, input_shape=(1040, 1), kernel_size=2, padding='same', name='ConvLayer1', strides=1))
model.add(LeakyReLU(alpha=0.1))
model.add(BatchNormalization()) # Try adding this after each activation function except the output layer
model.add(MaxPooling1D(pool_size=2, strides=1, padding='same', name='PoolingLayer1'))
I would also add early stopping and/or model check-pointing to stop the training once the validation loss stops improving and allows you to load the weights for the best validation loss.