Related
I'm trying to use VGG16 from keras to train a model for image detection.
Based on these articles (https://www.pyimagesearch.com/2019/06/03/fine-tuning-with-keras-and-deep-learning/ and https://learnopencv.com/keras-tutorial-fine-tuning-using-pre-trained-models/), I've put some addition Dense layer to the VGG 16 model. However, the training accuracy with 20 epoche is around 35% to 41% which doesn't match the result of these articles (above 90%).
Due to this, I would like to know, did I do something wrong with my code below.
Basic setting
url='/content/drive/My Drive/fer2013.csv'
batch_size = 64
img_width,img_height = 48,48
# 0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral
num_classes = 7
model_path = '/content/drive/My Drive/Af/cnn.h5'
df=pd.read_csv(url)
def _load_fer():
# Load training and eval data
df = pd.read_csv(url, sep=',')
train_df = df[df['Usage'] == 'Training']
eval_df = df[df['Usage'] == 'PublicTest']
return train_df, eval_df
def _preprocess_fer(df,label_col='emotion',feature_col='pixels'):
labels, features = df.loc[:, label_col].values.astype(np.int32), [
np.fromstring(image, np.float32, sep=' ')
for image in df.loc[:, feature_col].values]
labels = [to_categorical(l, num_classes=num_classes) for l in labels]
features = np.stack((features,) * 3, axis=-1)
features /= 255
features = features.reshape(features.shape[0], img_width, img_height,3)
return features, labels
# Load fer data
train_df, eval_df = _load_fer()
# preprocess fer data
x_train, y_train = _preprocess_fer(train_df)
x_valid, y_valid = _preprocess_fer(eval_df)
gen = ImageDataGenerator(
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
train_generator = gen.flow(x_train, y_train, batch_size=batch_size)
predict_size_train = int(np.math.ceil(len(x_train) / batch_size))
input_tensor = Input(shape=(img_width, img_height, 3))
Now comes the model training part
baseModel = VGG16(
include_top=False, weights='imagenet',
input_tensor=input_tensor
)
# Construct the head of the model that will be placed on top of the base model (fine tuning)
headModel = baseModel.output
headModel = Flatten()(headModel)
headModel = Dense(1024, activation="relu")(headModel)
#headModel = Dropout(0.5)(headModel)
headModel = BatchNormalization()(headModel)
headModel = Dense(num_classes, activation="softmax")(headModel)
model = Model(inputs=baseModel.input, outputs=headModel)
for layer in baseModel.layers:
layer.trainable = False
model summary
model.compile(loss='categorical_crossentropy',
optimizer=tf.keras.optimizers.Adam(lr=0.001),
metrics=['accuracy'])
history = model.fit(train_generator,
steps_per_epoch=predict_size_train * 1,
epochs=20,
validation_data=valid_generator,
validation_steps=predict_size_valid)
Result:
Result after training
It will be very thankful for you advice.
Best Regards.
Since you are freezing all layers, only one dense layer might not give you desired accuracy. Also if you are not in hurry, you may not set up the validation_steps and steps_per_epochs parameters. Also in this tutorial, model is having fluctuations, which do not want.
I suggest:
for layer in baseModel.layers:
layer.trainable = False
base_out = baseModel.get_layer('block3_pool').output // layer name may be different,
check with model baseModel.summary
With that you can get spefic layer's output. After got the output, you can add some convolutions. After convolutions try stacking more dense layers like:
x = tf.keras.layers.Flatten()(x)
x = Dense(512, activation= 'relu')(x)
x = Dropout(0.3)(x)
x = Dense(256, activation= 'relu')(x)
x = Dropout(0.2)(x)
output_model = Dense(num_classes, activation = 'softmax')(x)
If you don't want to add convolutions and use baseModel completely, that's also fine however you can do something like this:
for layer in baseModel.layers[:12]: // 12 is random you can try different ones. Not
all layers are frozen this time.
layer.trainable = False
for i, layer in enumerate(baseModel.layers):
print(i, layer.name, layer.trainable)
// check frozen layers
After that, you can try to set:
headModel = baseModel.output
headModel = Flatten()(headModel)
headModel = Dense(1024, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(512, activation="relu")(headModel)
headModel = Dense(num_classes, activation="softmax")(headModel)
If you see your model is learning, but your loss having fluctuations then you can reduce learning rate. Or you can use ReduceLROnPlateau callback:
rd_lr = ReduceLROnPlateau(monitor='val_loss', factor = np.sqrt(0.1), patience= 4, verbose = 1, min_lr = 5e-8)
Parameters are totally up to your model. For more details you can see docs
what is the form of the content of y_train. If they are integer values then you need to convert them to one hot vectors with
y_train=tf.keras.utils.to_categorical(train, num_classes)
since you are using loss='categorical_crossentropy' in model.compile. In addition VGG16 requires that the pixels be scaled between -1 and +1 so in include
gen = ImageDataGenerator(tf.keras.applications.vgg16.preprocess_input, etc
When you are training you have
for layer in baseModel.layers:
layer.trainable = False
so you are only training the dense layer which is OK but may not give you high accuracy. You might want to leave VGG as trainable but of course this will take longer. Or after you train with VGG not trainable, then change it back to trainable and run a few more epochs to fine tune the model.
currently I am working on an image error classifier using tensorflow and the on ImageNet pre-trained EfficientNetB0 from keras applications. As metrics, I am using false positive (fp), true positive (tp), false negative (fn), true negative (tn), ...
The problem I have with the metrics like fp, tp, fn and tn is that they are not real integer values during training (i.e. tp = 4883.6257) and only during validation they are integer values. As far as I know, these metrics should always be integers as they are only then number of i.e. false positive predicted samples. Is there something I am missing what keras does for comuting these values during training?
As input pipeline I am using the tensorflow ImageDataGenerator and the .flow_from_dataframe() function:
# create data generators in order to load the images
datagen_train = ImageDataGenerator(horizontal_flip = True, vertical_flip = True, brightness_range = (0.9, 1.1),
rescale=1. / 255, fill_mode = "constant", zoom_range = 0.3, channel_shift_range=100.0)
datagen_val = ImageDataGenerator(rescale = 1. / 255)
train_generator = datagen_train.flow_from_dataframe(
dataframe = balanced_df[:N_Train],
directory = bitmap_folder_path,
x_col = "filename",
y_col = "particle",
batch_size = batch_size,
shuffle = True,
class_mode = "binary",
target_size = (250,250),
color_mode = "rgb",
seed = 42)
valid_generator = datagen_val.flow_from_dataframe(
dataframe = balanced_df[N_Train:],
directory = bitmap_folder_path,
x_col = "filename",
y_col = "particle",
batch_size = batch_size,
shuffle = True,
class_mode = "binary",
target_size = (250,250),
color_mode = "rgb",
seed = 42)
Setting up the model:
from tensorflow.keras.applications import EfficientNetB0
input_shape = (img_height, img_width, 3) # use depth=3 because imagenet is trained on RGB images
model = EfficientNetB0(weights='imagenet', include_top = False, input_shape = input_shape)
# add a global spatial average pooling layer
x = model.output
x = keras.layers.GlobalAveragePooling2D()(x)
# and a fully connected output/classification layer
predictions = keras.layers.Dense(1, activation='sigmoid')(x)
# create the full network so we can train on it
model_B0 = keras.models.Model(inputs=model.input, outputs=predictions)
batch_size = 16
num_epochs = 30
# setup optimizer similar to used one in original paper
# they used: RMSProp with decay of 0.9 and momentum of 0.9, batch norm momentum of 0.99, a initial learning rate of
# 0.256 that decays by 0.97 every 2.4 epochs
initial_learning_rate = 1e-5
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate,
decay_steps= int(2.4 * steps_per_epoch_train),
decay_rate=0.97,
staircase=True)
opt_efficientNet = tf.keras.optimizers.RMSprop(learning_rate=lr_schedule,
rho=0.9, momentum=0.9, name="RMSprop")
For better analysis, I've added the following metrics:
METRICS = [
keras.metrics.TruePositives(name='tp'),
keras.metrics.FalsePositives(name='fp'),
keras.metrics.TrueNegatives(name='tn'),
keras.metrics.FalseNegatives(name='fn'),
keras.metrics.BinaryAccuracy(name='accuracy'),
keras.metrics.Precision(name='precision'),
keras.metrics.Recall(name='recall'),
keras.metrics.AUC(name='auc'),
]
model_B0.compile(
loss="binary_crossentropy",
optimizer=opt_efficientNet,
metrics=METRICS)
I think you should define parameter thresholds in those metrics. By default, BinaryAccuracy metrics has thresholds of 0.5, which you can adjust according to the accuracy.
Example:
keras.metrics.TruePositives(name='tp', thresholds=0.5)
I'm working on CNN model and I'm curious to know-how converts the output given by datagen.flow_from_directory() into a bumpy array. The format of datagen.flow_from_directory() is directoryiterator.
Apart from ImageDataGenerator is any other way also to fetch data from the directory.
img_width = 150
img_height = 150
datagen = ImageDataGenerator(rescale=1/255.0, validation_split=0.2)
train_data_gen = directory='/content/xray_dataset_covid19',
target_size = (img_width, img_height),
class_mode='binary',
batch_size=16,
subset='training')
vali_data_gen = datagen.flow_from_directory(directory='/content/xray_dataset_covid19',
target_size = (img_width, img_height),
class_mode='binary',
batch_size=16,
subset='validation')
First Method:
import numpy as np
data_gen = ImageDataGenerator(rescale = 1. / 255)
data_generator = datagen.flow_from_directory(
data_dir,
target_size=(img_height, img_width),
batch_size=batch_size,
class_mode='categorical')
data_list = []
batch_index = 0
while batch_index <= data_generator.batch_index:
data = data_generator.next()
data_list.append(data[0])
batch_index = batch_index + 1
# now, data_array is the numeric data of whole images
data_array = np.asarray(data_list)
Alternatively, you can use PIL and numpy process the image by yourself:
from PIL import Image
import numpy as np
def image_to_array(file_path):
img = Image.open(file_path)
img = img.resize((img_width,img_height))
data = np.asarray(img,dtype='float32')
return data
# now data is a tensor with shape(width,height,channels) of a single image
Second Method: you should use ImageDataGenerator.flow, which takes numpy arrays directly. This replaces the flow_from_directory call, all other code using the generator should be the same
You need to use like this, is much more effective than the other methods in case of RAM usage.
img_width = 150
img_height = 150
datagen = ImageDataGenerator(rescale=1/255.0, validation_split=0.2)
train_data_gen = directory='/content/xray_dataset_covid19',
target_size = (img_width, img_height),
class_mode='binary',
batch_size=16,
subset='training')
vali_data_gen = datagen.flow_from_directory(directory='/content/xray_dataset_covid19',
target_size = (img_width, img_height),
class_mode='binary',
batch_size=16,
subset='validation')
x_train=np.concatenate([train_data_gen .next()[0] for i in range(train_data_gen .__len__())])
y_train=np.concatenate([train_data_gen .next()[1] for i in range(train_data_gen .__len__())])
x_val=np.concatenate([vali_data_gen .next()[0] for i in range(vali_data_gen .__len__())])
y_val=np.concatenate([vali_data_gen .next()[1] for i in range(vali_data_gen .__len__())])
Now you can use the x_train and y_train as an array
You can iterate through the generator.
def sample_from_generator(gen, nb_sample):
cur_x, cur_y = next(gen)
input_shape = list(cur_x.shape)[1:]
num_classes = cur_y.shape[1]
batch_size = len(cur_x)
X_sample = np.zeros([nb_sample] + list(input_shape))
Y_sample = np.zeros((nb_sample, num_classes))
for i in range(0, nb_sample, batch_size):
cur_x, cur_y = next(gen)
if len(X_sample[i:i + batch_size]) < len(cur_x):
cur_x = cur_x[:len(X_sample[i:i + batch_size])]
cur_y = cur_y[:len(Y_sample[i:i + batch_size])]
X_sample[i:i + batch_size] = cur_x
Y_sample[i:i + batch_size] = cur_y
return X_sample, Y_sample
I am trying to create a model which can tell whether there are birds in an image or not.
I was using categorical classification to train the model to recognize Bird v.s. Flowers, the results turned to be very successful in terms of recognizing these 2 classes.
BUT, when I change it to Binary Classification to detect the existence of birds in an images, the accuracy dropped dramatically.
The reason why I changed to use Binary classification is that if I
provided a dog to my Categorical Classification trained model, it
recognized the dog as a bird.
btw, here is my data set structure:
Training:
5000 images for birds and 2000 images for not-birds
Validating:
1000 images for birds and 500 images for not-birds
Someone said, the inblanced dataset will also cause problems. Is it true?
Could someone please point out where I get wrong in the following code?
def get_num_files(path):
if not os.path.exists(path):
return 0
return sum([len(files) for r, d, files in os.walk(path)])
def get_num_subfolders(path):
if not os.path.exists(path):
return 0
return sum([len(d) for r, d, files in os.walk(path)])
def create_img_generator():
return ImageDataGenerator(
preprocessing_function=preprocess_input,
rotation_range=30,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True
)
INIT_LT = 1e-3
Image_width, Image_height = 299, 299
Training_Epochs = 30
Batch_Size = 32
Number_FC_Neurons = 1024
Num_Classes = 2
train_dir = 'to my train folder'
validate_dir = 'to my validation folder'
num_train_samples = get_num_files(train_dir)
num_classes = get_num_subfolders(train_dir)
num_validate_samples = get_num_files(validate_dir)
num_epoch = Training_Epochs
batch_size = Batch_Size
train_image_gen = create_img_generator()
test_image_gen = create_img_generator()
train_generator = train_image_gen.flow_from_directory(
train_dir,
target_size=(Image_width, Image_height),
batch_size = batch_size,
seed = 42
)
validation_generator = test_image_gen.flow_from_directory(
validate_dir,
target_size=(Image_width, Image_height),
batch_size=batch_size,
seed=42
)
Inceptionv3_model = InceptionV3(weights='imagenet', include_top=False)
print('Inception v3 model without last FC loaded')
x = Inceptionv3_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(Number_FC_Neurons, activation='relu')(x)
predictions = Dense(num_classes, activation='softmax')(x)
# model = Model(inputs=Inceptionv3_model.input, outputs=predictions)
v3model = Model(inputs=Inceptionv3_model.input, outputs=predictions)
# Use new Sequential model to add v3model and add a bath normalization layer after
model = Sequential()
model.add(v3model)
model.add(BatchNormalization()) # added normalization
print(model.summary())
print('\nFine tuning existing model')
Layers_To_Freeze = 172
for layer in model.layers[:Layers_To_Freeze]:
layer.trainable = False
for layer in model.layers[Layers_To_Freeze:]:
layer.trainable = True
optizer = Adam(lr=INIT_LT, decay=INIT_LT / Training_Epochs)
# optizer = SGD(lr=0.0001, momentum=0.9)
model.compile(optimizer=optizer, loss='binary_crossentropy', metrics=['accuracy'])
cbk_early_stopping = EarlyStopping(monitor='val_acc', mode='max')
history_transfer_learning = model.fit_generator(
train_generator,
steps_per_epoch = num_train_samples,
epochs=num_epoch,
validation_data=validation_generator,
validation_steps = num_validate_samples,
class_weight='auto',
callbacks=[cbk_early_stopping]
)
model.save('incepv3_transfer_mini_binary.h5', overwrite=True, include_optimizer=True)
Categorical
Use Num_Classes = 2
Use one-hot-encoded targets (example: Bird = [1, 0], Flower = [0, 1])
Use 'softmax' activation
Use 'categorical_crossentropy'
Binary
Use Num_Classes = 1
Use binary targets (example: is flower = 1 | not flower = 0)
Use 'sigmoid' activation
Use 'binary_crossentropy'
Details here: Using categorical_crossentropy for only two classes
I'm currently working on a CNN model that classifies food images. So far, I have managed to build a functioning CNN but I would like to improve the accurracy. For the dataset, I have used some images from Kaggle and few from my own collection.
Here is some information about the dataset:
There are 91 classes of food images.
Each class has around 500 to 650 images.
The dataset has been manually cleaned and checked for unrelated or bad quality images (the photos are of different sizes).
Here is my CNN model:
classifier = Sequential()
def cnn_layer_creation(classifier):
classifier.add(InputLayer(input_shape=[224,224,3]))
classifier.add(Conv2D(filters=32,kernel_size=5,strides=1,padding='same',activation='relu',data_format='channels_first'))
classifier.add(MaxPooling2D(pool_size=5,padding='same'))
classifier.add(Conv2D(filters=50,kernel_size=5,strides=1,padding='same',activation='relu'))
classifier.add(MaxPooling2D(pool_size=5,padding='same'))
classifier.add(Conv2D(filters=80,kernel_size=5,strides=1,padding='same',activation='relu',data_format='channels_last'))
classifier.add(MaxPooling2D(pool_size=5,padding='same'))
classifier.add(Dropout(0.25))
classifier.add(Flatten())
classifier.add(Dense(64,activation='relu'))
classifier.add(Dropout(rate=0.5))
classifier.add(Dense(91,activation='softmax'))
# Compiling the CNN
classifier.compile(optimizer="RMSprop", loss = 'categorical_crossentropy', metrics = ['accuracy'])
data_initialization(classifier)
def data_initialization(classifier):
# Part 2 - Fitting the CNN to the images
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
training_set = train_datagen.flow_from_directory('food_image/train',
target_size = (224, 224),
batch_size = 100,
class_mode = 'categorical')
test_set = test_datagen.flow_from_directory('food_image/test',
target_size = (224, 224),
batch_size = 100,
class_mode = 'categorical')
classifier.fit_generator(training_set,
steps_per_epoch = 100,
epochs = 100,
validation_data = test_set,
validation_steps = 100)
classifier.save("brynModelGPULite.h5")
classifier.summary()
def main():
cnn_layer_creation(classifier)
Training is done on GPU (nVidia 980M)
Unfortunately, the accuracy has not exceeded 10%. Things I've tried are:
Increase the number of epochs.
Change the optimizer (ADAM, RMSPROP).
Change the activation function.
Reduce the image input size.
Increase the batch size.
Change the filter size to 32, 64, 128.
None of these have improved the accuracy.
Could anyone shine some light on how I might improve my model accuracy?
You can augment only the training data.
The following code
test_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
should be
test_datagen = ImageDataGenerator(rescale = 1./255)
Firstly, I assume you are building your model from scratch. As such training on fewer epochs(I assume you would not have trained your model for more than 1000 epochs), will not help as the network would not evolve completely because the representations would not have been completely learnt in so few epochs when you train a model from scratch. You can try increasing the number of epochs to around 10000 and see. Rather why not try and use transfer learning for the same, additionally you can also using feature extraction and fine tuning or both using a pre trained convnet. For reference you can have a look at chapter 5 in the book by Francois Chollet titled Deep Learning with Python.
I had same problem with another dataset and I replaced the flatten layer with globalAveragepooling and it solved the problem.
I'm not sure this is going to work for you but as my model has a structure similar to yours, I think this can help you. But the difference is that I trained my model for 3 classes.