How to view the image from test generator to see if the prediction is correct or not - tensorflow

I am training a fruit classification model. As of right now my classes are:
['Fresh Apples', 'Fresh Bananas', 'Fresh Oranges']
I am using train, validation, and test generators using ImageDataGenerator and flow_from_directory. I have trained the model and am now wanting to feed the test generator into the model to see the performance of the model. Right now I only have 2 images in the test generator. I have the following code to make predictions:
predictions = tuned_model.predict(test_generator)
score = tf.nn.softmax(predictions[0])
print(
'This image most likely belongs to {} with a {:.2f} percent
confidence.'.format(
class_names[np.argmax(score)], 100 * np.max(score)
)
)
And am getting the following as a result:
This image most likely belongs to Fresh Apples with a 46.19 percent confidence.
Yes it is low accuracy, I have only trained for 10 epochs lol. BUT, is there a way I can see which image is being tested? Or a way to know if this prediction is the correct prediction?
EDIT:
Including code of generators...
generator = ImageDataGenerator(
rotation_range=45,
rescale=1./255,
horizontal_flip=True,
vertical_flip=True,
validation_split=.2
)
train_generator = generator.flow_from_directory(
train_path,
target_size=(im_height, im_width),
batch_size = batch_size,
subset='training'
)
validation_generator = generator.flow_from_directory(
train_path,
target_size=(im_height, im_width),
batch_size=batch_size,
subset='validation'
)
test_generator = generator.flow_from_directory(
test_path,
target_size= (im_height, im_width),
batch_size= batch_size,
)
In terms of my class labels, as of right now I simply hard coded them
class_names = ['Fresh Apples', 'Fresh Bananas', 'Fresh Bananas']
I know I should probably import os and create labels based on file structure, but I will do this later unless I absolutely need to.

I assume when you created the test generator you set shuffle=False in flow_from_directory. Then use
files=test_generator.filenames
class_dict=test_generator.class_indices # a dictionary of the form class name: class index
rev_dict={}
for key, value in class_dict.items()
rev_dict[value]=key # dictionary of the form class index: class name
files is a list of filename in the order the files were presented for prediction.
Then do
predictions = tuned_model.predict(test_generator)
Then iterate through the predictions
for i, p in enumerate(predictions)
index=np.argmax(p)
klass=rev_dict[index]
prob=p[index]
print('for file ', files[i], ' predicted class is ', klass,' with probability ',prob)
Of course you could also display the image as well

Related

How to "update" from module tf.keras.preprocessing.image to tf.keras.utils.image_dataset_from_directory for features extraction

This code part is common to both "problematic" codes below:
BATCH_SIZE = 32
IM_DIR = '/content/drive/My Drive/101_ObjectCategories/'
IM_HEIGHT = 224
IM_WIDTH = 224
NUM_IM = 8686
NUM_EPOCHS = int(math.ceil(NUM_IM / BATCH_SIZE))
#load pre-trained base model
model = ResNet50(weights='imagenet',
include_top=False,
input_shape=(IM_WIDTH, IM_HEIGHT, CH),
pooling='max')
The following code I successfully use to extract features of a set of images using module tf.keras.preprocessing.image.
datagen = tf.keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)
dataset = datagen.flow_from_directory(IM_DIR,
target_size=(IM_HEIGHT, IM_WIDTH),
class_mode=None,
shuffle=False)
feature_list = []
feature_list = model.predict(dataset, num_epochs)
Thereafter I train a simple nearest-neighbor model using brute-force algorithm and I'm able to find three other images that are really similar to the query image as you can see below:
But as pointed in the documentation this preprocessing module is deprecated.
So, I would like to "update" the code as suggested in the documentation: "Prefer loading data with tf.keras.utils.image_dataset_from_directory, and then transforming the output tf.data.Dataset with preprocessing layers".
For that I'm trying the following:
#load images
dataset = tf.keras.utils.image_dataset_from_directory(
IM_DIR,
labels='inferred', #'inferred', None
label_mode='categorical', #'int', 'categorical', 'binary' or None
class_names=None,
color_mode='rgb', #'grayscale', 'rgb' or 'rgba'
batch_size=BATCH_SIZE,
image_size=(IM_HEIGHT, IM_WIDTH),
shuffle=True,
seed=51719,
validation_split=None,
subset=None, #'training', 'validation' or 'both'
interpolation='bilinear', #'bilinear', 'nearest', 'bicubic', 'area', 'lanczos3', 'lanczos5', 'gaussian' or 'mitchellcubic'
follow_links=False,
crop_to_aspect_ratio=False
)
#"transforming the output with preprocessing layers"
#rescale (normalize) dataset
rescale_layer = tf.keras.layers.Rescaling(1./255)
rescaled_dataset = dataset.map(lambda x, y: (rescale_layer(x), y))
im_batch, labels_batch = next(iter(rescaled_dataset))
#configure dataset for performance
#https://www.tensorflow.org/tutorials/load_data/images#configure_the_dataset_for_performance
AUTOTUNE = tf.data.AUTOTUNE
tuned_dataset = dataset.cache().prefetch(buffer_size=AUTOTUNE)
And now I begin with the features extraction
#features extraction
#https://www.tensorflow.org/api_docs/python/tf/keras/Model#predict
feature_list = []
feature_list = model.predict(
tuned_dataset,
batch_size=None,
verbose='auto',
steps=None,
callbacks=None,
max_queue_size=10,
workers=1,
use_multiprocessing=False
)
#save features
pickle.dump(
feature_list,
open(DATA_DIR + 'features.pickle', 'wb'))
After that I do the same and train the nearest neighbor model with this features, but the results are catastrophic as you can see below:
What I'm doing so wrong that I have such different results?
== EDIT 1 ==
Answering #DWKOT using the same image we have following results:
#Query image with first code
im_idx = 75
distances, indices = neighbors.kneighbors([feature_list[im_idx]])
plt.imshow(mpimg.imread(filenames[im_idx]), interpolation='lanczos')
#Similar image
plt.imshow(mpimg.imread(filenames[indices[0][1]]), interpolation='lanczos')
And the code that give us the distance to the 5 nearest neighbors:
for i in range(5):
print(distances[0][i])
With the following results:
0.0
185.60701
185.75049
195.71657
196.4056
With the second code we have following result for query / similar image:
/
And following results for the first five "similar" images:
0.0
0.81401
0.88622
0.92734
0.9346
What is also strange as I would expect similar images having values next to zero and different ones far from zero...

How can I fine-tune EfficientNetB3 model and retain some of its exisiting labels?

I've tested EfficientNetB3 model (trained on ImageNet) on my large image set and it recognizes some classes of images that I have with varying accuracy, the others are not recognized at all.
For example, it does a great job for school buses: ('n04146614', 'school_bus') and a decent job for ('n04487081', 'trolleybus'), ('n02701002', 'ambulance'), ('n03977966', 'police_van').
So I would like to keep these labels and feed more images to the model to improve their detection rate.
At the same time, while it detects police vans, it completely misses other police vehicles, so I would have to create new labels for them.
How should I approach? Is this possible in one training session?
Using a model trained on imagnet will do a reasonably good job of identify images if they were included in the original imagenet dataset. If they were not present as a class the model will perform very poorly. What you normally do is customize the model for the unique classes in your dataset. This process is called transfer learning. First you have to decide what classes you want to have and gather the appropriate images associated with each class. For examples lets say you have the classes police car, school bus, fire trucks, garbage truck and delivery van. So you need to gather the appropriate images for each class. Typically you need about 120 to 150 images for each class as a minimum. So we now have 5 classes. Create a single directory call is sdir. Below sdir create 5 subdirectories one for each class. Name these as police car, school bus, etc. Now put the images into their respective subdirectories. Now the function below can be used to split the dataset into three datasets called train_df, test_df and valid_df.
def preprocess (sdir, trsplit, vsplit):
filepaths=[]
labels=[]
classlist=os.listdir(sdir)
for klass in classlist:
classpath=os.path.join(sdir,klass)
if os.path.isdir(classpath):
flist=os.listdir(classpath)
for f in flist:
fpath=os.path.join(classpath,f)
filepaths.append(fpath)
labels.append(klass)
Fseries=pd.Series(filepaths, name='filepaths')
Lseries=pd.Series(labels, name='labels')
df=pd.concat([Fseries, Lseries], axis=1)
dsplit=vsplit/(1-trsplit)
strat=df['labels']
train_df, dummy_df=train_test_split(df, train_size=trsplit, shuffle=True, random_state=123, stratify=strat)
strat=dummy_df['labels']
valid_df, test_df= train_test_split(dummy_df, train_size=dsplit, shuffle=True, random_state=123, stratify=strat)
print('train_df length: ', len(train_df), ' test_df length: ',len(test_df), ' valid_df length: ', len(valid_df))
print(list(train_df['labels'].value_counts()))
return train_df, test_df, valid_df
Now call the function
sdir=r'C:\sdir'
trsplit=.8 # percent of images to use for training
vsplit=.1 # percent of images to use for validation
train_df, test_df, valid_df= preprocess(sdir,trsplit, vsplit)
Now you need to create 3 generators using ImageDataGenerator.flow_from_dataframe. Documentation is here.
channels=3
batch_size=20 # set batch size based on model complexity and sie of images
img_shape=(img_size[0], img_size[1], channels)
# calculate test_batch_size and test_steps so that test_batch_size X test_steps = number of test images
# this ensures you go through the test set exactly once when doing predictions on the test set
length=len(test_df)
test_batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=80],reverse=True)[0]
test_steps=int(length/test_batch_size)
print ( 'test batch size: ' ,test_batch_size, ' test steps: ', test_steps)
trgen=ImageDataGenerator(horizontal_flip=True)
tvgen=ImageDataGenerator()
msg=' for the train generator'
print(msg, '\r', end='')
train_gen=trgen.flow_from_dataframe( train_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',
color_mode='rgb', shuffle=True, batch_size=batch_size)
msg=' for the test generator'
print(msg, '\r', end='')
test_gen=tvgen.flow_from_dataframe( test_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',
color_mode='rgb', shuffle=False, batch_size=test_batch_size)
msg=' for the validation generator'
print(msg, '\r', end='')
valid_gen=tvgen.flow_from_dataframe( valid_df, x_col='filepaths', y_col='labels', target_size=img_size, class_mode='categorical',
color_mode='rgb', shuffle=True, batch_size=batch_size)
classes=list(train_gen.class_indices.keys())
class_count=len(classes)
train_steps=int(np.ceil(len(train_gen.labels)/batch_size))
labels=test_gen.labels
Now create your model. A suggested model is shown below using EfficientNetB3
def make_model(img_img_size, class_count,lr=.001, trainable=True):
img_shape=(img_size[0], img_size[1], 3)
model_name='EfficientNetB3'
base_model=tf.keras.applications.efficientnet.EfficientNetB3(include_top=False, weights="imagenet",input_shape=img_shape, pooling='max')
base_model.trainable=trainable
x=base_model.output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x = Dense(256, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
bias_regularizer=regularizers.l1(0.006) ,activation='relu')(x)
x=Dropout(rate=.45, seed=123)(x)
output=Dense(class_count, activation='softmax')(x)
model=Model(inputs=base_model.input, outputs=output)
model.compile(Adamax(learning_rate=lr), loss='categorical_crossentropy', metrics=['accuracy'])
return model, base_model # return the base_model so the callback can control its training state
Now call the function
model, base_model=make_model(img_size, class_count)
Now you can train your model
history=model.fit(x=train_gen, epochs=epochs, verbose=0, validation_data=valid_gen,
validation_steps=None, shuffle=False, initial_epoch=0)
After training you can evaluate your models performance on the test set
loss, acc=model.evaluate(test_gen, steps=test_steps)

CNN with imbalanced data stuck with 70% testing accuracy

I'm working on image classification task for diabetic retinopathy with fundus image data. There are 5 classes. The data distribution is 1805 images (class 1), 370 images (class 2), 999 images (class 3), 193 images (class 4), 295 images (class 5).
Here are the steps that I have tried to run:
Preprocessing (resized 224 * 224)
The divide of train and test data is 85% : 15%
x_train, xtest, y_train, ytest = train_test_split(
x_train, y_train,
test_size = 0.15,
random_state=SEED,
stratify = y_train
)
Data agumentation
ImageDataGenerator(
zoom_range=0.15,
fill_mode='constant',
cval=0.,
horizontal_flip=True,
vertical_flip=True,
)
Training with the ResNet-50 model and cross-validation
def getResNet():
modelres = ResNet50(weights=None, include_top=False, input_shape= (IMAGE_HEIGHT,IMAGE_HEIGHT, 3))
x = modelres.output
x = GlobalAveragePooling2D()(x)
x = Dense(5, activation= 'softmax')(x)
model = Model(inputs = modelres.input, outputs = x)
return model
num_folds = 5
skf = StratifiedKFold(n_splits = 5, shuffle=True, random_state=2021)
cvscores = []
fold = 1
for train, val in skf.split(x_train, y_train.argmax(1)):
print('Fold: ', fold)
Xtrain = x_train[train]
Xval = x_train[val]
Ytrain = y_train[train]
Yval = y_train[val]
data_generator = create_datagen().flow(Xtrain, Ytrain, batch_size=32, seed=2021)
model = getResNet()
model.compile(loss='categorical_crossentropy',
optimizer=Adam(lr=0.0001),
metrics=['accuracy'])
with tf.compat.v1.device('/device:GPU:0'):
model_train = model.fit(data_generator,
validation_data=(Xval, Yval),
epochs=30, batch_size = 32, verbose=1)
model_name = 'cnn_keras_aug_Fold_'+str(fold)+'.h5'
model.save(model_name)
scores = model.evaluate(xtest, ytest, verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
cvscores.append(scores[1] * 100)
fold = fold +1
The maximum results I got from this method were training accuracy of 81.2%, validation accuracy of 72.2%, and test accuracy of 70.73%.
Can anyone give me an idea to improve the model so that I can get the test accuracy above 90% as possible?
Later, I will use this model as a pre-trained model to train diabetic retinopathy data as well but from other sources.
BTW, I've tried replacing my preprocessing with this method:
def preprocessing(path):
image = cv2.imread(path)
image = crop_image_from_gray(image)
green = image[:,:,1]
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
cl = clahe.apply(green)
image[:,:,0] = image[:,:,0]
image[:,:,2] = image[:,:,2]
image[:,:,1] = cl
image = cv2.resize(image, (224,224))
return image
I've also tried to replace my model with VGG16, EfficientNetB0. However, none of that had much effect on my results. I'm still stucked with about 70% accuracy.
Please help me come up with ideas to improve my modeling results. I hope.
Your training accuracy is 81.2%. It is generally impossible to have testing accuracy higher that training accuracy, i.e. with current setup you will not achieve 90%.
However, your validation (and also testing) accuracy is about 70-72%. I can suggest that on your small dataset your model is overfitting. So if you add model regularization (e.g. dropout), it is possible that the gap between your training and your validation (and test) will decrease. This way you can improve your validation score.
To further increase the score, you need to check your data manually and try to understand which classes contribute the most to the errors and figure out how those errors can be reduced (e.g. updating your preprocessing pipeline).

why am I getting error in transfer learning?

I am training a model for Optical Character Recognition of Gujarati Language. The input image is a character image. I have taken 37 classes. Total training images are 22200 (600 per class) and testing images are 5920 (160 per class). My input images are 32x32
Below is my code:
model = tf.keras.applications.DenseNet121(include_top=False, weights='imagenet', pooling='max')
base_inputs = model.layers[0].input
base_outputs = model.layers[-1].output # NOTICE -1 not -2
prefinal_outputs = layers.Dense(1024)(base_outputs)
final_outputs = layers.Dense(37)(prefinal_outputs)
new_model = keras.Model(inputs=base_inputs, outputs=base_outputs)
from tensorflow.keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=False)
test_datagen = ImageDataGenerator(horizontal_flip = False)
training_set = train_datagen.flow_from_directory('C:/Users/shweta/Desktop/characters/train',
target_size = (32, 32),
batch_size = 64,
class_mode = 'categorical')
test_set = test_datagen.flow_from_directory('C:/Users/shweta/Desktop/characters/test',
target_size = (32, 32),
batch_size = 64,
class_mode = 'categorical')
new_model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
new_model.fit_generator(training_set,
epochs = 25,
validation_data = test_set, shuffle=True)
new_model.save('alphanumeric.mod')
I am getting following output:
Thanks in advance!
First of all, very well written code.
These are some of the things, I have noticed while I was going through the code and tf,keras docs.
I would like to ask what kind of labels have you got beacuse you know categorical_crossentropy expects ONE HOT CODED labels.(Check this).So, if your labels are integers, use sparsecategoricalentropy.
Similar issue
There was post where someone was trying to classsify into 2 and used categorical instead of binary crossentropy. If you want to look at.
Cheers
Let me know how it goes!
PS: #gerry made a very good point and if labels are One hot encoded use categoricalcrossentropy!
The code should be:
model = tf.keras.applications.DenseNet121(include_top=False, weights='imagenet, pooling='max', input_shape=(32,32,3))
base_outputs = model.layers[-1].output
prefinal_outputs = layers.Dense(1024)(base_outputs)
final_outputs = layers.Dense(37)(prefinal_outputs)
new_model = keras.Model(inputs=model.input, outputs=final_outputs)
new_model.compile(Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
Also you should use model.fit in the future. Model.fit can now work with generators and model.fit_generator will be depreciate in future versions of tensorflow. I ran against your dataset and got accurate results in about 10 epochs. Here is some additional advice. It is best to use and adjustable learning rate. The keras callback ReduceLROnPlateau makes this easy to do. Documentation is here. Set it to monitor the validation loss. My use is shown below.
lr_adjust=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1, verbose=1, mode="auto",
min_delta=0.00001, cooldown=0, min_lr=0)
Also I recommend using the callback ModelCheckpoint. Documentation is here. Set it up to monitor validation loss and it will save the weights that achieved the lowest validation loss. My implementation is shown below.
sav_loc=r'c:\Temp' # set this to the path where you want to save the weights
checkpoint=tf.keras.callbacks.ModelCheckpoint(filepath=save_loc, monitor='val_loss', verbose=1, save_best_only=True,
save_weights_only=True, mode='auto', save_freq='epoch', options=None)
callbacks=[checkpoint, lr_adjust]
In model.fit include callbacks=callbacks. When training is completed you want to load these saved weights into the model, then save the model. You can use the saved model to make predictions. Code is below.
model.load_weights(save_loc)
model.save(save_loc)

Why is my confusion matrix "shifted" to the right?

I built a classifier of 4 flower types based on ResNet 50. The accuracy is really high during training, and everything seems good. However, once I plot my confusion matrix, I see that the values are "shifted" to the right instead of in the main diagonal.
What does this mean? Is it a problem with my dataset, or my code?
Here's what I did to use ResNet 50:
def create_model(input_shape, top='flatten'):
if top not in ('flatten', 'avg', 'max'):
raise ValueError('unexpected top layer type: %s' % top)
# connects base model with new "head"
BottleneckLayer = {
'flatten': Flatten(),
'avg': GlobalAvgPooling2D(),
'max': GlobalMaxPooling2D()
}[top]
base = InceptionResNetV2(input_shape=input_shape,
include_top=False,
weights='imagenet')
x = BottleneckLayer(base.output)
x = Dense(NUM_OF_FLOWERS, activation='linear')(x)
model = Model(inputs=base.inputs, outputs=x)
return model
base = ResNet50(input_shape=input_shape, include_top=False)
x = Flatten()(base.output)
x = Dense(NUM_OF_FLOWERS, activation='softmax')(x)
model = Model(inputs=base.inputs, outputs=x)
Confusion Matrix Generation:
# Predict the values from the validation dataset
Y_pred = model.predict_generator(validation_generator, nb_validation_samples // batch_size+1)
# Convert predictions classes to one hot vectors
Y_pred_classes = numpy.argmax(Y_pred, axis = 1)
# Convert validation observations to one hot vectors
Y_true = validation_generator.classes
# compute the confusion matrix
confusion_mtx = confusion_matrix(Y_true, Y_pred_classes)
# plot the confusion matrix
plot_confusion_matrix(confusion_mtx, classes = range(4))
As requested, this is how I created the generators:
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
color_mode='rgb',
class_mode='categorical',
shuffle=True)
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
color_mode='rgb',
class_mode='categorical',
shuffle=False)
Here is an image album of my confusion matrix. Every time I execute model.predict(), the predictions change, always shifting one cell to the right.
Confusion Matrix Album
Yes I imagine it is the code, check your indexing where you create your confusion matrix, it will be off by one
look the validation_generator class. when you use data_generator.flow_from_directory you need see if param shuffle is equal to False like the example above:
val_generator = val_data_generator.flow_from_directory(
test_data_dir,
target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
batch_size=100,
class_mode="binary",
classes=['dog','cat'],
shuffle=False)
because the default param is True and the only shuffle the images and not labels.
This is an interesting problem. It can be fixed by reloading the imagedatagenerator right before you do a model.predict.
So:
validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
color_mode='rgb',
class_mode='categorical',
shuffle=False)
Y_pred = model.predict_generator(validation_generator, nb_validation_samples // batch_size+1)
# Convert predictions classes to one hot vectors
Y_pred_classes = numpy.argmax(Y_pred, axis = 1)
# Convert validation observations to one hot vectors
Y_true = validation_generator.classes
# compute the confusion matrix
confusion_mtx = confusion_matrix(Y_true, Y_pred_classes)
# plot the confusion matrix
plot_confusion_matrix(confusion_mtx, classes = range(4))