I am currently creating a machine learning model with the ultimate goal of deploying the model in an iOS app. The app would be used in the field, where light conditions are highly variable compared to the testing and training set.
Would adding a high channel_shift_range to my image data generator improve my model's ability to recognize the images even when the light conditions are highly variable?
I am currently using
train_datagen = ImageDataGenerator(
rotation_range=360,
width_shift_range=0.2,
height_shift_range=0.2,
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
channel_shift_range=100,
data_format=ch_format,
brightness_range=(0.75, 1.25),
fill_mode='nearest')
test_datagen = ImageDataGenerator(rescale = 1./255,
data_format=ch_format)
for my data generators and using flow_from_directory to load my images into the model.
On a conceptual level, would this work and create the desired results?
Also, if I added a channel_shift_range to the test_datagen would it more accurately reflect my model's performance in variable light situations?
Thank you for any help!
Normally, you would want to apply all sorts of data augmentation, not only channel_shift_range.
Note that, on principle, you should not touch your test dataset. In your case, if you apply a kind of augmentation on the training set, you do not want to apply it also on the test set.
The idea of using the augmentation is to provide the model with many examples, so that it is robust when it goes 'to production'. If you augmented the true test dataset in the exact way you augment the training set, you would just 'mini-cheat'; as you know you gave a lot of examples with, say channel_shift_range with value K to your training set, giving the same exact value to the test set would be just be almost like copying training data to your test data; you do not want to do that.
Ensure that you use a numerous and relevant augmentations in your case (for instance, color shift when comparing apples and oranges is not a wise augmentation).
Related
Hello everyone so I made this cnn model.
My data:
Train folder->30 classes->800 images each->24000 all together
Validation folder->30 classes->100 images each->3000 all together
Test folder->30 classes -> 100 images each -> 3000 all together
-I've applied data augmentation. ( on the train data)
-I got 5 conv layers with filters 32->64->128->128->128
each with maxpooling and batch normalization
-Added dropout 0.5 after flattening layers
Train part looks good. Validation part has a lot of 'jumps' though. Does it overfit?
Is there any way to fix this and make validation part more stable?
Note: I plann to increase epochs on my final model I'm just experimenting to see what works best since the model takes a lot of time in order to train. So for now I train with 20 epochs.
I've applied data augmentation (on the train data).
What does this mean? What kind of data did you add and how much? You might think I'm nitpicking, but if the distribution of the augmented data is different enough from the original data, then this will indeed cause your model to generalize poorly to the validation set.
Increasing your epochs isn't going to help here, your training loss is already decreasing reasonably. Training your model for longer is a good step if the validation loss is also decreasing nicely, but that's obviously not the case.
Some things I would personally try:
Try decreasing the learning rate.
Try training the model without the augmented data and see how the validation loss behaves.
Try splitting the augmented data so that it's also contained in the validation set and see how the model behaves.
Train part looks good. Validation part has a lot of 'jumps' though. Does it overfit?
the answer is yes. The so-called 'jumps' in the validation part may indicate that the model is not generalizing well to the validation data and therefore your model might be overfitting.
Is there any way to fix this and make validation part more stable?
To fix this you can use the following:
Increasing the size of your training set
Regularization techniques
Early stopping
Reduce the complexity of your model
Use different hyperparameters like learning rate
My validation accuracy is stuck at 50% while my training accuracy manages to converge to 100%. The pitfall is that i have very few data: 46 images in train set and 12 in validation set.
Therefore, I am augmenting my data while training but i am running out of data too early. and as i saw from previous answers that i should specify steps_per_epoch.
however, using steps_per_epoch=46/batch_size is not returning that much of iteration (maximum of 10 if i specify a very low batch size).
I assume data augmentation is not being applied? How can i be sure my data is indeed being augmented? Below is my data augmentation code:
gen=ImageDataGenerator(rotation_range=180,
horizontal_flip=True,
vertical_flip=True,
)
train_batches=gen.flow(
x=x_train,
y=Y_train,
batch_size=5,
subset=None,
shuffle=True
)
val_batches=gen.flow(
x=x_val,
y=Y_val,
batch_size=3,
subset=None,
shuffle=True
)
history= model.fit(
train_batches,
batch_size=32,
# steps_per_epoch=len(x_train)/batch_size,
epochs=50,
verbose=2,
validation_data=val_batches,
validation_steps=len(x_val)/batch_size)
I will really appreciate your help!
I think the mistake is not in your code.
You have a very small dataset, you are using only 2 augmentations, and (I assume) you initialize your model with random weights. Your model expectedly overfits.
Here are a couple of ideas that may help you:
Add more argumentations. Vertical and horizontal flips - are just not enough (with your small dataset). Think about crops, rotations, color changes etc. BTW here is a good tutorial on image augmentation where you'll find more ideas on what types of data augmentation you can use for your task: https://notrocketscience.blog/complete-guide-to-data-augmentation-for-computer-vision/
Transfer learning - is a must-do for small datasets. If you are using popular/default architecture, PyTorch and Tensorflow allow you to load model weights trained on ImageNet, for instance. If your architecture is custom - download some open-source dataset (better similar to your task) and pretrain model with this data.
Appropriate validation. Consider n-fold cross-validation, because a fixed train and test set is not a good idea for the small datasets. Your validation accuracy may be low by chance (for instance, all "hard" images are in the test set), but not because the model is bad.
Let me know if it helps!
When You use ImageDataGenerator, there are lots of preprocessing functions given and we can pass our own preprocessing input too. Let us suppose we use MobileNet as a pre-trained model and we'll use the corresponding preprocesss_input function. Let us suppose we init the class as below:
tf.keras.preprocessing.image.ImageDataGenerator(
featurewise_center=True,featurewise_std_normalization=True,zca_whitening=True,
rotation_range=13, width_shift_range=0.13,height_shift_range=0.2, shear_range=0.06, zoom_range=0.1,
horizontal_flip=True, vertical_flip=True,
rescale=1./255.,preprocessing_function=preprocess_input)
What will be the sequence of transformations here? Whether the rescaling will happen first or the preprocess_function or such. If we use ZCA, the covariance shift will result in range change and then there will be no meaning of using rescale. Same will happen with std_norm. Also, if we use preprocess_function, different models use different such as -1 to 1 or 0-1 range so the rescale won't be affective. Different sequence combinations will lead in different statistics for the image.
For the other transformations like zoom, crop, rotation etc,
Will these happen all at once? OR the class will select one transformation at a time?
If it is selecting all at once, zooming -> flipping -> Shifting -> rotating -> shearing will lead to an alien object in the image. Where doing one or more in random order makes sense for the cases.
Can someone please shed some light on the inner workings of the order of the functions?
Whether the rescaling will happen first or the preprocess_function or
such
Rescaling will be applied after the transformations are done. preprocess_function after augmention.
For featurewise functionalities, you need to fit the generator to the data. From the docs:
datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True)
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)
Will these happen all at once? OR the class will select one
transformation at a time?
Transformations are applied randomly on each image. preprocess_function will be applied on each image after the augmention completed.
If it is selecting all at once, zooming -> flipping -> Shifting ->
rotating -> shearing will lead to an alien object in the image. Where
doing one or more in random order makes sense for the cases.
Exactly, they will seem awkard. But you specify ranges when using ImageDataGenerator. So transformations will be applied in that range, randomly.
Example:
generator = ImageDataGenerator(
zoom_range=0.9, horizontal_flip=True, vertical_flip=True, rotation_range=13, width_shift_range=0.13,height_shift_range=0.2, shear_range=0.6,
)
Will generate:
It is clear that not every image is zoomed with a factor of 0.9 or vice versa.
I have to classify between Real, Fake and Other images but I only have dataset of Real and Fake Faces, how do I add 'other' class, that is neither Real nor Fake face ?
This is how I loaded my dataset
TRAINING_DIR = "Dataset\Training Data"
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(TRAINING_DIR,
batch_size=16,
target_size=(300, 300))
and this is my output
Found 1944 images belonging to 2 classes.
Real Face 2. Fake Face 3. Other Object
There is this machine learning competition and they told us to add "other" class. and they didn't provide data, so that's why I was asking
Does this mean you are not allowed to use any additional data? If you can, take some other images that are not faces. Learn a second, separate model M2 that has two classes: FACE and OTHER. For this model, label all of your face images (all real and fake ones together) as FACE.
Train your original model M1 the way you are doing already, with the two classes REAL and FAKE.
After training those two models, follow a decision process such as this one:
For an input image `I`,
Does `M2` predict that the input is a `FACE`?
|--Yes: Does `M1` predict the image is `REAL`?
|--Yes: Output "real image".
|--No: Output "fake image".
|--No: Output "other"
If you cannot use any additional data, try Andrey's answer or look into methods that can detect out-of-distribution inputs.
You can predict based on the output of your network. If it predicts the first class with more than 90% probability - then it is the first class. If less then 10% - then it is the second. Otherwise - it is "Other"
I am fairly new to ML and am currently implementing a simple 3D CNN in python using tensorflow and keras. I want to optimize based on the AUC and would also like to use early stopping/save the best network in terms of AUC score. I have been using tensorflow's AUC function for this as shown below, and it works well for the training. However, the hdf5 file is not saved (despite the checkpoint save_best_only=True) and hence I cannot get the best weights for the evaluation.
Here are the relevant lines of code:
model.compile(loss='binary_crossentropy',
optimizer=keras.optimizers.Adam(lr=lr),
metrics=[tf.keras.metrics.AUC()])
model.load_weights(path_weights)
filepath = mypath
check = tf.keras.callbacks.ModelCheckpoint(filepath, monitor=tf.keras.metrics.AUC(), save_best_only=True,
mode='auto')
earlyStopping = tf.keras.callbacks.EarlyStopping(monitor=tf.keras.metrics.AUC(), patience=hyperparams['pat'],mode='auto')
history = model.fit(X_trn, y_trn,
batch_size=bs,
epochs=n_epochs,
verbose=1,
callbacks=[check, earlyStopping],
validation_data=(X_val, y_val),
shuffle=True)
Interestingly, if I only change monitor='val_loss' in the early stopping and checkpoint (not the 'metrics' in model.compile), the hdf5 file is saved but obviously gives the best result in terms of validation loss. I have also tried using mode='max' but the problem is the same.
I would very much appreciate your advise, or any other constructive ideas how to work around this problem.
Turns out that even if you add a non-keyword metric, you still need to use its handle to refer to in when you want to monitor it. In your case you can do this:
auc = tf.keras.metrics.AUC() # instantiate it here to have a shorter handle
model.compile(loss='binary_crossentropy',
optimizer=keras.optimizers.Adam(lr=lr),
metrics=[auc])
...
check = tf.keras.callbacks.ModelCheckpoint(filepath,
monitor='auc', # even use the generated handle for monitoring the training AUC
save_best_only=True,
mode='max') # determine better models according to "max" AUC.
if you want to monitor the validation AUC (which makes more sense), simply add val_ in the beginning of the handle:
check = tf.keras.callbacks.ModelCheckpoint(filepath,
monitor='val_auc', # validation AUC
save_best_only=True,
mode='max')
Another problem is that you ModelCheckpoint is saving the weights based on the minimum AUC instead of the max, which you want.
This can be changed by setting mode='max'.
What does mode='auto' do?
This setting essentially checks if the argument of monitor contains 'acc' and sets it to max. In any other case it sets uses mode='min', which is what is happening in your case.
You can confirm this here
The answer posted by Djib2011 should solve your problem. I just wanted to address the use of early stopping. Typically this is used to stop training when over fitting starts to cause the loss to increase. I think it is more effective to address the over fitting issue directly which should enable you to achieve a lower loss. You did not list your model so it is not clear how to address over fitting but some simple guidelines are as follows. If you havee several dense hidden layers at the top of the model delete most of them and just keep the final top dense layer. The more complex the model the more it is prone to over fitting. If that leads to lower training accuracy then keep the layers but add dropout layers. You might also try using regularization in the hidden dense layers. I also find it is beneficial to use the callback ReduceLROnPlateau. Set it up to monitor AUC and reduce the learning rate if it fails to improve.