I would like to know at what stage testing dataset is used CNNs? Is it used after completion of each batch or one epoch during training or is it used after completion of all the epochs ? I am a bit confused as to how these two processes run together ? Similarly gradient updation is done after each batch or each epoch ?
model.fit_generator(
aug.flow(x_train, y_train, batch_size=BATCH_SIZE),
validation_data=(x_test, y_test),
steps_per_epoch=len(x_train) // BATCH_SIZE,
epochs=EPOCHS, verbose=1, callbacks = callbacks)
From fit_generator it is only clear that images are loaded batch by batch onto memory.
Keras is using validation datasets in the end of every epoch (if you didn't change validation_freq in the fit function). Each epoch your model trains on the whole train dataset and later evaluates itself on the validation dataset
Related
I am new in the deep neuron network world. I tried to train my one model using the TensorFlow Keras toolkit.
I managed to train a model using the fit function. The accuracy, for 50 epochs, was good - around 96% and the model predicts well with new data. The problem is when I try to evaluate the loaded model the results are like the model wasn't trained at all (accuracy around 50%).
I prepare the small test. I evaluate the model after a fit. Then I save the model, load it, and evaluate it once again. The results are so different. I thought that maybe weights aren't loaded properly, but the documentation suggests that save and load functions operate on the whole model. Here is my code:
CNNmodelHistory = model.fit(train_data, batch_size=batch_size, validation_data=test_data, steps_per_epoch=train_data.samples // batch_size,
epochs=echos)
scores = model.evaluate(test_data, verbose=0)
print(f'Test loss: {scores[0]} / Test accuracy: {scores[1]}')
# save the model to disk
model.save('gender_detection.modelTest')
modelLoaded = keras.models.load_model('gender_detection.modelTest')
scores = modelLoaded.evaluate(test_data, verbose=0)
print(f'Test loss: {scores[0]} / Test accuracy: {scores[1]}')
And here are the results:
Do you have any tips on what I am doing wrong?
I am trying to approximate a function that smoothly maps five inputs to a single probability using Keras, but seem to have hit a limit. A similar problem was posed here (Keras Regression to approximate function (goal: loss < 1e-7)) for a ten-dimensional function and I have found that the architecture proposed there, namely:
model = Sequential()
model.add(Dense(128,input_shape=(5,), activation='tanh'))
model.add(Dense(64,activation='tanh'))
model.add(Dense(1,activation='sigmoid'))
model.compile(optimizer='adam', loss='mae')
gives me my best results, converging to a best loss of around 7e-4 on my validation data when the batch size is 1000. Adding or removing more neurons or layers seems to reduce the accuracy. Dropout regularisation also reduces accuracy. I am currently using 1e7 training samples, which took two days to generate (hence the desire to approximate this function). I would like to reduce the mae by another order of magnitude, does anyone have any suggestions how to do this?
I recommend use utilize the keras callbacks ReduceLROnPlateau, documentation is [here][1] and ModelCheckpoint, documentation is [here.][2]. For the first, set it to monitory validation loss and it will reduce the learning rate by a factor(factor) if the loss fails to reduce after a fixed number (patience) of consecutive epochs. For the second also monitor validation loss and save the weights for the model with the lowest validation loss to a directory. After training load the weights and use them to evaluate or predict on the test set. My code implementation is shown below.
checkpoint=tf.keras.callbacks.ModelCheckpoint(filepath=save_loc, monitor='val_loss', verbose=1, save_best_only=True,
save_weights_only=True, mode='auto', save_freq='epoch', options=None)
lr_adjust=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1, verbose=1, mode="auto",
min_delta=0.00001, cooldown=0, min_lr=0)
callbacks=[checkpoint, lr_adjust]
history = model.fit_generator( train_generator, epochs=EPOCHS,
steps_per_epoch=STEPS_PER_EPOCH,validation_data=validation_generator,
validation_steps=VALIDATION_STEPS, callbacks=callbacks)
model.load_weights(save_loc) # load the saved weights
# after this use the model to evaluate or predict on the test set.
# if you are satisfied with the results you can then save the entire model with
model.save(save_loc)
[1]: https://keras.io/api/callbacks/reduce_lr_on_plateau/
[2]: https://keras.io/api/callbacks/model_checkpoint/
I know this a very bad thing to do but I noticed something strange using keras mobilenet :
I use the same data for training and validation set :
train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(IM_WIDTH, IM_HEIGHT),
batch_size=batch_size,
class_mode = "categorical"
)
validation_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(IM_WIDTH, IM_HEIGHT),
class_mode = "categorical"
)
but I don't get the same accuracy on both !
epoch 30/30 - loss: 0.3485 - acc: 0.8938 - val_loss: 1.7545 - val_acc: 0.4406
It seems that I am overfitting the training set compared to the validation set.. but they are supposed to be the same ! How is that possible ?
The training loss is calculated on the fly and only the validation loss is calculated after the epoch is trained. So at the beginning a nearly untrained net will make the training loss worse that it actually is. This effect should vanish in later epochs, since then one epochs mpact on the scoring is not that big anymore.
This behaviour is adressed in keras faq.
If you evaluate both at the end of epoch with a self written callback, they should be the same.
For people reading this after a while :
I still don't understand how this issue happened but it helped a lot working on the batchsize (reducing it).
For research purposes, I am training a neural network that is updating its weights differently depending on the parity of the epoch:
1) If the epoch is even, change the weights of the NN with backpropagation
2) If the epoch is odd, only update the model with update_weights_with_custom_function() therefore freeze the network.
Here is a simplified part of the code that implements this (notice the epochs=1):
for epoch in range(nb_epoch):
if epoch % 2 == 0:
model.trainable = True # Unfreeze the model
else:
model.trainable = False # Freeze the model
model.compile(optimizer=optim, loss=gaussian_loss, metrics=['accuracy'])
hist = model.fit(X_train, Y_train,
batch_size=batch_size,
epochs=1,
shuffle=True,
verbose=1,
callbacks=[tbCallBack, csv_epochs, early_stop],
validation_data=(X_val, Y_val))
if epoch % 2 == 1:
update_weights_with_custom_function()
Problem: after a few epoch, keras throws a ResourceExhaustedError but only with tensorflow, not with theano. It seems that looping over compile() is creating models without releasing them.
Therefore, what should I do? I know that K.clear_session() releases memory but it requires to save the model and reload it (see) which gives me some issues as load_model() in my case does not work out of the box.
I'm also open to other ways to do what I am trying to achieve (i.e. freezing a NN model depending on the parity of the epoch).
Summary: keras with tensorflow backend is throwing a ResourceExhaustedError because I am looping over compile().
As Marcin Możejko pointed out, using eval() is doing exactly what I was trying to achieve.
I added a custom callback (inspiration was here), which avoids the loop over compile()
The problem is now solved, even if the tensorflow issue was not addressed directly.
Using Keras, one typically gets metrics (e.g. accuracy) as part of the progress bar for free. Using the example here:
https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py
After running e.g.
history = model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test))
Keras will start fitting the model, and will show progress output with something like:
3584/60000 [>.............................] - ETA: 10s - loss: 0.0308 - acc: 0.9905
Suppose I wanted to accomplish the same thing using a TensorFlow canned estimator -- extract the current accuracy for a classifier, and display that as part of a progress bar (done by e.g. a SessionRunHook).
It seems like accuracy metrics aren't provided as part of the default set of operations on a graph. Is there a way I can manually add it myself with a session run hook?
(It looks like it's possible to add operations to the graph as part of the begin() hook, but I'm not sure how I can e.g. request the computation of the model accuracy there.)
accuracy is one of the default metrics in canned classifiers. But it will be calculated by Estimator.evaluate call not by Estimator.train. You can create a for loop to do what you want:
for ...
estimator.train(training_data)
metrics = estimator.evaluate(evaluation_data)