I'm new with TFLearn.
I was studying this introduction tutorial to TFLearn, in which a fixed amount of epochs is set. However I would like to know if it is possible to use the combination learning_rate and accuracy to determine the end of the network training ...
for example: according to the accuracy decrease or increase the learning_rate ... or according to the accuracy stop the training.
# Build neural network
net = tflearn.input_data(shape=[None, 6])
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net)
# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(data, labels, n_epoch=10, batch_size=16, show_metric=True)
:)
Look into http://tflearn.org/models/dnn/ and best_checkpoint_path and best_val_accuracy. The parameters will save your best checkpoint.
If you want to stop the training, you have to program a callback yourself to stop the training. Here is a nice tutorial about early stopping with TFlearn: http://mckinziebrandon.me/TensorflowNotebooks/2016/11/28/early-stop-solution.html
Related
Hey guys I am new to machine learning and I am running this code from https://machinelearningmastery.com/how-to-develop-a-convolutional-neural-network-to-classify-photos-of-dogs-and-cats/
I want to understand why my val_loss starts low and then increasing. is this overfitting or under-fitting? also what can I use to improve the val_loss so it will give a better fit? in the blog post his cross entropy plot is a lot different from mine.
def define_model():
# load model
model = VGG16(include_top=False, input_shape=(224, 224, 3))
# mark loaded layers as not trainable
for layer in model.layers:
layer.trainable = False
# add new classifier layers
flat1 = Flatten()(model.layers[-1].output)
class1 = Dense(128, activation='relu', kernel_initializer='he_uniform')(flat1)
output = Dense(1, activation='sigmoid')(class1)
# define new model
model = Model(inputs=model.inputs, outputs=output)
# compile model
opt = SGD(lr=0.001, momentum=0.9)
model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])
return model
# plot diagnostic learning curves
def summarize_diagnostics(history):
# plot loss
pyplot.subplot(211)
pyplot.title('Cross Entropy Loss')
pyplot.plot(history.history['loss'], color='blue', label='train')
pyplot.plot(history.history['val_loss'], color='orange', label='test')
# plot accuracy
pyplot.subplot(212)
pyplot.title('Classification Accuracy')
pyplot.plot(history.history['accuracy'], color='blue', label='train')
pyplot.plot(history.history['val_accuracy'], color='orange', label='test')
# save plot to file
filename = sys.argv[0].split('/')[-1]
pyplot.savefig(filename + '_plot.png')
pyplot.close()
# run the test harness for evaluating a model
def run_test_harness():
# define model
model = define_model()
# create data generator
datagen = ImageDataGenerator(featurewise_center=True)
# specify imagenet mean values for centering
datagen.mean = [123.68, 116.779, 103.939]
# prepare iterator
train_it = datagen.flow_from_directory('dataset_dogs_vs_cats/train/',
class_mode='binary', batch_size=64, target_size=(224, 224))
test_it = datagen.flow_from_directory('dataset_dogs_vs_cats/test/',
class_mode='binary', batch_size=64, target_size=(224, 224))
# fit model
history = model.fit_generator(train_it, steps_per_epoch=len(train_it),
validation_data=test_it, validation_steps=len(test_it), epochs=10, verbose=1)
# evaluate model
_, acc = model.evaluate_generator(test_it, steps=len(test_it), verbose=1)
print('> %.3f' % (acc * 100.0))
# learning curves
summarize_diagnostics(history)
model.save('Transfer_Learning_Model.h5')
# entry point, run the test harness
run_test_harness()
A number of issues. The VGG model was trained with pixel values from -1 to +1 so you need to add the following
def scaler(x):
y=x/127.5-1
return y
datagen = ImageDataGenerator(preprocessing_function=scaler)
I would remove featurewise_center=True. If you use it you have to run fit on the train set.
In model.fit you have steps_per_epoch=len(train_it) and validation_steps=len(test_it).
Since you set a batch_size of 64 in the generators these values should be
steps_per_epoch=len(train_it.labels)//64 and validaion_steps=len(test_it)/64. Actually in model.fit leave these parameters out. Model.fit will calculate the right values internally. Your validation loss curve indicates a small degree of over fitting. Add a drop out layer after the class1 layer. Set the dropout rate to something like.2.
If you want to push for the highest degree of accuracy I recommend you incorporate two Keras callbacks. The EarlyStopping callback monitors validation loss and halts training if the loss fails to reduce after 'patience' number of consecutive epochs. Setting restore_best_weights=True will load the weights for the epoch with the lowest validation loss so you don't have to save then reload the weights. Set epochs to a large number to ensure this callback activates. Use the the keras callback ReduceLROnPlateau to automatically adjust the learning rate based on validation loss. The documentation for the callbacks is located here. The code I use is shown below
es=tf.keras.callbacks.EarlyStopping( monitor="val_loss", patience=3,
verbose=1, restore_best_weights=True)
rlronp=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1,
verbose=1)
callbacks=[es, rlronp]
in model.fit add callbacks=callbacks
I am training my model with a dataset of 200 images. I have created a binary classification CNN that looks like this one:
classifier = Sequential()
# Adding a first convolutional layer
classifier.add(Convolution2D(48, 3, input_shape = (320, 320, 3), activation = 'relu'))
classifier.add(MaxPooling2D())
# Adding a second convolutional layer
classifier.add(Convolution2D(48, 3, activation = 'relu'))
classifier.add(MaxPooling2D())
# Adding a third convolutional layer
classifier.add(Convolution2D(48, 3, activation = 'relu'))
classifier.add(MaxPooling2D())
#Flattening
classifier.add(Flatten())
#Full connected
classifier.add(Dense(256, activation = 'relu'))
#Full connected
classifier.add(Dense(256, activation = 'sigmoid'))
#Dropout
classifier.add(Dropout(0.5))
#Full connected
classifier.add(Dense(1, activation = 'sigmoid'))
# Compiling the CNN
opt = keras.optimizers.Adam(learning_rate=0.001)
classifier.compile(optimizer = opt, loss = 'binary_crossentropy', metrics = ['accuracy'])
classifier.summary()
I am also using Image Data Augmentation and Early Stopping based on val_accuracy with a patience of 10.
My results are the following:
Result graph validation accuracy
The best validation accuracy I get is 0.9231 at the 21st epoch. Should I stop the training with a custom callback once I surpass 92% or is it a bad practice?
Would it be a good practice to set a custom callback that stops training
The best practice here is to save the model every time the validation accuracy hits a maximum, but to keep training. Alternatively, you could save a model after each epoch, and choose the best one to use by checking the validation graph (I'd suggest epoch 11 here. After 11 the validation graph is just oscillating, which is mostly noise).
Finally, 200 images is rarely enough to get good results. You want thousands or tens of thousands at least. Even your validation set should have at least 100 images so that even minor changes to the model show smooth changes in the validation curve. You should also consider adding some data augmentation if you aren't doing it already.
I am training the Mask R-CNN model.
I saved the weights after training for 2 epochs the 'head' and I want to continue from epoch three. But the model.train() function does not have initial_epoch argument as model.fit in a Sequential model for example.
I have the following code but if I run it with the loaded weights it starts from the first epoch and I don't want that:
EPOCHS = [1, 3, 5, 8]
model.train(dataset_train, dataset_val,
learning_rate = LEARNING_RATE,
epochs = EPOCHS[1],
layers = 'all',
augmentation = augmentation)
I would appreciate if someone can tell me what is the substitute for initial_epoch in my case.
After first 2 epochs of fitting your model changed its weights. So, when you call fit once again the model will continue training. Your progress won't lost
New to the field of deep learning and currently working on this competition for predicting the earthquake damage to buildings.
The model I created starts at an accuracy of .56 but remains at this for any number of epochs i let it run. When finished, the model only predicts one of the three classes (which I one hot encoded into a dataframe with three columns). Changing the number of layers, optimizers, data preparation, dropout wont change anything. Even trying to overfit my model with the over-parameterization of the neural network will still have the same accuracy and a non-learning model.
What am I doing wrong?
This is my code:
model = keras.models.Sequential()
model.add(keras.layers.Dense(64, input_dim = 85, activation = "relu"))
keras.layers.Dropout(0.3)
model.add(keras.layers.Dense(128, activation = "relu"))
keras.layers.Dropout(0.3)
model.add(keras.layers.Dense(256, activation = "relu"))
keras.layers.Dropout(0.3)
model.add(keras.layers.Dense(512, activation = "relu"))
model.add(keras.layers.Dense(3, activation = "softmax"))
adam = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(optimizer = adam,
loss='categorical_crossentropy',
metrics = ['accuracy'])
history = model.fit(traindata, trainlabels,
epochs = 5,
validation_split = 0.2,
verbose = 1,)
There's nothing visually wrong with your model, but it may be too haevy to learn any useful features.
Try normalizing your input with https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
Start with only 2 layers, and a few numbers of neurons.
Increase batch_size and try learning_rate scheduling.
Observe the validation_accuracy, stop when it starts to overfit.
Finally, for a 3-class classification, 56% accuracy is better than baseline, remmeber it's a competition so the data is not dummy playground data which you can expect to get a 90% accuracy with an MLP in the first try.
Finally, try hyperparameter optimization with tuner.
I want to combine the prediction output of one frozen model in a training phase of another.
I have tried using different graph sessions but it resets the default graph in the training phase.
predictions = model1.model(input1, input2, mode)
predictions2 = model2.predict(predictions)
loss1 = mean_squared_error(predictions, labels)
loss2 = mean_squared_error(input2, predictions2)
total_loss = loss1+loss2
optimizer.minimize(total_loss)
ValueError: Tensor Tensor("output_layer/BiasAdd:0", shape=(?, 100), dtype=float32) is not an element of this graph
I just figured this out!
In the tensorflow's estimator framework, load the model in the 'model_fn' of your Estimator space with the attribute:
'keras_model.trainable=False'
eg snippet:
def model_fn(inputs):
.......
#some operations
.......
model2=load_model('frozen_model.h5')
model2.trainable=False
model2.summary()
predictions= model2(inputs=predictions)