Can I plot accuracy, loss... if I didn't assign model.fit to something.
I just wrote model.fit and trained the model.
Thanks
There are 2 ways to do this without a history in keras:
take the text output of the keras training and manually take the loss values of every epoch and do the plot by hand by filling 2 numpy arrays with the values ( one for the loss and an other for the validation loss).
This seems a long task but with a text editor like visual studio code it's a matter of seconds.
write a callback that after every epoch writes the result of the epoch on an external text file and then take the values in a similar way as point 1.
something like this:
class print_log_Callback(Callback):
def __init__(self, logpath, steps):
self.logpath = logpath
self.losslst = np.zeros(steps)
def on_epoch_end(self, epoch, logs=None):
with open(logpath, 'a') as writefile:
with redirect_stdout(writefile):
print("The average loss for epoch {} is {:7.2f} and val_loss is {:7.2f}.".format(epoch, logs["loss"], logs['val_loss']))
writefile.write("\n")
print("The mean train loss is: ", np.mean(self.losslst))
writefile.write("\n")
writefile.write("\n")
Related
I am currently trying to get the loss and accuracy of each batch for both the training and validation of my Keras Model. I have managed to successfully do so for the loss and accuracy training but am running into problems when trying to obtain the equivalent for the validation loss and accuracy.
I was basing my work of this query and have adapted the code slightly for my application. The issue is that I just receive a list of 'None' values.
I created my own LossHistory class shown below. I want to be able to get the metrics for each batch and then the each epoch.
class LossHistory(keras.callbacks.Callback):
def on_train_begin(self, logs={}):
self.history = {'loss':[],'val_loss':[], 'accuracy':[],'val_accuracy':[], 'loss_avg':[],'val_loss_avg':[], 'accuracy_avg':[],'val_accuracy_avg':[]}
def on_batch_end(self, batch, logs={}):
self.history['loss'].append(logs.get('loss'))
self.history['val_loss'].append(logs.get('val_loss'))
self.history['accuracy'].append(logs.get('accuracy'))
self.history['val_accuracy'].append(logs.get('val_accuracy'))
def on_epoch_end(self, epoch, logs={}):
self.history['loss_avg'].append(logs.get('loss'))
self.history['val_loss_avg'].append(logs.get('val_loss'))
self.history['accuracy_avg'].append(logs.get('accuracy'))
self.history['val_accuracy_avg'].append(logs.get('val_accuracy'))
I can still get the average values of the validation loss and accuracy. In other words, the validation metrics after each epoch. I am just not able to get those metrics for each batch.
Would anyone know why this is the case? I tried looking at the Keras documentation about customer callbacks but couldn't make find anything much regarding the validation metrics.
Not sure if I'm missing anything obvious.
Validation loss and metrics are evaluated at the end of each epoch, there is no per-batch validation loss and metrics as computed by Keras.
I am training a model with keras for one epoch only:
history = model.fit([x], y,
validation_split=0.2, epochs=1, batch_size=2)
print(history.history['accuracy'])
The history now obviously only contains one value from the end of the epoch. How can I evaluate the training accuracy or loss during the epoch? I expect these to be the values that are shown in the console during training.
To be clear: I want a history to be written after every batch (not after every epoch, as per usual).
I assume you want to save the accuracy and loss at the end of each batch. To do that you need to create a custom callback as shown below
class BCP(keras.callbacks.Callback):
batch_accuracy = [] # accuracy at given batch
batch_loss = [] # loss at given batch
def __init__(self):
super(BCP,self).__init__()
def on_train_batch_end(self, batch, logs=None):
BCP.batch_accuracy.append(logs.get('accuracy'))
BCP.batch_loss.append(logs.get('loss'))
now in model.fit include
callbacks = [BCP()]
now train for 1 epoch. at the end of the epoch the values of the accuracy and loss for each batch is stored in BCP.batch_accuracy and BCP.batch_loss. You can print them out
as follows:
print('{0:^4s}{1:^22s}{2:^10s}'.format('Batch', 'Loss', 'Accuracy'))
for i in range (len(BCP.batch_accuracy)):
print('{0:^4s}{1:15.5f}{2:15.2f}'.format(str(i), BCP.batch_loss[i], BCP.batch_accuracy[i]* 100))
I am using the tensorflow DataSet for input data pipeline. I am wondering how to run training without data shuffling in first epoch and start shuffling data from the second epoch.
the graph is usually built before iterative training start and during training it seems not straight-forward how to change the DataSet shuffling behavior since it looks to me kinds of like changing the graph.
any idea?
thanks,
Harry
The buffer_size argument to Dataset.shuffle() can be a computed tf.Tensor, so you can use the following code that uses Dataset.range(NUM_EPOCHS).flat_map(...) to transform a sequence of epoch numbers to the (shuffled or otherwise) elements of a per_epoch_dataset:
NUM_EPOCHS = ... # The total number of epochs.
BUFFER_SIZE = ... # The shuffle buffer size to use from the second epoch on.
per_epoch_dataset = ... # A `Dataset` representing the elements of a single epoch.
def shuffle_after_first_epoch(epoch):
# Set `epoch_buffer_size` to 1 (i.e. no shuffling) in the 0th epoch,
# and `BUFFER_SIZE` thereafter.
epoch_buffer_size = tf.cond(tf.equal(epoch, 0),
lambda: tf.constant(1, tf.int64),
lambda: tf.constant(BUFFER_SIZE, tf.int64))
return per_epoch_dataset.shuffle(epoch_buffer_size)
dataset = tf.data.Dataset.range(NUM_EPOCHS).flat_map(shuffle_after_first_epoch)
I'm trying to train multiple models in parallel on a single graphics card. To achieve that I need to resume training of models from saved weights which is not a problem. The model.fit() method has even a parameter initial_epoch that lets me tell the model which epoch the loaded model is on. However when i pass a TensorBoard callback to the fit() method in order to monitor the training of the models, on Tensorboard all data is shown on x=0.
Is there a ways to overcome this and adjust the epoch on tensorboard?
By the way: Im running Keras 2.0.6 and Tensorflow 1.3.0.
self.callbacks = [TensorBoardCallback(log_dir='./../logs/'+self.model_name, histogram_freq=0, write_graph=True, write_images=False, start_epoch=self.step_num)]
self.model.fit(x=self.data['X_train'], y=self.data['y_train'], batch_size=self.input_params[-1]['batch_size'], epochs=1, validation_data=(self.data['X_test'], self.data['y_test']), verbose=verbose, callbacks=self.callbacks, shuffle=self.hyperparameters['shuffle_data'], initial_epoch=self.step_num)
self.model.save_weights('./weights/%s.hdf5'%(self.model_name))
self.model.load_weights('./weights/%s.hdf5'%(self.model_name))
self.model.fit(x=self.data['X_train'], y=self.data['y_train'], batch_size=self.input_params[-1]['batch_size'], epochs=1, validation_data=(self.data['X_test'], self.data['y_test']), verbose=verbose, callbacks=self.callbacks, shuffle=self.hyperparameters['shuffle_data'], initial_epoch=self.step_num)
self.model.save_weights('./weights/%s.hdf5'%(self.model_name))
The resulting graph on Tensorboard looks like this which is not what i was hoping for:
Update:
When passing epochs=10 to the first model.fit() the 10 epoch results are displayed in TensorBoard (see picture).
However when reloading the model and running it (with the same callback attached) the on_epoch_end method of the callback gets never called.
Turns out that when i pass the number of episodes to model.fit() to tell it how long to train, it has to be the number FROM the initial_epoch specified. So if initial_epoch=self.step_num then , epochs=self.step_num+10 if i want to train for 10 episodes.
Say we just started fitting our model and our first time epoch count is 30
(please ignore other paramterers just look at epochs and initial_epoch)
model.fit(train_dataloader,validation_data = test_dataloader,epochs =30,steps_per_epoch = len(train_dataloader),callbacks = callback_list)
Now say ,after 30 epoch we want to start again from 31st epoch (you can see this in tesnorboard) by changing our Adam optimizer(or nay optimizer) learning rate
so we can do is
model.optimizer.learning_rate = 0.0005
model1.fit(train_dataloader,validation_data = test_dataloader,initial_epoch=30,epochs =55,steps_per_epoch = len(train_dataloader),callbacks = callback_list)
=> So here initial_epoch= where we have left training last time;
epochs= initial_epoch+num_epoch we want to run for this second fit
I'm new to Keras and I intend to store the output of my network for every epoch. To that end, I want to use Tensorbaord to observe the output layer in its environment.
class OutputObserver(Callback):
""""
callback to observe the output of the network
"""
def on_train_begin(self, logs={}):
self.epoch = []
self.out_log = []
def on_epoch_end(self, epoch, logs={}):
self.epoch.append(epoch)
self.out_log.append(self.model.get_layer('Dense03/Output').output)
This will store the outputs tensors into a list. The problem is that I can do neither 1. convert this to a Numpy array so that can be read a CSV, ... file, 2. write a summary using Tensorflow (as Keras does not have this ability) and then analyze the output in Tensorboard.
I would be pleased to hear your opinions on storing and visualizing the output layer in every epoch of training.
Sincerely,
Saeed.
To save the output layer for every epoch you need to pass the training/validation data to the callback object. The callback I used is as follow,
class OutputObserver(Callback):
""""
callback to observe the output of the network
"""
def __init__(self, xy):
self.out_log = []
self.xy = xy
def on_epoch_end(self, epoch, logs={}):
self.out_log.append(self.model.predict(self.xy.x_train, batch_size=p.bath_size))
which xy.x_train is the training data.
now, the out_log array is a numpy.ndarray of shape (epoch_number, data_number, prediction_length):
type(prediction_logs[0])
Out[62]: numpy.ndarray