Naming TensorFlow/Keras checkpoints - tensorflow

Question

I am following the "Text generation with an RNN" tutorial on TensorFlow (link). I have trained the model for 10 epochs, and would like to train it some more. I have already written the code that allows the model to resume training. (This resumes training starting from the most recent checkpoint -- in this case, checkpoint 10). It trains just fine. However, the saved checkpoints are overwriting the previous checkpoints. This is because when I rerun the code, the epoch number starts at 1 again. Therefore, when I have finished epochs 11 - 20, I still have only 10 checkpoints (1 - 10), but they have overwritten the previous 10 checkpoints. I would like to rename the new checkpoints to checkpoints 11 - 20, but have failed to do so. Here is the pertinent segment of the code:
# Directory where the checkpoints will be saved
checkpoint_dir = './training_checkpoints'
# Name of the checkpoint files
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch+10}")
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
filepath=checkpoint_prefix,
save_weights_only=True)
EPOCHS = 10
The only difference from the original code from the TensorFlow website is that I have modified the original line
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch}")
to
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt_{epoch+10}")
However, it does not work. Here is the error:
KeyError: 'epoch+10'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "project/RNN_text_generator_finetune.py", line 102, in <module>
history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1137, in fit
callbacks.on_epoch_end(epoch, epoch_logs)
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 412, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 1249, in on_epoch_end
self._save_model(epoch=epoch, logs=logs)
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 1282, in _save_model
filepath = self._get_file_path(epoch, logs)
File "/opt/miniconda3/envs/newest11142020/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 1332, in _get_file_path
raise KeyError('Failed to format this callback filepath: "{}". '
KeyError: 'Failed to format this callback filepath: "./training_checkpoints/ckpt_{epoch+10}". Reason: \'epoch+10\''
Is there any way to rename the checkpoints in the code?

Answer 1

You can set as follows when resuming training
model.fit(...,
initial_epoch=epoch,
..)
Here, initial_epoch is an integer. Epoch at which to start training, it's useful for resuming a previous training run). Let's say you've trained a model at epoch 10 and stop training. So, when resuming the training, set the initial_epoch at 10. Src, and insightfull discussin.