Tensorflow error calling model.fit after load_model - tensorflow

Question

I'm training a small, simple neural net for a basic problem of regulating a motor's speed. I want to be able to save the model and exit the program, then load it later and resume training.
Here's the relevant code:
self.model = Sequential()
self.model.add(InputLayer(2))
self.model.add(Dense(6, activation='relu'))
self.model.add(Dense(9, activation='linear'))
self.model.compile(loss='mse', optimizer='adam', metrics=['mae'])
# ... Loop for training and Evaluation (Deep Q Learner) ...
learn(self.model)
self.model.save('motor_model', save_format='tf')
Now after it's trained I want to be able to load the model and continue training
self.model = models.load_model('motor_model', compile=False)
# ... Loop for training and Evaluation (Deep Q Learner) ...
learn(self.model)
The first time I run the model it works fine. However, after saving and loading the model it does not. Upon loading the model I am able to call the predict function:
prediction = self.model.predict(currentInput)
However, It fails when I call the predict function:
self.model.fit(self.input, target_vec.reshape(-1, 9), epochs=1, verbose=0)
The error I get is:
2019-12-07 07:22:00.762174: W tensorflow/c/c_api.cc:326] Operation
'{name:'sequential/dense/StatefulPartitionedCall'
id:33 op device:{} def:{{{node
sequential/dense/StatefulPartitionedCall}} =
StatefulPartitionedCall[Tin=[DT_FLOAT, DT
_RESOURCE, DT_RESOURCE], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _gradient_op_type="PartitionedCall-298", conf
ig="",
config_proto="\n\007\n\003CPU\020\001\n\007\n\003GPU\020\0002\002J\0008\001",
executor_type="", f=__forward_re
stored_function_body_509[]](input_1, dense/kernel, dense/bias)}}' was
changed by setting attribute after it was run b
y a session. This mutation will have no effect, and will trigger an
error in the future. Either don't modify nodes af
ter running them or create a new session. 2019-12-07 07:22:03.320478:
W tensorflow/python/util/util.cc:299] Sets are not currently
considered sequences, but th
is may change in the future, so consider avoiding using them.
Traceback (most recent call last): File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1363, in _do_call
return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1346, in _run_fn
self._extend_graph() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1386, in _extend_graph
tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "ct2.py", line 47, in
leftController.to_position(target, overrideAction) File "/opt/mowzr/motor_controller.py", line 94, in to_position
self.model.fit(self.prevInput, target_vec.reshape(-1, 9), epochs=1, verbose=0) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py",
line 766, in fit
use_multiprocessing=use_multiprocessing) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_arrays.py",
line 680, in
fit
steps_name='steps_per_epoch') File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training_arrays.py",
line 275, in
model_iteration
model.reset_metrics() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/training.py",
line 953, in reset_m
etrics
m.reset_states() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/metrics.py",
line 209, in reset_states
K.batch_set_value([(v, 0) for v in self.variables]) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 3343, in batch_set_valu
e
get_session().run(assign_ops, feed_dict=feed_dict) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 490, in get_session
_initialize_variables(session) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 905, in _initialize_var
iables
[variables_module.is_variable_initialized(v) for v in candidate_vars]) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 956, in run
run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1179, in _run
feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1357, in _do_run
run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1382, in _do_call
raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1363, in _do_call
return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1346, in _run_fn
self._extend_graph() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1386, in _extend_graph
tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "ct2.py", line 53, in
leftController.saveModel() File "/opt/mowzr/motor_controller.py", line 116, in saveModel
self.model.save('motor_model', save_format='tf') File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/engine/network.py",
line 986, in save
signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/saving/save.py",
line 115, in save_model
signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/saving/saved_model/save.py",
line 74, in
save
save_lib.save(model, filepath, signatures, options) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/saved_model/save.py",
line 924, in save
object_saver.save(utils_impl.get_variables_path(export_dir)) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/util.py",
line 1161, in save
session = get_session() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/tracking/util.py",
line 71, in get_ses
sion
session = keras_backend.get_session() File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 490, in get_session
_initialize_variables(session) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/backend.py",
line 905, in _initialize_var
iables
[variables_module.is_variable_initialized(v) for v in candidate_vars]) File
"/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 956, in run
run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1179, in _run
feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1357, in _do_run
run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py",
line 1382, in _do_call
raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Node
'training/Adam/gradients/gradients/sequential/dens
e_1/StatefulPartitionedCall_grad/PartitionedCall': Connecting to
invalid output 1 of source node sequential/dense_1/S
tatefulPartitionedCall which has 1 outputs.

Answer 1

I got the same error.
I don't know what exactly produces this error but there is a way to solve it (not a pretty one though). Create the model with the same architecture and just set the weights to the loaded model weights:
self.model = self.create_model()
self.model.set_weights(load_model("sample.model").get_weights())