Keras callbacks Tensorboard multi-output Error - tensorflow

Question

I can't figure out what is causing this error for multi-output Keras Model when using callbacks.TensorBoard.
tbCallBack = keras.callbacks.TensorBoard(log_dir = logdir, histogram_freq = 1, write_graph = 1, write_images = 0, write_grads = 1)
###No errror when not using callbacks
regr.fit( Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1)
###No errror when not using validation_data
regr.fit( Ax_train, [Ay_train_p, Ay_train_s], epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])
###Error Occurred
regr.fit( Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])
Error
Epoch 1/500
1280/1663 [======================>.......] - ETA: 0s - loss: 1.6230 - output_power_loss: 0.9627 - output_slack_loss: 0.66032017-07-11 03:17:27.964542: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.964589: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
[[Node: output_slack_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
2017-07-11 03:17:27.970690: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.970735: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
[[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
2017-07-11 03:17:27.972004: W tensorflow/core/framework/op_kernel.cc:1148] Invalid argument: Shape [-1] has negative dimensions
2017-07-11 03:17:27.972026: E tensorflow/core/common_runtime/executor.cc:644] Executor failed to create kernel. Invalid argument: Shape [-1] has negative dimensions
[[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Traceback (most recent call last):
File "tf_keras.py", line 183, in <module>
regr.fit( Ax_train, [Ay_train_p, Ay_train_s], validation_data= (Ax_test, [Ay_test_p, Ay_test_s]), epochs = 500, batch_size = 10, verbose = 1, callbacks=[tbCallBack])
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 1507, in fit
initial_epoch=initial_epoch)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 1176, in _fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 768, in on_epoch_end
result = self.sess.run([self.merged], feed_dict=feed_dict)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape [-1] has negative dimensions
[[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'output_power_sample_weights', defined at:
File "tf_keras.py", line 180, in <module>
regr = nn_model()
File "tf_keras.py", line 177, in nn_model
model.compile(optimizer = 'adam', loss ={'output_power': 'mean_squared_error', 'output_slack': 'binary_crossentropy'})
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/engine/training.py", line 870, in compile
name=name + '_sample_weights'))
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 431, in placeholder
x = tf.placeholder(dtype, shape=shape, name=name)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1530, in placeholder
return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1954, in _placeholder
name=name)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Shape [-1] has negative dimensions
[[Node: output_power_sample_weights = Placeholder[dtype=DT_FLOAT, shape=[?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
What does "Shape [-1] has negative dimensions" means? I had also tried with each output with callbacks.Tensorboard and there wasn't any error occurred. Also search with "Node: output_power_sample_weights" but no results.
regr.summary()
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
main_input (InputLayer) (None, 5) 0
____________________________________________________________________________________________________
Hidden (Dense) (None, 5) 30 main_input[0][0]
____________________________________________________________________________________________________
output_power (Dense) (None, 1) 6 Hidden[0][0]
____________________________________________________________________________________________________
output_slack (Dense) (None, 1) 6 Hidden[0][0]
====================================================================================================
Total params: 42
Trainable params: 42
Non-trainable params: 0

Answer 1

The "Shape [-1] has negative dimensions" error message is a (misleading) error printed when you don't feed a tf.placeholder() that has a partially-defined shape. This error message has been replaced with a more informative message in the nightly version of TensorFlow.
The error and the stack trace suggests that this Session.run() call is failing (source):
File "/home/roy/keras_tf/local/lib/python2.7/site-packages/keras/callbacks.py", line 768, in on_epoch_end
result = self.sess.run([self.merged], feed_dict=feed_dict)
Looking at the implementation, it appears that this is only called when the TensorBoard callback is installed, and self.validation_data is not None. From the implementation, it looks like this callback is incompatible with models that depend on placeholders that are not one of (i) model.inputs, (ii) model.targets, or (iii) model.sample_weights. It might be best to open an issue on the Keras GitHub issue tracker about this.