as_list() is not defined on an unknown TensorShape on y_t_rank = len(y_t.shape.as_list()) and related to metrics - tensorflow

Question

TF 2.3.0.dev20200620
I got this error during .fit(...) for a model with a sigmoid binary output. I used tf.data.Dataset as the input pipeline.
The strange thing is it depends on the metric:
Don't work:
model.compile(
optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
loss=tf.keras.losses.BinaryCrossentropy(),
metrics=['accuracy']
)
work:
model.compile(
optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
loss=tf.keras.losses.BinaryCrossentropy(),
metrics=[tf.keras.metrics.BinaryAccuracy()]
)
But as I understood, 'accuracy' should be fine. In fact, instead of using my own tf.data.Dataset custom setup (can be provided if needed), using tf.keras.preprocessing.image_dataset_from_directory give no such error. This is the case from tutorial https://keras.io/examples/vision/image_classification_from_scratch.
Trace is pasted below. Notice this is diff from other 2 older questions. it involves somehow the metrics.
ValueError: in user code:
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function *
return step_function(self, iterator)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2526 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2886 _call_for_each_replica
return fn(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step **
outputs = model.train_step(data)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:759 train_step
self.compiled_metrics.update_state(y, y_pred, sample_weight)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:388 update_state
self.build(y_pred, y_true)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:319 build
self._metrics, y_true, y_pred)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1139 map_structure_up_to
**kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1235 map_structure_with_tuple_paths_up_to
*flat_value_lists)]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1234 <listcomp>
results = [func(*args, **kwargs) for args in zip(flat_path_list,
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1137 <lambda>
lambda _, *values: func(*values), # Discards the path arg.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 _get_metric_objects
return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 <listcomp>
return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:440 _get_metric_object
y_t_rank = len(y_t.shape.as_list())
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1190 as_list
raise ValueError("as_list() is not defined on an unknown TensorShape.")
ValueError: as_list() is not defined on an unknown TensorShape.

Answer 1

Had exactly the same problem when using 'accuracy' metric.
I followed https://github.com/tensorflow/tensorflow/issues/32912#issuecomment-550363802 example:
def _fixup_shape(images, labels, weights):
images.set_shape([None, None, None, 3])
labels.set_shape([None, 19]) # I have 19 classes
weights.set_shape([None])
return images, labels, weights
dataset = dataset.map(_fixup_shape)
which helped me solve the problem.
But, in my case, instead of using one map function, as kawingkelvin did above, to load and set_shape inside, I needed to use two map functions because of some errors in the TF code.
The final solution for me was to use the following order:
dataset.batch.map(get_data).map(fix_shape).prefetch
NOTE: batch can be done both before and after map(get_data) depending on how your get_data function is created. Fix_shape must be done after.

Answer 2

I am able to fix this in such a way as to keep the metrics 'accuracy' (rather than using BinaryAccuracy). However, I do not quite understand why this is needed for 'accuracy', but not needed for other closely related one (e.g. BinaryAccuracy).
2 things:
construct a ds such that the batch label has shape of (batch_size, 1) but not (batch_size,). Following the keras.io tutorial mentioned, it should have been ok with the latter. This change aims to get rid of the "unknown" in the TensorShape.
add this to the ds pipeline:
label.set_shape([1])
def process_path(file_path):
label = get_label(file_path)
img = tf.io.read_file(file_path)
img = tf.image.decode_jpeg(img, channels=3)
label.set_shape([1])
return img, label
ds = ds.map(process_path, num_parallel_calls=AUTO).shuffle(1024).repeat().batch(batch_size).prefetch(buffer_size=AUTO)
This is the state before .batch(...) so a single sample should have 1 as the shape (and therefore, (batch_size, 1) after batching.
After doing so, the error didn't happen, and I used the exact same metrics 'accuracy' as in
https://keras.io/examples/vision/image_classification_from_scratch
Hope this helps anyone who got hit. I have to admit I don't truly understand why it didn't work in the first place. It still seems like a TF bug to me.