How do you use tensorflow ctc_batch_cost function with keras? - tensorflow

Question

I have been trying to implement a CTC loss function in keras for several days now.
Unfortunately, I have yet to find a simple way to do this that fits well with keras. I found tensorflow's tf.keras.backend.ctc_batch_cost function but there is not much documentation on it. I am confused about a few things. First, what are the input_length and label_length parameters? I am trying to make a handwriting recognition model and my images are 32x128, my RNN has 32 time steps, and my character list has a length of 80. I have tried to use 32 for both parameters and this gives me the error below.
Shouldn't the function already know the input_length and label_length from the shape of the first two parameters (y_true and y_pred)?
Secondly, do I need to encode my training data? Is this all done automatically?
I know tensorflow also has a function called tf.keras.backend.ctc_decode. Is this only used when making predictions?
def ctc_cost(y_true, y_pred):
return tf.keras.backend.ctc_batch_cost(
y_true, y_pred, 32, 32)
model = tf.keras.Sequential([
layers.Conv2D(32, 5, padding="SAME", input_shape=(32, 128, 1)),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPool2D(2, 2),
layers.Conv2D(64, 5, padding="SAME"),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPool2D(2, 2),
layers.Conv2D(128, 3, padding="SAME"),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPool2D((1, 2), (1, 2)),
layers.Conv2D(128, 3, padding="SAME"),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPool2D((1, 2), (1, 2)),
layers.Conv2D(256, 3, padding="SAME"),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.MaxPool2D((1, 2), (1, 2)),
layers.Reshape((32, 256)),
layers.Bidirectional(layers.LSTM(256, return_sequences=True)),
layers.Bidirectional(layers.LSTM(256, return_sequences=True)),
layers.Reshape((-1, 32, 512)),
layers.Conv2D(80, 1, padding="SAME"),
layers.Softmax(-1)
])
print(model.summary())
model.compile(tf.optimizers.RMSprop(0.001), ctc_cost)
Error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: squeeze_dims[0] not in [0,0). for 'loss/softmax_loss/Squeeze' (op: 'Squeeze') with input shapes: []
Model:
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 32, 128, 32) 832
batch_normalization (BatchNo (None, 32, 128, 32) 128
activation (Activation) (None, 32, 128, 32) 0
max_pooling2d (MaxPooling2D) (None, 16, 64, 32) 0
conv2d_1 (Conv2D) (None, 16, 64, 64) 51264
batch_normalization_1 (Batch (None, 16, 64, 64) 256
activation_1 (Activation) (None, 16, 64, 64) 0
max_pooling2d_1 (MaxPooling2 (None, 8, 32, 64) 0
conv2d_2 (Conv2D) (None, 8, 32, 128) 73856
batch_normalization_2 (Batch (None, 8, 32, 128) 512
activation_2 (Activation) (None, 8, 32, 128) 0
max_pooling2d_2 (MaxPooling2 (None, 8, 16, 128) 0
conv2d_3 (Conv2D) (None, 8, 16, 128) 147584
batch_normalization_3 (Batch (None, 8, 16, 128) 512
activation_3 (Activation) (None, 8, 16, 128) 0
max_pooling2d_3 (MaxPooling2 (None, 8, 8, 128) 0
conv2d_4 (Conv2D) (None, 8, 8, 256) 295168
batch_normalization_4 (Batch (None, 8, 8, 256) 1024
activation_4 (Activation) (None, 8, 8, 256) 0
max_pooling2d_4 (MaxPooling2 (None, 8, 4, 256) 0
reshape (Reshape) (None, 32, 256) 0
bidirectional (Bidirectional (None, 32, 512) 1050624
bidirectional_1 (Bidirection (None, 32, 512) 1574912
reshape_1 (Reshape) (None, None, 32, 512) 0
conv2d_5 (Conv2D) (None, None, 32, 80) 41040
softmax (Softmax) (None, None, 32, 80) 0
Here is the tensorflow documentation I was referencing:
https://www.tensorflow.org/api_docs/python/tf/keras/backend/ctc_batch_cost

Answer 1

First, what are the input_length and label_length parameters?
input_length is the length of the input sequence in time steps. label_length is the length of the text label.
For example, if you are trying to recognize:
and you are doing it in 32 time steps, then your input_length is 32 and your label_length is 12 (len("John Hancock")).
Shouldn't the function already know the input_length and label_length from the shape of the first two parameters (y_true and y_pred)?
You usually process input data in batches, which have to be padded to the largest element in the batch, so this information is lost. In your case the input_length is always the same, but the label_length varies.
When dealing with speech recognition, for example, input_length can vary as well.
Secondly, do I need to encode my training data? Is this all done automatically?
Not sure I understand what you are asking, but here is a good example written in Keras:
https://keras.io/examples/image_ocr/
I know tensorflow also has a function called tf.keras.backend.ctc_decode. Is this only used when making predictions?
In general, yes. You can also try to use it make you breakfast in the morning, but it's not very good at it ;)