Tensorflow model with custom loss function gets no training done - tensorflow

Question

I created a custom loss function as below:
import tensorflow.keras.backend as K
def custom_loss(y_true, y_pred):
y_true = K.cast(y_true, tf.float32)
y_pred = K.cast(y_pred, tf.float32)
mask = K.sign(y_true) * K.sign(y_pred)
mask = ((mask * -1) + 1) / 2
losses = K.abs(y_true * mask)
return K.sum(losses)
However, when I try to train the model using this loss function, I get no training done.
The model works normally with other loss functions such as mse and mae, and I've tried all learning rates and model complexities.
Below is how I know no training is being done.
model = get_compiled_model()
print(model.predict(train_x)[:10])
model.fit(train_x, train_y, epochs=5, verbose=1)
print(model.predict(train_x)[:10])
model.fit(train_x, train_y, epochs=5, verbose=1)
print(model.predict(train_x)[:10])
[[0.19206487]
[0.19201839]
[0.19199933]
[0.19199185]
[0.19206186]
[0.19208357]
[0.1920282 ]
[0.19203594]
[0.1919941 ]
[0.19202243]]
Epoch 1/5
1/1 [==============================] - 0s 1ms/step - loss: 0.0179
Epoch 2/5
1/1 [==============================] - 0s 2ms/step - loss: 0.0179
Epoch 3/5
1/1 [==============================] - 0s 1ms/step - loss: 0.0179
Epoch 4/5
1/1 [==============================] - 0s 1ms/step - loss: 0.0179
Epoch 5/5
1/1 [==============================] - 0s 2ms/step - loss: 0.0179
[[0.19206487]
[0.19201839]
[0.19199933]
[0.19199185]
[0.19206186]
[0.19208357]
[0.1920282 ]
[0.19203594]
[0.1919941 ]
[0.19202243]]
Epoch 1/5
1/1 [==============================] - 0s 1ms/step - loss: 0.0179
Epoch 2/5
1/1 [==============================] - 0s 2ms/step - loss: 0.0179
Epoch 3/5
1/1 [==============================] - 0s 2ms/step - loss: 0.0179
Epoch 4/5
1/1 [==============================] - 0s 951us/step - loss: 0.0179
Epoch 5/5
1/1 [==============================] - 0s 1ms/step - loss: 0.0179
[[0.19206487]
[0.19201839]
[0.19199933]
[0.19199185]
[0.19206186]
[0.19208357]
[0.1920282 ]
[0.19203594]
[0.1919941 ]
[0.19202243]]
The 2d array in the code above is the first 10 predictions of the model, and it does not change in the slightest even with 5 epochs of training.
My intuition tells me something is wrong with the loss function, but I have no idea what.
The model looks like as follows
def get_compiled_model():
model = tf.keras.Sequential([
tf.keras.layers.Dense(10, input_dim=2*training_size+1, activation='softmax'),
tf.keras.layers.Dense(10, activation='softmax'),
tf.keras.layers.Dense(1, activation='tanh')
])
opt = tf.keras.optimizers.Adam(learning_rate=0.0005)
model.compile(optimizer=opt,
loss=custom_loss,
metrics=[])
return model

Answer 1

I was playing around with some fake data using your model and loss function, and I wanted to check the derivatives.
if __name__=="__main__":
m = get_compiled_model()
x = numpy.random.random( (1000, 21))
x = numpy.array(x, dtype="float32")
exp_y = numpy.random.random( (1000, 1))
exp_y = (exp_y>0.5)*1.0
with tf.GradientTape() as tape:
y = m(x)
loss = custom_loss(y, exp_y)
#loss = keras.losses.mse(y, exp_y)
grad = tape.gradient(loss, m.trainable_variables)
for var, g in zip(m.trainable_variables, grad):
print(f'{var.name}, shape: {K.sum(g*g)}')
For the mse loss function:
dense/kernel:0, shape: 2817.013671875
dense/bias:0, shape: 530.52197265625
dense_1/kernel:0, shape: 3826.3974609375
dense_1/bias:0, shape: 25160.9375
dense_2/kernel:0, shape: 125238.34375
dense_2/bias:0, shape: 1241268.5
For the custom loss function
dense/kernel:0, shape: 34.87071228027344
dense/bias:0, shape: 6.609962463378906
dense_1/kernel:0, shape: 107.27591705322266
dense_1/bias:0, shape: 824.83740234375
dense_2/kernel:0, shape: 5944.91796875
dense_2/bias:0, shape: 59201.58203125
We can see that the sum of derivatives are orders of magnitude different. Even with this random data, the MSE loss function will cause the output of the model to change over time.
This might only be the case for the fake data I made.