I have written a custom loss function as follows:
def distance_loss(y_actual, y_pred):
loss = tf.math.sqrt(
tf.math.add(
tf.math.pow(
tf.math.subtract(
tf.cast(y_actual[0], tf.float32),
y_pred[0]),
tf.constant(2.0)
),
tf.math.pow(
tf.math.subtract(
tf.cast(y_actual[1], tf.float32),
y_pred[1]),
tf.constant(2.0)
)
)
)
return loss
I have used this across several model fits already and saved those weights.
I've realised that I should be normalising the loss value of this function so that: 0 <= loss <=1, because I want to compare the results across different models which take input images of different sizes.
Will I need to scrap the weights from my previous fits?
Related
I have two outputs for a model: y_pred_1, and y_pred_2 and I want to design a loss function in Keras where I can use binary cross-entropy on both of them using individual loss weights. My idea of the loss function is like this:
def my_loss_fn(y_true, y_pred_1, y_pred_2):
loss_1 = tf.keras.losses.BinaryCrossentropy(y_true, y_pred_1)
loss_2 = tf.keras.losses.BinaryCrossentropy(y_true, y_pred_2)
loss = alpha*loss_1 + (1-alpha)*loss_2
return loss
Here alpha is a predetermined value and the model will have to be trained based on this function. Is there a way to customize a loss function like this?
I want to train a Neural Network for a classification task in Keras using a TensorFlow backend with a custom loss function. In my loss, I want to give different weights to different training examples. I have some datapoints I consider important and some I do not consider as important. I want my loss function to take this into account and punish errors in important examples more than in less important ones.
I have already built my model:
input = tf.keras.Input(shape=(16,))
hidden_layer_1 = tf.keras.layers.Dense(5, kernel_initializer='glorot_uniform', activation='relu')(input)
output = tf.keras.layers.Dense(1, kernel_initializer='normal', activation='softmax')(hidden_layer_1)
model = tf.keras.Model(input, output)
model.compile(loss=custom_loss(input), optimizer='adam', run_eagerly=True, metrics = [tf.keras.metrics.Accuracy(), 'acc'])
and the currrent state of my loss function is:
def custom_loss(input):
def loss(y_true, y_pred):
return ...
return loss
I'm struggling with implementing the loss function in the way I explained above, mainly because I don't exactly know what input, y_pred and y_true are (KerasTensors, I know - but what is the content? And is it for one training example only or for the whole batch?). I'd appreciate help with
printing out the values of input, y_true and y_pred
converting the input value to a numpy ndarray ([1,3,7] for example) so I can use the array to look up my weight for this specific training data point
once I have my weigth as a number (0.5 for example), how do I implement the computation of the loss function in Keras? My loss for one training exaple should be 0 if the classification was correct and weight if it was incorrect.
I have written my custom training loop using tf.GradientTape(). My data has 2 classes. The classes are not balanced; class1 data contributes almost 80% and class2 contributes remaining 20%. Therefore in order to remove this imbalance I was trying to write custom loss function which will take into account this imbalance and apply the corresponding class weights and calculate the loss. i.e. I want to use the class_weights = [0.2, 0.8]. I am not able to find similar examples.
However all the examples I am seeing are using model.fit approach where its easier to pass the class_weights. I am not able to find out the example which uses class_weights with custom training loop using tf.GradientTape.
I did go through the suggestions of using sample_weight, however I don't have the data where in I can specify the weights for samples, therefore my preference is to use class weight.
I am using BinaryCrossentropy loss as loss function but I want to change the loss based on the class_weights. That's where I am stuck, how to tell BinaryCrossentropy to consider the class_weights.
Is my approach of using custom loss function correct or there is better way to make use of class_weights while training with custom training loop (not using model.fit)?
you can write your own loss function. in that loss function call BinaryCrossentropy and then multiply the result in the weight you want and return that
Here's an implementation that should work for n classes instead of just 2.
For your example of 80:20 split, calculate weights as below (assuming 100 samples in total).
Weight calculation (ref: Handling Class Imbalance: TensorFlow):
weight_class_0 = (1/count_for_class_0) * (total_samples / num_classes) # (80%) 0.625
weight_class_1 = (1/count_for_class_1) * (total_samples / num_classes) # (20%) 2.5
class_wts = tf.constant([weight_class_0, weight_class_1])
Loss function: Requires labels to be sparse and logits unscaled (no activations applied).
# Example logits=[[-3.2, 2.0], [1.2, 0.5], ...], (sparse)labels=[0, 1, ...]
def weighted_sparse_categorical_crossentropy(labels, logits, weights):
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels, logits)
class_weights = tf.gather(weights, labels)
return tf.reduce_mean(class_weights * loss)
You can supply this loss function to custom training loops.
I am training a U-Net image segmentation on whole slide pathology images. I was wondering how can I handle un-annotated areas? I am working with huge tissues and it’s impossible to annotate all or the vast majority of the tissue, so I have annotations from a pathologist who has annotated selected tissue structures of interest to us. That means that in many tiles I’m generating, there is a segment that’s not annotated.
Would it affect the U-Net negatively by indirectly indicating that the un-annotated area is negative to one category or another, although it’s not negative? How do I handle this important case? Does it make sense to mask the image to only the annotated parts, such that un-annotated regions are black?
Thanks
One way to deal with this is to use a weighted loss function where you simply assign a weight of zero to the class that you don't want to include. Essentially, you're treating the unannotated area as an additional class that doesn't contribute to the loss function. You can find the GitHub repo to a fully functional Keras implementation here.
Specifically, I would use a weighted categorical cross-entropy loss function. You can find an implementation for Keras here:
from keras import backend as K
def weighted_categorical_crossentropy(weights):
"""
A weighted version of keras.objectives.categorical_crossentropy
Variables:
weights: numpy array of shape (C,) where C is the number of classes
Usage:
weights = np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.
loss = weighted_categorical_crossentropy(weights)
model.compile(loss=loss,optimizer='adam')
"""
weights = K.variable(weights)
def loss(y_true, y_pred):
# scale predictions so that the class probas of each sample sum to 1
y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
# clip to prevent NaN's and Inf's
y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
# calc
loss = y_true * K.log(y_pred) * weights
loss = -K.sum(loss, -1)
return loss
return loss
And you can then compile your model for training like this:
model.compile(optimizer='adam', loss=weighted_categorical_crossentropy(np.array([background_weight, foreground_weight, 0])), metrics='accuracy')
I am currently trying to build a deep learning model with three different loss functions in Keras. The first loss function is the typical mean squared error loss. The other two loss functions are the ones I built myself, which finds the difference between a calculation made from the input image and the output image (this code is a simplified version of what I'm doing).
def p_autoencoder_loss(yTrue,yPred):
def loss(yTrue, y_Pred):
return K.mean(K.square(yTrue - yPred), axis=-1)
def a(image):
return K.mean(K.sin(image))
def b(image):
return K.sqrt(K.cos(image))
a_pred = a(yPred)
a_true = a(yTrue)
b_pred = b(yPred)
b_true = b(yTrue)
empirical_loss = (loss(yTrue, yPred))
a_loss = K.mean(K.square(a_true - a_pred))
b_loss = K.mean(K.square(b_true - b_pred))
final_loss = K.mean(empirical_loss + a_loss + b_loss)
return final_loss
However, when I train with this loss function, it is simply not converging well. What I want to try is to minimize the three loss functions separately, not together by adding them into one loss function.
I essentially want to do the second option here Tensorflow: Multiple loss functions vs Multiple training ops but in Keras form. I also want the loss functions to be independent from each other. Is there a simple way to do this?
You could have 3 outputs in your keras model, each with your specified loss, and then keras has support for weighting these losses. It will also then generate a final combined loss for you in the output, but it will be optimising to reduce all three losses. Be wary with this though as depending on your data/problem/losses you might find it stalls slightly or is slow if you have losses fighting each other. This however requires use of the functional API. I'm unsure as to whether this actually implements separate optimiser instances, however I think this is as close you will get in pure Keras that i'm aware of without having to start writing more complex TF training regimes.
For example:
loss_out1 = layers.Dense(1, activation='sigmoid', name='loss1')(x)
loss_out2 = layers.Dense(1, activation='sigmoid', name='loss2')(x)
loss_out3 = layers.Dense(1, activation='sigmoid', name='loss3')(x)
model = keras.Model(inputs=[input],
outputs=[loss1, loss2, loss3])
model.compile(optimizer=keras.optimizers.RMSprop(1e-3),
loss=['binary_crossentropy', 'categorical_crossentropy', 'custom_loss1'],
loss_weights=[1., 1., 1.])
This should compile a model with 3 outputs at the end from (x) which would be above. When you compile you set the outputs as a list as well as set the losses and loss weights as a list. Note that when you fit() that you'll need to supply your target outputs three times as a list too e.g. [y, y, y] as your model now has three outputs.
I'm not a Keras expert, but it's pretty high-level and i'm not aware of another way using pure Keras. Hopefully someone can come correct me with a better solution!
Since there is only one output, few things that can be done:
1.Monitor the individual loss components to see how they vary.
def a_loss(y_true, y_pred):
a_pred = a(yPred)
a_true = a(yTrue)
return K.mean(K.square(a_true - a_pred))
model.compile(....metrics=[...a_loss,b_loss])
2.Weight the loss components where lambda_a & lambda_b are hyperparameters.
final_loss = K.mean(empirical_loss + lambda_a * a_loss + lambda_b * b_loss)
Use a different loss function like SSIM.
https://www.tensorflow.org/api_docs/python/tf/image/ssim