I am currently trying to create my own loss function for Keras (using Tensorflow backend). This is a simple categorical crossentropy but I am applying a factor on the 1st column to penalize more loss from the 1st class.
Yet I am new to Keras and I can't figure out how to translate my function (below) as I have to use symbolic expressions and it seems I can't go element-wise:
def custom_categorical_crossentropy(y_true, y_pred):
y_pred = np.clip(y_pred, _EPSILON, 1.0-_EPSILON)
out = np.zeros(y_true.shape).astype('float32')
for i in range(0,y_true.shape[0]):
for j in range (0,y_true.shape[1]):
#penalize more all elements on class 1 so that loss takes its low proportion in the dataset into account
if(j==0):
out[i][j] = -(prop_database*(y_true[i][j] * np.log(y_pred[i][j]) + (1.0 - y_true[i][j]) * np.log(1.0 - y_pred[i][j])))
else:
out[i][j] = -(y_true[i][j] * np.log(y_pred[i][j]) + (1.0 - y_true[i][j]) * np.log(1.0 - y_pred[i][j]))
out = np.mean(out.astype('float32'), axis=-1)
return tf.convert_to_tensor(out,
dtype=tf.float32,
name='custom_loss')
Can someone help me?
Many thanks!
You can use class_weight in the fit method to penalize classes without creating functions:
weights = {
0:2,
1:1,
2:1,
3:1,
...
}
model.compile(optimizer=chooseOne, loss='categorical_crossentropy')
model.fit(......., class_weight = weights)
This will make the first class be twice as important as the others.
Related
Attached model shows how to add bias in case of the unbalanced classification problem initial_bias = np.log([pos/neg]). Is there a way to add bias if you have multi-class classification with unbalanced data, Say 5 classes where classes are have distribution (0.4,0.3,0.2.0.08 and 0.02)
2) also how to calculate and use class weights in such case?
update 1
I found a way to apply weights, still not sure how to use bias
#####adding weights 20 Feb
weight_for_0 = ( 1/ 370)*(370+ 977+ 795)/3
weight_for_1 = ( 1/ 977)*(370+ 977+ 795)/3
weight_for_2 = (1 / 795)*(370+ 977+ 795)/3
#array([0, 1, 2]), array([370, 977, 795])
class_weights_dict = {0: weight_for_0, 1: weight_for_1, 2:weight_for_2}
class_weights_dict
Dcnn.fit(train_dataset,
epochs=NB_EPOCHS,
callbacks=[MyCustomCallback()],verbose=2,validation_data=test_dataset, class_weight=class_weights_dict)
Considering that you're using 'softmax':
softmax = exp(neurons) / sum(exp(neurons))
And that you want the results of the classes to be:
frequency = [0.4 , 0.3 , 0.2 , 0.08 , 0.02]
Biases should be given by the equation (elementwise):
frequency = exp(biases) / sum(exp(biases))
This forms a system of equations:
f1 = e^b1 / (e^b1 + e^b2 + ... + e^b5)
f2 = e^b2 / (e^b1 + e^b2 + ... + e^b5)
...
f5 = e^b5 / (e^b1 + e^b2 + ... + e^b5)
If you can solve this system of equations, you get the biases you want.
I used excel and test-error method to determine that for the frequencies you wanted, your biases should be respectively:
[1.1 , 0.81 , 0.4 , -0.51 , -1.9]
I don't really know how to solve that system easily, but you can keep experimenting with excel or another thing until you reach the solution.
Adding the biases to the layer - method 1.
Use a name when defining the layer, like:
self.last_dense = layers.Dense(units=3, activation="softmax", name='last_layer')
You may need to build the model first, so:
dummy_predictions = model.predict(np.zeros((1,) + input_shape))
Then you get the weights:
weights_and_biases = model.get_layer('last_layer').get_weights()
w, b = weights_and_biases
new_biases = np.array([-0.45752, 0.51344, 0.30730])
model.get_layer('last_layer').set_weights([w, new_biases])
Method 2
def bias_init(bias_shape):
return K.variable([-0.45752, 0.51344, 0.30730])
self.last_dense = layers.Dense(units=3, activation="softmax", bias_initializer=bias_init)
Just in addition to #Daniel Möller's answer, to solve the system of equations
f1 = e^b1 / (e^b1 + e^b2 + ... + e^b5)
...
f5 = e^b5 / (e^b1 + e^b2 + ... + e^b5)
You don't need excel or anything. Just compute bi = ln(fi).
To calculate fi = e^bi / (sum of e^bj), note that fi/fj = e^(bi-bj). Suppose the lowest frequency is fk. You can set bk= 0 and then compute every other class bias with bi = bj + ln(fi/fj).
A complete answer is here:
### To solve that set of nonlinear equations, use scipy fsolve
from scipy.optimize import fsolve
from math import exp
# define the frequency of different classes
f=(0.4, 0.3, 0.2, 0.08, 0.02)
# define the equation
def eqn(x, frequency):
sum_exp = sum([exp(x_i) for x_i in x])
return [exp(x[i])/sum_exp - frequency[i] for i in range(len(frequency))]
# calculate bias init
bias_init = fsolve(func=eqn,
x0=[0]*len(f),
).tolist()
bias_init
To put all things together
def init_imbalanced_class_weight_bias(df:pd.DataFrame, lable:str):
"""To handle imbalanced classification, provide initial bias list and class weight dictionary to 2 places in a tf classifier
1) In the last layer of classifier: tf.keras.layers.Dense(..., bias_initializer = bias_init)
2) model.fit(train_ds, #x=dict(X_train), y=y_train,
batch_size=batch_size,
validation_data= valid_ds, #(dict(X_test), y_test),
epochs=epochs,
callbacks=callbacks,
class_weight=class_weight,
)
Args:
df:pd.DataFrame=train_df
label:str
Returns:
class_weight:dict, e.g. {0: 1.6282051282051282, 1: 0.7604790419161677, 2: 0.9338235294117647}
bias_init:list e.g. [0.3222079660508266, 0.1168690393701237, -0.43907701967633633]
Examples:
class_weight, bias_init = init_imbalanced_class_weight_bias(df=train_df, lable=label)
References:
1. https://www.tensorflow.org/tutorials/structured_data/imbalanced_data
2. https://stackoverflow.com/questions/60307239/setting-bias-for-multiclass-classification-python-tensorflow-keras#new-answer
"""
from scipy.optimize import fsolve
from math import exp
# to deal with imbalance classification, calculate class_weight
d = dict(df[label].value_counts())
m = np.mean(list(d.values()))
class_weight = {k:m/v for (k,v) in d.items()} #e.g. {0: 1.6282051282051282, 1: 0.7604790419161677, 2: 0.9338235294117647}
# define classes frequency list
frequency = list(list(d.values())/sum(d.values()))
# define equations to solve initial bias
def eqn(x, frequency=frequency):
sum_exp = sum([exp(x_i) for x_i in x])
return [exp(x[i])/sum_exp - frequency[i] for i in range(len(frequency))]
# calculate init bias
bias_init = fsolve(func=eqn,
x0=[0]*len(frequency),
).tolist()
return class_weight, bias_init
class_weight, bias_init = init_imbalanced_class_weight_bias(df=train_df, lable=label)
I will post a colab notebook if anyone interested.
In case your tf classifier complains about ValueError: ('Could not interpret initializer identifier:', then add the tf.keras.initializers.Constant() around bias_init:
def init_imbalanced_class_weight_bias(...)
...
return class_weight, tf.keras.initializers.Constant(bias_init)
I am implementing a CNN for an highly unbalanced classification problem and I would like to implement custum metrics in tensorflow to use the Select Best Model callback.
Specifically I would like to implement the balanced accuracy score, which is the average of the recall of each class (see sklearn implementation here), does someone know how to do it?
I was facing the same issue so I implemented a custom class based off SparseCategoricalAccuracy:
class BalancedSparseCategoricalAccuracy(keras.metrics.SparseCategoricalAccuracy):
def __init__(self, name='balanced_sparse_categorical_accuracy', dtype=None):
super().__init__(name, dtype=dtype)
def update_state(self, y_true, y_pred, sample_weight=None):
y_flat = y_true
if y_true.shape.ndims == y_pred.shape.ndims:
y_flat = tf.squeeze(y_flat, axis=[-1])
y_true_int = tf.cast(y_flat, tf.int32)
cls_counts = tf.math.bincount(y_true_int)
cls_counts = tf.math.reciprocal_no_nan(tf.cast(cls_counts, self.dtype))
weight = tf.gather(cls_counts, y_true_int)
return super().update_state(y_true, y_pred, sample_weight=weight)
The idea is to set each class weight inversely proportional to its size.
This code produces some warnings from Autograph but I believe those are Autograph bugs, and the metric seems to work fine.
There are 3 ways I can think of tackling the situation :-
1)Random Under-sampling - In this method you can randomly remove samples from the majority classes.
2)Random Over-sampling - In this method you can increase the samples by replicating them.
3)Weighted cross entropy - You can also use weighted cross entropy so that the loss value can be compensated for the minority classes. See here
I have personally tried method2 and it does increase my accuracy by significant value but it may vary from dataset to dataset
NOTE
It appears that the implementation/API of the Recall class, which I used as a template for my answer, has been modified in the newer TF versions (as pointed out by #guilaumme-gaudin), so I recommend you look at the Recall implementation used in your current TF version and take it from there to implement the metric using the same approach I describe in the original post, this way I don't have to update my answer every time the TF team modifies the implementation/API of its metrics.
Original post
I'm not an expert in Tensorflow but using a bit of pattern matching between metrics implementations in the tf source code I came up with this
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.metrics import Metric
from tensorflow.python.keras.utils import metrics_utils
from tensorflow.python.ops import init_ops
from tensorflow.python.ops import math_ops
from tensorflow.python.keras.utils.generic_utils import to_list
class BACC(Metric):
def __init__(self, thresholds=None, top_k=None, class_id=None, name=None, dtype=None):
super(BACC, self).__init__(name=name, dtype=dtype)
self.init_thresholds = thresholds
self.top_k = top_k
self.class_id = class_id
default_threshold = 0.5 if top_k is None else metrics_utils.NEG_INF
self.thresholds = metrics_utils.parse_init_thresholds(
thresholds, default_threshold=default_threshold)
self.true_positives = self.add_weight(
'true_positives',
shape=(len(self.thresholds),),
initializer=init_ops.zeros_initializer)
self.true_negatives = self.add_weight(
'true_negatives',
shape=(len(self.thresholds),),
initializer=init_ops.zeros_initializer)
self.false_positives = self.add_weight(
'false_positives',
shape=(len(self.thresholds),),
initializer=init_ops.zeros_initializer)
self.false_negatives = self.add_weight(
'false_negatives',
shape=(len(self.thresholds),),
initializer=init_ops.zeros_initializer)
def update_state(self, y_true, y_pred, sample_weight=None):
return metrics_utils.update_confusion_matrix_variables(
{
metrics_utils.ConfusionMatrix.TRUE_POSITIVES: self.true_positives,
metrics_utils.ConfusionMatrix.TRUE_NEGATIVES: self.true_negatives,
metrics_utils.ConfusionMatrix.FALSE_POSITIVES: self.false_positives,
metrics_utils.ConfusionMatrix.FALSE_NEGATIVES: self.false_negatives,
},
y_true,
y_pred,
thresholds=self.thresholds,
top_k=self.top_k,
class_id=self.class_id,
sample_weight=sample_weight)
def result(self):
"""
Returns the Balanced Accuracy (average between recall and specificity)
"""
result = (math_ops.div_no_nan(self.true_positives, self.true_positives + self.false_negatives) +
math_ops.div_no_nan(self.true_negatives, self.true_negatives + self.false_positives)) / 2
return result
def reset_states(self):
num_thresholds = len(to_list(self.thresholds))
K.batch_set_value(
[(v, np.zeros((num_thresholds,))) for v in self.variables])
def get_config(self):
config = {
'thresholds': self.init_thresholds,
'top_k': self.top_k,
'class_id': self.class_id
}
base_config = super(BACC, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
I've simply taken the Recall class implementation from the source code as a template and I extended it to make sure it has a TP,TN,FP and FN defined.
After that I modified the result method so that it calculates balanced accuracy and voila :)
I compared the results from this with sklearn's balanced accuracy score and the values matched so I think it's correct, but do double check just in case.
I have not tested this code yet, but looking at the source code of tensorflow==2.1.0, this might work for the binary classification case:
from tensorflow.keras.metrics import Recall
from tensorflow.python.ops import math_ops
class BalancedBinaryAccuracy(Recall):
def result(self):
result = (math_ops.div_no_nan(self.true_positives, self.true_positives + self.false_negatives) +
math_ops.div_no_nan(self.true_negatives, self.true_negatives + self.false_positives)) / 2
return result[0] if len(self.thresholds) == 1 else result
As an alternative to writing a custom metric, you can write a custom callback using the metrics already implemented ad available via the training logs. For example you can define the training balanced accuracy callback like this:
class TrainBalancedAccuracyCallback(tf.keras.callbacks.Callback):
def __init__(self, **kargs):
super(TrainBalancedAccuracyCallback, self).__init__(**kargs)
def on_epoch_end(self, epoch, logs={}):
train_sensitivity = logs['tp'] / (logs['tp'] + logs['fn'])
train_specificity = logs['tn'] / (logs['tn'] + logs['fp'])
logs['train_sensitivity'] = train_sensitivity
logs['train_specificity'] = train_specificity
logs['train_balacc'] = (train_sensitivity + train_specificity) / 2
print('train_balacc', logs['train_balacc'])
and the same for the validation:
class ValBalancedAccuracyCallback(tf.keras.callbacks.Callback):
def __init__(self, **kargs):
super(ValBalancedAccuracyCallback, self).__init__(**kargs)
def on_epoch_end(self, epoch, logs={}):
val_sensitivity = logs['val_tp'] / (logs['val_tp'] + logs['val_fn'])
val_specificity = logs['val_tn'] / (logs['val_tn'] + logs['val_fp'])
logs['val_sensitivity'] = val_sensitivity
logs['val_specificity'] = val_specificity
logs['val_balacc'] = (val_sensitivity + val_specificity) / 2
print('val_balacc', logs['val_balacc'])
and then you can use these as values to the callback argument of the fit method of the model.
I have been struggling on this and could not get it to work. hope someone can help me with this.
I want to calculate the entropy on each row of the tensor. Because my data are float numbers not integers I think I need to use bin_histogram.
For example a sample of my data is tensor =[[0.2, -0.1, 1],[2.09,-1.4,0.9]]
Just for information My model is seq2seq and written in keras with tensorflow backend.
This is my code so far: I need to correct rev_entropy
class entropy_measure(Layer):
def __init__(self, beta,batch, **kwargs):
self.beta = beta
self.batch = batch
self.uses_learning_phase = True
self.supports_masking = True
super(entropy_measure, self).__init__(**kwargs)
def call(self, x):
return K.in_train_phase(self.rev_entropy(x, self.beta,self.batch), x)
def get_config(self):
config = {'beta': self.beta}
base_config = super(entropy_measure, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
def rev_entropy(self, x, beta,batch):
for i in x:
i = pd.Series(i)
p_data = i.value_counts() # counts occurrence of each value
entropy = entropy(p_data) # get entropy from counts
rev = 1/(1+entropy)
return rev
new_f_w_t = x * (rev.reshape(rev.shape[0], 1))*beta
return new_f_w_t
Any input is much appreciated:)
It looks like you have a series of questions that come together on this issue. I'll settle it here.
You calculate entropy in the following form of scipy.stats.entropy according to your code:
scipy.stats.entropy(pk, qk=None, base=None)
Calculate the entropy of a distribution for given probability values.
If only probabilities pk are given, the entropy is calculated as S =
-sum(pk * log(pk), axis=0).
Tensorflow does not provide a direct API to calculate entropy on each row of the tensor. What we need to do is to implement the above formula.
import tensorflow as tf
import pandas as pd
from scipy.stats import entropy
a = [1.1,2.2,3.3,4.4,2.2,3.3]
res = entropy(pd.value_counts(a))
_, _, count = tf.unique_with_counts(tf.constant(a))
# [1 2 2 1]
prob = count / tf.reduce_sum(count)
# [0.16666667 0.33333333 0.33333333 0.16666667]
tf_res = -tf.reduce_sum(prob * tf.log(prob))
with tf.Session() as sess:
print('scipy version: \n',res)
print('tensorflow version: \n',sess.run(tf_res))
scipy version:
1.329661348854758
tensorflow version:
1.3296613488547582
Then we need to define a function and achieve for loop through tf.map_fn in your custom layer according to above code.
def rev_entropy(self, x, beta,batch):
def row_entropy(row):
_, _, count = tf.unique_with_counts(row)
prob = count / tf.reduce_sum(count)
return -tf.reduce_sum(prob * tf.log(prob))
value_ranges = [-10.0, 100.0]
nbins = 50
new_f_w_t = tf.histogram_fixed_width_bins(x, value_ranges, nbins)
rev = tf.map_fn(row_entropy, new_f_w_t,dtype=tf.float32)
new_f_w_t = x * 1/(1+rev)*beta
return new_f_w_t
Notes that the hidden layer will not produce a gradient that cannot propagate backwards since entropy is calculated on the basis of statistical probabilistic values. Maybe you need to rethink your hidden layer structure.
I train an image classifier using Keras up to around 98% test accuracy. Now I know that the overall accuracy is 98%, but i want to know the accuracy/error per distinct class/label.
Has Keras a builtin function for that or would I have to test this myself per class/label?
Update: Thanks #gionni. I didn't know the actual term was "Confusion Matrix". But that's what I am actually looking for. That being said, is there a function to generate one? I have to use Keras 1.2.2 by the way.
I had similar issue so I could share my code with you. The following function computes a single class accuracy:
def single_class_accuracy(interesting_class_id):
def fn(y_true, y_pred):
class_id_preds = K.argmax(y_pred, axis=-1)
# Replace class_id_preds with class_id_true for recall here
positive_mask = K.cast(K.equal(class_id_preds, interesting_class_id), 'int32')
true_mask = K.cast(K.equal(y_true, interesting_class_id), 'int32')
acc_mask = K.cast(K.equal(positive_mask, true_mask), 'float32')
class_acc = K.mean(acc_mask)
return class_acc
return fn
Now - if you want to get an accuracy for 0 class you could add it to metrics while compiling a model:
model.compile(..., metrics=[..., single_class_accuracy(0)])
If you want to have all classes accuracy you could type:
model.compile(...,
metrics=[...] + [single_class_accuracy(i) for i in range(nb_of_classes)])
There may be better options, but you can use this:
import numpy as np
#gather each true label
distinct, counts = np.unique(trueLabels,axis=0,return_counts=True)
for dist,count in zip(distinct, counts):
selector = (trueLabels == dist).all(axis=-1)
selectedX = testData[selector]
selectedY = trueLabels[selector]
print('\n\nEvaluating for ' + str(count) + ' occurrences of class ' + str(dist))
print(model.evaluate(selectedX,selectedY,verbose=0))
The Keras losses page says that if we have a custom loss function, then "the actual optimized objective is the mean of the output array across all datapoints." Is there any way we can optimize the median of the output array across all datapoints (instead of the mean)?
In order to do this, you would need to go down to tensorflow level
import keras
import tensorflow
def pick_median(arg_tensor):
the_upper_tensor = tensorflow.contrib.distributions.percentile(arg_tensor, 50, interpolation='higher')
the_lower_tensor = tensorflow.contrib.distributions.percentile(arg_tensor, 50, interpolation='lower')
final_tensor = (the_upper_tensor + the_lower_tensor) / 2
# print(the_count.eval(session=keras.backend.get_session()))
return final_tensor
Here is how you would define, let's say, median_squared_error loss function:
def median_squared_error(arg_y_true,
arg_y_pred):
final_tensor = keras.backend.square(arg_y_pred - arg_y_true)
final_tensor = pick_median(arg_tensor=final_tensor)
return final_tensor