setting bias for multiclass classification python tensorflow keras - tensorflow

Question

Attached model shows how to add bias in case of the unbalanced classification problem initial_bias = np.log([pos/neg]). Is there a way to add bias if you have multi-class classification with unbalanced data, Say 5 classes where classes are have distribution (0.4,0.3,0.2.0.08 and 0.02)
2) also how to calculate and use class weights in such case?
update 1
I found a way to apply weights, still not sure how to use bias
#####adding weights 20 Feb
weight_for_0 = ( 1/ 370)*(370+ 977+ 795)/3
weight_for_1 = ( 1/ 977)*(370+ 977+ 795)/3
weight_for_2 = (1 / 795)*(370+ 977+ 795)/3
#array([0, 1, 2]), array([370, 977, 795])
class_weights_dict = {0: weight_for_0, 1: weight_for_1, 2:weight_for_2}
class_weights_dict
Dcnn.fit(train_dataset,
epochs=NB_EPOCHS,
callbacks=[MyCustomCallback()],verbose=2,validation_data=test_dataset, class_weight=class_weights_dict)

Answer 1

Considering that you're using 'softmax':
softmax = exp(neurons) / sum(exp(neurons))
And that you want the results of the classes to be:
frequency = [0.4 , 0.3 , 0.2 , 0.08 , 0.02]
Biases should be given by the equation (elementwise):
frequency = exp(biases) / sum(exp(biases))
This forms a system of equations:
f1 = e^b1 / (e^b1 + e^b2 + ... + e^b5)
f2 = e^b2 / (e^b1 + e^b2 + ... + e^b5)
...
f5 = e^b5 / (e^b1 + e^b2 + ... + e^b5)
If you can solve this system of equations, you get the biases you want.
I used excel and test-error method to determine that for the frequencies you wanted, your biases should be respectively:
[1.1 , 0.81 , 0.4 , -0.51 , -1.9]
I don't really know how to solve that system easily, but you can keep experimenting with excel or another thing until you reach the solution.
Adding the biases to the layer - method 1.
Use a name when defining the layer, like:
self.last_dense = layers.Dense(units=3, activation="softmax", name='last_layer')
You may need to build the model first, so:
dummy_predictions = model.predict(np.zeros((1,) + input_shape))
Then you get the weights:
weights_and_biases = model.get_layer('last_layer').get_weights()
w, b = weights_and_biases
new_biases = np.array([-0.45752, 0.51344, 0.30730])
model.get_layer('last_layer').set_weights([w, new_biases])
Method 2
def bias_init(bias_shape):
return K.variable([-0.45752, 0.51344, 0.30730])
self.last_dense = layers.Dense(units=3, activation="softmax", bias_initializer=bias_init)

Answer 2

Just in addition to #Daniel Möller's answer, to solve the system of equations
f1 = e^b1 / (e^b1 + e^b2 + ... + e^b5)
...
f5 = e^b5 / (e^b1 + e^b2 + ... + e^b5)
You don't need excel or anything. Just compute bi = ln(fi).

Answer 3

To calculate fi = e^bi / (sum of e^bj), note that fi/fj = e^(bi-bj). Suppose the lowest frequency is fk. You can set bk= 0 and then compute every other class bias with bi = bj + ln(fi/fj).

Answer 4

A complete answer is here:
### To solve that set of nonlinear equations, use scipy fsolve
from scipy.optimize import fsolve
from math import exp
# define the frequency of different classes
f=(0.4, 0.3, 0.2, 0.08, 0.02)
# define the equation
def eqn(x, frequency):
sum_exp = sum([exp(x_i) for x_i in x])
return [exp(x[i])/sum_exp - frequency[i] for i in range(len(frequency))]
# calculate bias init
bias_init = fsolve(func=eqn,
x0=[0]*len(f),
).tolist()
bias_init
To put all things together
def init_imbalanced_class_weight_bias(df:pd.DataFrame, lable:str):
"""To handle imbalanced classification, provide initial bias list and class weight dictionary to 2 places in a tf classifier
1) In the last layer of classifier: tf.keras.layers.Dense(..., bias_initializer = bias_init)
2) model.fit(train_ds, #x=dict(X_train), y=y_train,
batch_size=batch_size,
validation_data= valid_ds, #(dict(X_test), y_test),
epochs=epochs,
callbacks=callbacks,
class_weight=class_weight,
)
Args:
df:pd.DataFrame=train_df
label:str
Returns:
class_weight:dict, e.g. {0: 1.6282051282051282, 1: 0.7604790419161677, 2: 0.9338235294117647}
bias_init:list e.g. [0.3222079660508266, 0.1168690393701237, -0.43907701967633633]
Examples:
class_weight, bias_init = init_imbalanced_class_weight_bias(df=train_df, lable=label)
References:
1. https://www.tensorflow.org/tutorials/structured_data/imbalanced_data
2. https://stackoverflow.com/questions/60307239/setting-bias-for-multiclass-classification-python-tensorflow-keras#new-answer
"""
from scipy.optimize import fsolve
from math import exp
# to deal with imbalance classification, calculate class_weight
d = dict(df[label].value_counts())
m = np.mean(list(d.values()))
class_weight = {k:m/v for (k,v) in d.items()} #e.g. {0: 1.6282051282051282, 1: 0.7604790419161677, 2: 0.9338235294117647}
# define classes frequency list
frequency = list(list(d.values())/sum(d.values()))
# define equations to solve initial bias
def eqn(x, frequency=frequency):
sum_exp = sum([exp(x_i) for x_i in x])
return [exp(x[i])/sum_exp - frequency[i] for i in range(len(frequency))]
# calculate init bias
bias_init = fsolve(func=eqn,
x0=[0]*len(frequency),
).tolist()
return class_weight, bias_init
class_weight, bias_init = init_imbalanced_class_weight_bias(df=train_df, lable=label)
I will post a colab notebook if anyone interested.
In case your tf classifier complains about ValueError: ('Could not interpret initializer identifier:', then add the tf.keras.initializers.Constant() around bias_init:
def init_imbalanced_class_weight_bias(...)
...
return class_weight, tf.keras.initializers.Constant(bias_init)