To create a custom metrics for regression according to the rule book of Keras' documentation - tensorflow

I found two main sources about it.
A tutorial, not done according to the rule book (I prefer avoid)
Keras' documentation (what I prefer to avoid surprises)
I prefer to follow Keras' documentation to avoid memory leak as it's the case for some people who try custom approaches with Keras.
But what Keras' is showing in the documentation is about classification. This is not my case.
So I tried to look at the source code of Keras. Precisely in the file: /lib/python3.7/site-packages/tensorflow_core/python/keras/metrics.py. It does not help me at all because most of metrics (some exception are classification metrics) are all done with a wrapper as the following code:
#keras_export('keras.metrics.MeanSquaredError')
class MeanSquaredError(MeanMetricWrapper):
"""Computes the mean squared error between `y_true` and `y_pred`.
For example, if `y_true` is [0., 0., 1., 1.], and `y_pred` is [1., 1., 1., 0.]
the mean squared error is 3/4 (0.75).
Usage:
```python
m = tf.keras.metrics.MeanSquaredError()
m.update_state([0., 0., 1., 1.], [1., 1., 1., 0.])
print('Final result: ', m.result().numpy()) # Final result: 0.75
```
Usage with tf.keras API:
```python
model = tf.keras.Model(inputs, outputs)
model.compile('sgd', metrics=[tf.keras.metrics.MeanSquaredError()])
```
"""
def __init__(self, name='mean_squared_error', dtype=None):
super(MeanSquaredError, self).__init__(
mean_squared_error, name, dtype=dtype)
As you can see there's only the constructor method, no good inspiration available for the udpate_state method that I need.
Where can I find it ?
python 3.7.7
tensorflow 2.1.0
keras-applications 1.0.8
keras-preprocessing 1.1.0

You can use a loss function as a metric, so you can extend keras.losses.Loss instead. You only need to override call as shown in the documentation
import tensorflow as tf
class MeanSquaredError(tf.keras.losses.Loss):
def call(self, y_true, y_pred):
y_true = tf.cast(y_true, y_pred.dtype)
return tf.math.reduce_mean(tf.math.square(y_pred - y_true), axis=-1)

Related

batch axis in keras custom layer

I want to make a custom layer that does the following, given a batch of input vectors.
For each vector a in the batch:
get the first element a[0].
multiply the vector a by a[0] elementwise.
So if the batch is
[[ 1., 2., 3.],
[ 4., 5., 6.],
[ 7., 8., 9.],
[10., 11., 12.]]
This should be a batch of 4 vectors, each with dimension 3 (or am I wrong here?).
Then my layer should transform the batch to the following:
[[ 1., 2., 3.],
[ 16., 20., 24.],
[ 49., 56., 63.],
[100., 110., 120.]]
Here is my implementation for the layer:
class MyLayer(keras.layers.Layer):
def __init__(self, activation=None, **kwargs):
super().__init__(**kwargs)
self.activation = keras.activations.get(activation)
def call(self, a):
scale = a[0]
return self.activation(a * scale)
def get_config(self):
base_config = super().get_config()
return {**base_config,
"activation": keras.activations.serialize(self.activation)}
But the output is different from what I expected:
batch = tf.Variable([[1,2,3],
[4,5,6],
[7,8,9],
[10,11,12]], dtype=tf.float32)
layer = MyLayer()
print(layer(batch))
Output:
tf.Tensor(
[[ 1. 4. 9.]
[ 4. 10. 18.]
[ 7. 16. 27.]
[10. 22. 36.]], shape=(4, 3), dtype=float32)
It looks like the implementation actually treats each column as a vector, which is strange to me because other pre-written models, such as the sequential model, specify the input shape to be (batch_size, ...), which means each row, instead of column, is a vector.
How should I modify my code so that it behaves the way I want?
Actually, your input shape is (4,3). So when you slice this tensor by a[0] it gets the first row which is [1,2,3]. To get what you want you should instead get the first column and then transpose your matrix to give you the desired matrix like this:
def call(self, a):
scale = a[:,0]
return tf.transpose(self.activation(tf.transpose(a) * scale))

How to specify spearman rank correlation as a loss function in keras?

I wanted to write a loss function that maximizes the spearman rank correlation between two vectors in keras. Unfortunately I could not find an existing implementation, nor a good method to calculate the rank of a vector in keras, so that I could use the formula to implement it myself
def rank_correlation(y_true, y_pred):
pass
model = tensorflow.keras.Sequential()
#### More model code
model.compile(loss=rank_correlation)
Can anyone please help me implement rank_correlation ?
You can try something like as follows, referenced.
from scipy.stats import spearmanr
def compute_spearmanr(y, y_pred):
spearsum = 0
cnt = 0
for col in range(y_pred.shape[1]):
v = spearmanr(y_pred[:,col], y[:,col]).correlation
if np.isnan(v):
continue
spearsum += v
cnt += 1
res = spearsum / cnt
return res
a = np.array([[2., 1., 2., 3.],[3., 3., 4., 5.]] )
b = np.array([[1., 0., 0., 3.], [1., 0., 3., 3.]])
compute_spearmanr(a, b)
0.9999999999999999

Custom Keras binary_crossentropy loss function not working

I’m trying to re-define keras’s binary_crossentropy loss function so that I can customize it but it’s not giving me the same results as the existing one.
I'm using TF 1.13.1 with Keras 2.2.4.
I went through Keras’s github code. My understanding is that the loss in model.compile(optimizer='adam', loss='binary_crossentropy', metrics =['accuracy']), is defined in losses.py, using binary_crossentropy defined in tensorflow_backend.py.
I ran a dummy data and model to test it. Here are my findings:
The custom loss function outputs the same results as keras’s one
Using the custom loss in a keras model gives different accuracy results
from numpy.random import seed
seed(1)
from tensorflow import set_random_seed
set_random_seed(2)
import tensorflow as tf
from keras import losses
import keras.backend as K
import keras.backend.tensorflow_backend as tfb
from keras.layers import Dense
from keras import Sequential
#Dummy check of loss output
def binary_crossentropy_custom(y_true, y_pred):
return K.mean(binary_crossentropy_custom_tf(y_true, y_pred), axis=-1)
def binary_crossentropy_custom_tf(target, output, from_logits=False):
"""Binary crossentropy between an output tensor and a target tensor.
# Arguments
target: A tensor with the same shape as `output`.
output: A tensor.
from_logits: Whether `output` is expected to be a logits tensor.
By default, we consider that `output`
encodes a probability distribution.
# Returns
A tensor.
"""
# Note: tf.nn.sigmoid_cross_entropy_with_logits
# expects logits, Keras expects probabilities.
if not from_logits:
# transform back to logits
_epsilon = tfb._to_tensor(tfb.epsilon(), output.dtype.base_dtype)
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
output = tf.log(output / (1 - output))
return tf.nn.sigmoid_cross_entropy_with_logits(labels=target,
logits=output)
logits = tf.constant([[-3., -2.11, -1.22],
[-0.33, 0.55, 1.44],
[2.33, 3.22, 4.11]])
labels = tf.constant([[1., 1., 1.],
[1., 1., 0.],
[0., 0., 0.]])
custom_sigmoid_cross_entropy_with_logits = binary_crossentropy_custom(labels, logits)
keras_binary_crossentropy = losses.binary_crossentropy(y_true=labels, y_pred=logits)
with tf.Session() as sess:
print('CUSTOM sigmoid_cross_entropy_with_logits: ', sess.run(custom_sigmoid_cross_entropy_with_logits), '\n')
print('KERAS keras_binary_crossentropy: ', sess.run(keras_binary_crossentropy), '\n')
#CUSTOM sigmoid_cross_entropy_with_logits: [16.118095 10.886106 15.942386]
#KERAS keras_binary_crossentropy: [16.118095 10.886106 15.942386]
#Dummy check of model accuracy
X_train = tf.random.uniform((3, 5), minval=0, maxval=1, dtype=tf.dtypes.float32)
labels = tf.constant([[1., 0., 0.],
[0., 0., 1.],
[1., 0., 0.]])
model = Sequential()
#First Hidden Layer
model.add(Dense(5, activation='relu', kernel_initializer='random_normal', input_dim=5))
#Output Layer
model.add(Dense(3, activation='sigmoid', kernel_initializer='random_normal'))
#I ran model.fit for each model.compile below 10 times using the same X_train and provide the range of accuracy measurement
# model.compile(optimizer='adam', loss='binary_crossentropy', metrics =['accuracy']) #0.748 < acc < 0.779
# model.compile(optimizer='adam', loss=losses.binary_crossentropy, metrics =['accuracy']) #0.761 < acc < 0.778
model.compile(optimizer='adam', loss=binary_crossentropy_custom, metrics =['accuracy']) #0.617 < acc < 0.663
history = model.fit(X_train, labels, steps_per_epoch=100, epochs=1)
I'd expect the custom loss function to give similar model accuracy output but it does not. Any idea? Thanks!
Keras automatically selects which accuracy implementation to use according to the loss, and this won't work if you use a custom loss. But in this case you can just explictly use the right accuracy, which is binary_accuracy:
model.compile(optimizer='adam', loss=binary_crossentropy_custom, metrics =['binary_accuracy'])

Is the a bug in function save_weights/load_weights with custom layer in tensorflow 1.13.1?

It took me more than a day, so frustrated. I doubt this is a bug in Tensorflow 1.13.1 (a stable version).
In summary, I created a custom model in Model Subclassing style, which contained only 1 custom layer. After initial, I dumped its trainable weights to file and restore it, by using save_weights and load_weights functions. The trainable weights before and after saving were different.
I also ran the same test on Tensorflow 2.0.0a0, and it turned out this version did not get this phenomenon.
My custom layer:
class EncodingLayer(tf.keras.layers.Layer):
def __init__(self, out_size):
super().__init__()
self.rnn_layer = tf.keras.layers.GRU(out_size, return_sequences=True, return_state=True, recurrent_initializer='glorot_uniform')
def call(self, X, **kwargs):
output, state = self.rnn_layer(X)
return output, state
This is the main part:
class EncodingModel(tf.keras.Model):
def __init__(self):
super().__init__()
self.encoder_layer = EncodingLayer(out_size=1)
def infer(self, inputs):
output, state = self.encoder_layer(inputs)
return output
if __name__ == '__main__':
# Comment line below for running in TF 2.0
tf.enable_eager_execution()
# shape == (2, 3, 2)
inputs = tf.convert_to_tensor([
[[1., 2.], [2., 3.], [4., 4.]],
[[1., 2.], [2., 3.], [4., 4.]],
])
model = EncodingModel()
# Just for building the graph
model.infer(inputs)
print('Before saving model: ', model.trainable_weights[0].numpy().mean())
model.save_weights('weight')
new_model = EncodingModel()
new_model.infer(inputs)
new_model.load_weights('weight')
print('Loaded model: ', new_model.trainable_weights[0].numpy().mean())
The result when running in TF 1.13.1:
Before saving model: 0.28864467
Loaded model: 0.117300846
The result when running in TF 2.0.0a0:
Before saving model: -0.06922924
Loaded model: -0.06922924
Although the result suggests that it might be a bug, I was not so sure. Since the code is very basic, if a bug like that exists, it should be discovered easily. I did a lot of searching but found no one mentioning about it. Thus, I'd guess there's something I misunderstood :)

How is the gradient and hessian of logarithmic loss computed in the custom objective function example script in xgboost's github repository?

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script.
I've simplified the function to take numpy arrays, and generated y_hat and y_true which are a sample of the values used in the script.
Here is a simplified example:
import numpy as np
def loglikelihoodloss(y_hat, y_true):
prob = 1.0 / (1.0 + np.exp(-y_hat))
grad = prob - y_true
hess = prob * (1.0 - prob)
return grad, hess
y_hat = np.array([1.80087972, -1.82414818, -1.82414818, 1.80087972, -2.08465433,
-1.82414818, -1.82414818, 1.80087972, -1.82414818, -1.82414818])
y_true = np.array([1., 0., 0., 1., 0., 0., 0., 1., 0., 0.])
loglikelihoodloss(y_hat, y_true)
The log loss function is the sum of where .
The gradient (with respect to p) is then however in the code its .
Likewise the second derivative (with respect to p) is however in the code it is .
How are the equations equal?
The log loss function is given as:
where
Taking the partial derivative we get the gradient as
Thus we get the negative of gradient as p-y.
Similar calculations can be done to obtain the hessian.