I don't know hot to create a model that is maximizing binary cross_entropy loss in a keras model.
research:
1.https://intellipaat.com/community/17707/how-to-maximize-loss-function-in-keras
that said:
Simply multiply the loss by -1 to maximize the loss function while trying to minimize it:
new_loss = -loss
but using:
model.compile(loss=-1 * 'binary_crossentropy', optimizer=adam_optimizer())
resulted in this error:
ValueError: The model cannot be compiled because it has no loss to optimize.
https://stats.stackexchange.com/questions/303229/why-does-keras-binary-crossentropy-loss-function-return-wrong-values
gave me a custom function that approximates the keras binary_crossentropy loss:
import keras.backend as K
def binary_crossentropy(y_true, y_pred):
result = []
for i in range(len(y_pred)):
y_pred[i] = [max(min(x, 1 - K.epsilon()), K.epsilon()) for x in y_pred[i]]
result.append(-np.mean([y_true[i][j] * math.log(y_pred[i][j]) + (1 - y_true[i][j]) * math.log(1 - y_pred[i][j]) for j in range(len(y_pred[i]))]))
return np.mean(result)
but I can not use it since it results in the error:
len is not well defined for symbolic Tensors. (43_54/Sigmoid:0) Please call `x.shape` rather than `len(x)` for shape information.
when I replace len with .shape[0]
I get the another error:
__index__ returned non-int (type NoneType)
I tinkered with the syntax in several more ways but nothing seems to work.
any ideas?
python 3.6
tensorflow 1.15
keras 2.3.1
You just need to define a new loss, based on the keras implementation:
def neg_binary_crossentropy(y_true, y_pred):
return -1.0 * keras.losses.binary_crossentropy(y_true, y_pred)
And then use it in model.compile:
model.compile(loss=neg_binary_crossentropy, optimizer="adam")
Related
I am using VGG16 for image segmentation with the loss function "balanced categorical entropy" using the code
beta=0.5
def balanced_cross_entropy(beta):
def loss(y_true, y_pred):
weight_a = beta * tf.cast(y_true, tf.float32)
weight_b = (1 - beta) * tf.cast(1 - y_true, tf.float32)
o = (tf.math.log1p(tf.exp(-tf.abs(y_pred))) + tf.nn.relu(-y_pred)) * (weight_a + weight_b) + y_pred * weight_b
return tf.reduce_mean(o)
return loss
Everything works fine. Now I save this model in the h5 file using the code.
vgg.save('vgg.h5')
But when I use the load_model from Keras
model = load_model('vgg.h5', custom_objects={'balanced_cross_entropy(beta)': balanced_cross_entropy(beta)})
I encounter an error.
Unknown loss function: loss. Please ensure this object is passed to the `custom_objects` argument.
Can anybody help, I suspect the problem may be due to beta?
If you want to only perform inference, you can avoid this problem by specifying
model = load_model('vgg.h5',compile=False)
Otherwise, you need to load the in the following way:
model = load_model("vgg.h5", custom_objects={'loss': balanced_cross_entropy(beta)}); in your code you wrote balanced_cross_entropy(beta) instead of loss.
Short explanation:
The name of the key in custom_object is actually the name of the inner function (which is in fact returned by balanced_cross_entropy(beta); the name of the outer function is actually the value of the <key,value> pair in the custom_object dictionary.
I am trying to use the output of one neural network to compute the loss value for another network. As the first network is approximating another function (L2 distance) I would like to provide the gradients myself, as if it had come from an L2 function.
An example of my loss function in simplified code is:
#tf.custom_gradient
def loss_function(model_1_output):
def grad(dy, variables=None):
gradients = 2 * pred
return gradients
pred = model_2(model_1_output)
loss = pred ** 2
return loss, grad
This is called in a standard tensorflow 2.0 custom training loop such as:
with tf.GradientTape() as tape:
model_1_output = model_1(training_data)
loss = loss_function(model_1_output)
gradients = tape.gradient(loss, model_1.trainable_variables)
optimizer.apply_gradients(zip(gradients, model.trainable_variables)
However, whenever I try to run this I keep getting the error:
ValueError: Attempt to convert a value (<model.model_2 object at 0x7f41982e3240>) with an unsupported type (<class 'model.model_2'>) to a Tensor.
The whole point of using the custom_gradients decorator is that I don't want the model_2 in the loss function to be included in the back propagation as I give it the gradients manually.
How can I make tensorflow completely ignore anything inside the loss function? So that for example I could do non-differetiable operations. I have tried using with tape.stop_recording() but I always result in a no gradients found error.
Using:
OS: Ubuntu 18.04
tensorflow: 2.0.0
python: 3.7
I'm trying to train DNN that outputs 3 values (x,y,z) where x and y are coordinates of the object I'm looking for and z is the probability that object is present
I need custom loss function:
If z_true<0.5 I don't care of x and y values, so error should be equal to (0, 0, sqr(z_true - z_pred))
otherwise error should be like (sqr(x_true - x_pred), sqr(y_true - y_pred), sqr(z_true - z_pred))
I'm in a struggle with mixing tensors and if statements together.
Maybe this example of a custom loss function will get you up and running. It shows how you can mix tensors with if statements.
def conditional_loss_function(l):
def loss(y_true, y_pred):
if l == 0:
return loss_funtion1(y_true, y_pred)
else:
return loss_funtion2(y_true, y_pred)
return loss
model.compile(loss=conditional_loss_function(l), optimizer=...)
Use switch from Keras backend: https://keras.io/backend/#switch
It is similar to tf.cond
How to create a custom loss in Keras described here: Make a custom loss function in keras
I have a problem working with Tensorflow and keras. That problem we could explain in this way:
We have a model (convolutional neural network) which has output of the form [None, 7, 7, 6]. We have a function 'custom_loss'. This function has y_true and y_pred parameters. They are of the form [7,7,6]. When I compile it, I got error message: TypeError: must be real number, not Tensor. I suppose there is mistake when I call y_pred[k][l][m] and y_true[k][l][m] but I don't know how to fix this to include this None in [None, 7, 7, 6]. Please help.
Update: Here is the code
def custom_loss(y_true, y_pred):
loss = 0
for i in range(S*S):
k, l = i%S, i//S
first_part = 5* sum([(y_pred[k][l][m] - y_true[k][l][m])**2 for m in range(1,3)])
second_part = 5 * sum([(math.sqrt(y_pred[k][l][m]) - math.sqrt(y_true[k][l][m])) ** 2 for m in range(3, 5)])
third_part = 5* sum([(y_pred[k][l][m] - y_true[k][l][m])**2 for m in [0, 5]])
if y_true[k][l][0] > 0.5:
loss += first_part + second_part + third_part
else:
loss += 0.5 * (y_pred[k][l][0] - y_true[k][l][0])**2
return loss
In keras (and TensorFlow without eager execution) you cannot access the content of a tensor. Therefore, lines as
loss += 0.5 * (y_pred[k][l][0] - y_true[k][l][0])**2
will fail. You can try to use the eager execution mode of TensorFlow together with keras as explained here.
In general you should always try to express these things just with built-in functions of the keras backend or with TensorFlow operations. Just try to express your loss function using matrix/vector notation and then it is easier (maybe we can also help you) to express this in keras.
When you wirte a loss function in keras (with tensorflow backend) it's for building your execution graph but not for execution directly.
You have to use tensorflow or keras backend function to define your loss function. If you compile your model keras (and tensorflow as backend) try to build your execution graph and therefore send tensors trough your loss function. the math package does not support tensors. Also is not possible to use if in your loss function since it's not derivable. Instead you could use a sigmoid function which is very close to a step function.
I am trying to write a cusom Keras loss function in which I process the tensors in sub-vector chunks. For example, if an output tensor represented a concatenation of quaternion coefficients (i.e. w,x,y,z,w,x,y,z...) I might wish to normalize each quaternion before calculating the mean squared error in a loss function like:
def norm_quat_mse(y_true, y_pred):
diff = y_pred - y_true
dist = 0
for i in range(0,16,4):
dist += K.sum( K.square(diff[i:i+4] / K.sqrt(K.sum(K.square(diff[i:i+4])))))
return dist/4
While Keras will accept this function without error and use in training, it outputs a different loss value from when applied as an independent function and when using model.predict(), so I suspect it is not working properly. None of the built-in Keras loss functions use this per-chunk processing approach, is it possible to do this within Keras' auto-differentiation framework?
Try:
def norm_quat_mse(y_true, y_pred):
diff = y_pred - y_true
dist = 0
for i in range(0,16,4):
dist += K.sum( K.square(diff[:,i:i+4] / K.sqrt(K.sum(K.square(diff[:,i:i+4])))))
return dist/4
You need to know that shape of y_true and y_pred is (batch_size, output_size) so you need to skip first dimension during computations.