I am trying to use some_model.predict(x) within a custom loss function.
I found this custom loss function:
_EPSILON = K.epsilon()
def _loss_tensor(y_true, y_pred):
y_pred = K.clip(y_pred, _EPSILON, 1.0-_EPSILON)
out = -(y_true * K.log(y_pred) + (1.0 - y_true) * K.log(1.0 - y_pred))
return K.mean(out, axis=-1)
But the problem is that model.predict() is expecting a numpy array.
So I looked for how to convert a tensor (y_pred) to a numpy array.
I found tmp = K.tf.round(y_true) but this returns a tensor.
I have also found: x = K.eval(y_true) which takes a Keras variable and returns a numpy array.
This produces the error: You must feed a value for placeholder tensor 'dense_78_target' with dtype float.....
Some people suggested setting the learning phase to true. I did that, but it did not help.
What I just want to do:
def _loss_tensor(y_true, y_pred):
y_tmp_true = first_decoder.predict(y_true)
y_tmp_pred = first_decoder.predict(y_pred)
return keras.losses.binary_crossentropy(y_tmp_true,y_tmp_pred)
Any help would be appreciated.
This works:
sess = K.get_session()
with sess.as_default():
tmp = K.tf.constant([1,2,3]).eval()
print(tmp)
I also tried this now:
tmp = first_decoder(y_true)
This fails the assertion:
assert input_shape[-1]
Maybe someone knows how to resolve this?
Update:
I can now feed it through the model with:
y_t = first_decoder(K.reshape(y_true, (1,512)))
y_p = first_decoder(K.reshape(y_pred, (1,512)))
But when I try to return the binary cross entropy the shape is not right:
Input to reshape is a tensor with 131072 values, but the requested shape has
512
I figured out that 131072 was the product of my batch size and input size (256*512). I then adopted my code to reshape to (256,512) size. The first batch runs fine, but then I get another error that says that the passed size was (96,512).
[SOLVED]Update:
It works now:
def _loss_tensor(y_true, y_pred):
num_ex = K.shape(y_true)[0]
y_t = first_decoder(K.reshape(y_true, (num_ex,512)))
y_p = first_decoder(K.reshape(y_pred, (num_ex,512)))
return keras.losses.binary_crossentropy(y_t,y_p)
Related
I have implemented UNET network described in here
The network is working fine, but in the paper, they have mentioned adding weighted maps into the network for better boundary separation. The weight maps are calculated this way as far as I understand
def unet_weight_map(y, wc=None, w0 = 10, sigma = 5):
"""
Parameters
----------
mask: Numpy array
2D array of shape (image_height, image_width) representing binary mask
of objects.
wc: dict
Dictionary of weight classes.
w0: int
Border weight parameter.
sigma: int
Border width parameter.
Returns
-------
Numpy array
Training weights. A 2D array of shape (image_height, image_width).
"""
labels = label(y)
no_labels = labels == 0
label_ids = sorted(np.unique(labels))[1:]
if len(label_ids) > 1:
distances = np.zeros((y.shape[0], y.shape[1], len(label_ids)))
for i, label_id in enumerate(label_ids):
distances[:,:,i] = distance_transform_edt(labels != label_id)
distances = np.sort(distances, axis=2)
d1 = distances[:,:,0]
d2 = distances[:,:,1]
w = w0 * np.exp(-1/2*((d1 + d2) / sigma)**2) * no_labels
else:
w = np.zeros_like(y)
if wc:
class_weights = np.zeros_like(y)
for k, v in wc.items():
class_weights[y == k] = v
w = w + class_weights
return w
Until here everything is fine. But, my question is that how I can get use of these weight maps in the network. I have a weighted binary cross entropy loss defined as below
def weighted_binary_crossentropy( y_true, y_pred, weight=[1.,2.]):
y_true = K.clip(y_true, K.epsilon(), 1-K.epsilon())
y_pred = K.clip(y_pred, K.epsilon(), 1-K.epsilon())
logloss = -(y_true * K.log(y_pred) * weight[0] + (1 - y_true) * K.log(1 - y_pred)*weight[1])
return K.mean( logloss, axis=-1)
But, here I give the weights as a [a, b] array into the loss for class weights and then feed this loss to the network when compiling. My question is that should I feed those maps into this customized loss function? if so, how? if not, what other way can I use in Keras? Please help. I have read many stack overflow questions related to this problem, but I could not get an answer. I can provide any information regarding my network if needed.
In order to pass your own parameters to a custom loss function, you have 2 ways. You should either subclass loss, or use a wrapper function.
For example you can set a wrapper function like this:
def wrapper_loss(weights=[1.,2.]):
def weighted_binary_crossentropy(y_true, y_pred):
y_true = K.clip(y_true, K.epsilon(), 1-K.epsilon())
y_pred = K.clip(y_pred, K.epsilon(), 1-K.epsilon())
logloss = -(y_true * K.log(y_pred) * weight[0] + (1 - y_true) * K.log(1 - y_pred)*weight[1])
return K.mean(logloss, axis=-1)
return weighted_binary_crossentropy
Then, pass it to the model.compile() like this:
model.compile(loss=wrapper_loss(weights=[1.,2.]), optimizer=...)
P.S: You may need to check these out:
tf.nn.weighted_cross_entropy_with_logits
class_weight argument for model.fit()
I realized how to use those maps. First I define an Input (with the same shape as ground truth labels) as the way we define Input when feeding the input images. Something like
weights = Input(shape=(shape_of_groundtruth_labels))
I define the customized loss with the same structure as wrapper_loss defined above; with weight maps this time, not class weights [1, 2]. Then, when defining the model which needs input and output. I give the input as both input images and input weights. something like:
model = Model(inputs=[images, weights], outputs=...)
where weights are the one I defined in the Input layer. In the model.compile(), I give the loss as the name of my customized loss (wrapper_loss) with the inputs weights. something like
model.compile(optimizer=..., loss=wrapper_loss(weight = weights), ...)
where the second 'weights' is the one defined in Input layer.
Now, the last thing to do is to do the same in model.fit; I give the weight maps with the images with the same structure as above.
underneeth you can find my custom loss function.
def custom_loss_function(y_true, y_pred):
y_pred_bool = tf.math.less_equal(y_pred, tf.constant(0.5))
y_pred_float = 1 - tf.cast(y_pred_bool, dtype=tf.int32)
y_true = tf.cast(y_true, dtype=tf.int32)
mask_bool_loss = tf.math.less(y_true, tf.constant(0))
mask_loss = 1 - tf.cast(mask_bool_loss, dtype=tf.int32)
mask = tf.math.reduce_min(mask_loss, axis=2)
y_multiply = tf.math.multiply(y_true, y_pred_float)
y = tf.math.reduce_sum(y_multiply, axis=2)
y_loss = 1 - y
y_loss = tf.math.multiply(y_loss, mask)
return y_loss
I know some functions of tensorflow are not differentiable, but I really don't know which functions or how to get around it? Any suggestions for me?
I get this error:
ValueError: No gradients provided for any variable: ['bidirectional_7/forward_lstm_7/lstm_cell_22/kernel:0'/, ...
As soon as you cast your variables to int or bool, all gradient information is lost. So the gradients are broken in this first line.
y_pred_bool = tf.math.less_equal(y_pred, tf.constant(0.5))
This is the reason why we usually use things like the binary cross-entropy, as it gives us a differentiable approximation to the non-differentiable 0-1 loss.
I'm trying to write a custom loss function in Keras for a CNN I'm working on. Y_true and Y_pred will both be tensors of grayscale images, so I expect a shape of [a, x, y, 1], where x and y are the dimensions of my images and a is the batch size.
The plan is to:
Threshold each image of Y_true by its mean pixel intensity
Use the non-zero elements of this mask to get an array of pixel values from Y_true and Y_pred
Measure the cosine similarity (using the built-in Keras loss function) of these arrays and return the average result of the batch as the loss
My main question is how I can efficiently implement this process?
Does the cosine_similarity function work on 1D arrays?
I know that I should avoid for loops to maintain efficiency but it's the only way I can think of implementing this function. Is there a more efficient way to implement this function using the Keras backend or numpy?
EDIT
Basic implementation and an unexpected error when compiling the model with this function:
def masked_cosine_similarity(y_true, y_pred):
loss = 0
for i in range(y_true.shape[0]):
true_y = y_true[i,:,:,0]
pred_y = y_pred[i,:,:,0]
mask = true_y > np.mean(true_y)
elements = np.nonzero(mask)
true_vals = np.array([true_y[x,y] for x, y in zip(elements[0], elements[1])])
pred_vals = np.array([pred_y[x,y] for x, y in zip(elements[0], elements[1])])
loss += cosine_similarity(true_vals, pred_vals)
return loss / y_true.shape[0]
Error message:
64 loss = 0
---> 65 for i in range(y_true.shape[0]):
66 true_y = y_true[i,:,:,0]
67 pred_y = y_pred[i,:,:,0]
TypeError: 'NoneType' object cannot be interpreted as an integer
The shape of a tensor in Keras/TF is usually [None, height, width, channels].
This is due to the support of an arbitrary batch size, you don't want to build a model that only works for a specific batch size. For that reason, your code collapses on:
for i in range(y_true.shape[0]):
since y_true.shape[0] == None.
Why do you loop over the batch? You don't need to do it.
For example, given some element-wize loss function (MSE/cosine loss etc.) you can do something like:
def my_loss(y_true, y_pred):
mask = tf.keras.backend.cast(y_true >= tf.math.reduce_mean(y_true, axis=[1,2], keepdims=True), 'float32')
masked_loss = K.sum(mask * elementwize_loss(y_true, y_pred), axis=-1)
num_valid_pixels = K.maximum(1.0, K.cast(K.sum(mask), 'float32'))
return masked_loss / num_valid_pixels
I am using TensorFlow 2. I am trying to optimize a function which uses the loss of a trained tensorflow model (poison).
#tf.function
def totalloss(x):
xt = tf.multiply(x, (1.0 - m)) + tf.multiply(m, d)
label = targetlabel*np.ones(xt.shape[0])
loss1 = poison.evaluate(xt, label, steps=1)
loss2 = tf.linalg.norm(m, 1)
return loss1 + loss2
I am not able to execute this function, however, when I comment the #tf.function line the function works!
I need to use this function as a tensorflow op so as to optimize 'm' & 'd'.
Value Error: Unknown graph. Aborting.
This is how I am defining the model and variables:
# mask
m = tf.Variable(tf.zeros(shape=(1, 784)), name="m")
d = tf.Variable(tf.zeros(shape=(1, 784)), name="d")
# target
targetlabel = 6
poison = fcn()
poison.load_weights("MNISTP.h5")
adam = tf.keras.optimizers.Adam(lr=.002, decay=1e-6)
poison.compile(optimizer=adam, loss=tf.losses.sparse_categorical_crossentropy)
This is how I am calling the function later: (Executing this line results in an error listed below. However if I comment off the #tf.function line, this command works!)
loss = totalloss(ptestdata)
This is the entire traceback call:
ValueError: in converted code:
<ipython-input-52-4841ad87022f>:5 totalloss *
loss1 = poison.evaluate(xt, label, steps=1)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:746 evaluate
use_multiprocessing=use_multiprocessing)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py:693 evaluate
callbacks=callbacks)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py:187 model_iteration
f = _make_execution_function(model, mode)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py:555 _make_execution_function
return model._make_execution_function(mode)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:2034 _make_execution_function
self._make_test_function()
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:2010 _make_test_function
**self._function_kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:3544 function
return EagerExecutionFunction(inputs, outputs, updates=updates, name=name)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:3429 __init__
raise ValueError('Unknown graph. Aborting.')
ValueError: Unknown graph. Aborting.
The purpose of #tf.function decorator is to convert Tensorflow operations written in Python into Tensorflow graph to achieve better performance. The error might come when you tried to use a pre-trained model with a serialized graph. Thus, the decorator cannot make the graph-to-graph conversion.
I've reported this error here: https://github.com/tensorflow/tensorflow/issues/33997
A (temporary) solution is that your loss function should be separated into two small functions. The decorator should only be used in the function not including the pre-trained model. In this way, you still can achieve better performance in other operations but not with the part of using the pre-trained model.
For example:
#tf.function
def _other_ops(x):
xt = tf.multiply(x, (1.0 - m)) + tf.multiply(m, d)
label = targetlabel * np.ones(xt.shape[0])
loss2 = tf.linalg.norm(m, 1)
return xt, label, loss2
def total_loss(x):
xt, label, loss2 = _other_ops(x)
loss1 = poison.evaluate(xt, label, steps=1)
return loss1 + loss2
Update:
According to the discussion in the above TF issue link, an elegant solution is to manually pass the input through each layer of the model. You could get a list of layers in your model by calling your_model.layers
In your case, you might calculate the loss from the prediction of your output with the label in the last layer. Thus, I think you should skip the last layer and calculate the loss outside of the loop:
#tf.function
def totalloss(x):
xt = tf.multiply(x, (1.0 - m)) + tf.multiply(m, d)
label = targetlabel*np.ones(xt.shape[0])
feat = xt
# Skip the last layer which calculates loss1
for i in range(len(poison.layers) - 1):
layer = poison.layers[i]
feat = layer(feat)
# Now, calculate loss by yourself
loss1 = tf.keras.losses.sparse_categorical_crossentropy(feat, label)
loss2 = tf.linalg.norm(m, 1)
return loss1 + loss2
The way that the TF engineers explain for this issue is that a model might wrap high-level processing which does guarantee by the #tf.function. So, putting a model inside a function decorated with #tf.function is not recommended. Thus, we need to break the model to smaller pieces to bypass it.
I need a custom weighted MSE loss function. I defined it in keras.backend
from keras import backend as K
def weighted_loss(y_true, y_pred):
return K.mean( K.square(y_pred - y_true) *
K.exp(-K.log(1.7) * (K.log(1. + K.exp((y_true - 3)/5 ))))
,axis=-1 )
However, a test run returns
weighted_loss(1,2)
ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("Exp_37:0", shape=(), dtype=float32)'
or
weighted_loss(1.,2.)
ZeroDivisionError: integer division or modulo by zero
I wonder what mistakes am I making here.
Whether you are using Tensorflow or Theano is irrelevant for your question. Google the meaning of 'tensor' if the term confuses you.
Take a look at how Keras own loss function tests have been implemented here:
def test_metrics():
y_a = K.variable(np.random.random((6, 7)))
y_b = K.variable(np.random.random((6, 7)))
for metric in all_metrics:
output = metric(y_a, y_b)
print(metric.__name__)
assert K.eval(output).shape == (6,)
You can't simply feed a float or int into tensor calculations. Note also the use of K.eval to obtain the result you're looking for.
So try something similar with your function:
from keras import backend as K
import numpy as np
y_a = K.variable(np.random.random((6, 7)))
y_b = K.variable(np.random.random((6, 7)))
output = weighted_loss(y_a,y_b)
result = K.eval(output)
There is also no need to define your custom function in keras.backend - what if you decide to update Keras later on?
Instead you could do the following in your own code: define a function that returns your loss function
def weighted_loss(y_true, y_pred):
return K.mean( K.square(y_pred - y_true) * K.exp(-K.log(1.7) * (K.log(1. + K.exp((y_true - 3)/5 )))),axis=-1 )
Then when you want to compile your model with your loss function, you can do:
model.compile(loss = weighted_loss)
In case you want to define a more general loss function, where the weighting depends on some input, you'll need to wrap the function. So for example:
def get_weighted_loss(my_input):
def weighted_loss(y_true, y_pred):
return K.mean( K.square(y_pred - y_true) * K.exp(-K.log(1.7) * (K.log(1. + K.exp((y_true - 3)/my_input )))),axis=-1 )
return weighted_loss
Then when you want to compile your model with your loss function, you can do:
model.compile(loss = get_weighted_loss(5))