I am implementing a face estimation model where I am using a pretrained ResNet50 and added some layers on top. Now I would like to implement a Mean-Variance Loss in Keras but I am very new to Keras and just couldn't figure out how.
This is how the last layers of my model look like (where the predict layer contains a sigmoid):
Last Layers:
Right now I train the model with the following loss:
model.compile(tf.keras.optimizers.Adam(learning_rate=1e-4),loss=tf.keras.losses.MeanSquaredError(),metrics=['mae'])
I know that I just have to replace the MeanSquaredError by my custom function but I don't know how to implement the Mean-Variance-Loss.
I am trying to convert my CNN written with tensorflow layers to use the keras api in tensorflow (I am using the keras api provided by TF 1.x), and am having issue writing a custom loss function, to train the model.
According to this guide, when defining a loss function it expects the arguments (y_true, y_pred)
https://www.tensorflow.org/guide/keras/train_and_evaluate#custom_losses
def basic_loss_function(y_true, y_pred):
return ...
However, in every example I have seen, y_true is somehow directly related to the model (in the simple case it is the output of the network). In my problem, this is not the case. How do implement this if my loss function depends on some training data that is unrelated to the tensors of the model?
To be concrete, here is my problem:
I am trying to learn an image embedding trained on pairs of images. My training data includes image pairs and annotations of matching points between the image pairs (image coordinates). The input feature is only the image pairs, and the network is trained in a siamese configuration.
I am able to implement this successfully with tensorflow layers and train it sucesfully with tensorflow estimators.
My current implementations builds a tf Dataset from a large database of tf Records, where the features is a dictionary containing the images and arrays of matching points. Before I could easily feed these arrays of image coordinates to the loss function, but here it is unclear how to do so.
There is a hack I often use that is to calculate the loss within the model, by means of Lambda layers. (When the loss is independent from the true data, for instance, and the model doesn't really have an output to be compared)
In a functional API model:
def loss_calc(x):
loss_input_1, loss_input_2 = x #arbirtray inputs, you choose
#according to what you gave to the Lambda layer
#here you use some external data that doesn't relate to the samples
externalData = K.constant(external_numpy_data)
#calculate the loss
return the loss
Using the outputs of the model itself (the tensor(s) that are used in your loss)
loss = Lambda(loss_calc)([model_output_1, model_output_2])
Create the model outputting the loss instead of the outputs:
model = Model(inputs, loss)
Create a dummy keras loss function for compilation:
def dummy_loss(y_true, y_pred):
return y_pred #where y_pred is the loss itself, the output of the model above
model.compile(loss = dummy_loss, ....)
Use any dummy array correctly sized regarding number of samples for training, it will be ignored:
model.fit(your_inputs, np.zeros((number_of_samples,)), ...)
Another way of doing it, is using a custom training loop.
This is much more work, though.
Although you're using TF1, you can still turn eager execution on at the very beginning of your code and do stuff like it's done in TF2. (tf.enable_eager_execution())
Follow the tutorial for custom training loops: https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough
Here, you calculate the gradients yourself, of any result regarding whatever you want. This means you don't need to follow Keras standards of training.
Finally, you can use the approach you suggested of model.add_loss.
In this case, you calculate the loss exaclty the same way I did in the first answer. And pass this loss tensor to add_loss.
You can probably compile a model with loss=None then (not sure), because you're going to use other losses, not the standard one.
In this case, your model's output will probably be None too, and you should fit with y=None.
I’m transforming a TensorFlow model to Pytorch. And I’d like to initialize the mean and variance of BatchNorm2d using TensorFlow model.
I’m doing it in this way:
bn.running_mean = torch.nn.Parameter(torch.Tensor(TF_param))
And I get this error:
RuntimeError: the derivative for 'running_mean' is not implemented
But is works for bn.weight and bn.bias. Is there any way to initialize the mean and variance using my pre-trained Tensorflow model? Is there anything like moving_mean_initializer and moving_variance_initializer in Pytorch?
Thanks!
The running mean and variance of a batch norm layer are not nn.Parameters, but rather a buffer of the layer.
I think you can simply assign a torch.tensor, no need to wrap a nn.Parameter around it.
I am using a pretrained model that someone else has created, they have only released the model weights. Currently I am importing the model weights into my graph and calling them by the tensor names. However, this seems incompatible with cleverhans' code that seems to require a model object which has the method predict.
Is there any work around for this which does not require me to rewrite most of the cleverhans attacks because I do not have the model class and predict method?
What you are describing should be possible but may be somewhat intensive on resources because it may recreate the graph several times. Basically, you can implement a CleverHans model class that takes in a graph checkpoint in the init method. The get_logits or fprop method should take an input tensor and load the graph to obtain the corresponding output tensor by performing some graph surgery to replace the checkpoint graph's input tensor with your own tensor: see the input_map argument in `tf.import_graph_de: https://www.tensorflow.org/api_docs/python/tf/graph_util/import_graph_def
I want to use a separate loss function in the DNNClassifier as the data is highly imbalanced i want to use
tf.nn.weighted_cross_entropy_with_logits as the loss function but i guess i need to build a new estimator for it?
Is it possible to change the loss function in the existing pre baked DNNClassifier by tensorflow Estimator API?
You can set the classifier's optimizer and the activation function in the hidden layers, but I don't think you can define a custom loss function.
Since your input data is "highly imbalanced," you can set custom weights by assigning your weights to the constructor's weight_column argument. The documentation is here.