I am trying to change the activation function in the LSTM cell from the new 1.0 release of Tensorflow but am having difficulty.
There is tf.contrib.rnn.LSTMcell which the API states should allow for changing functions but it does not seem to be implemented yet for this cell.
Furthermore, tf.contrib.rnn.BasicLSTMCell, which also should allow for different activation functions doesn't seem to exist anymore.
Do I just need to wait or is there another solution?
When you instantiate both tf.contrib.rnn.LSTMcell and tf.contrib.rnn.BasicLSTMCell you can pass the activation function as the activation parameter. If you look at the linked documentation, you'll see, for example, that the constructor's signature for BasicLSTMCell is
__init__(num_units, forget_bias=1.0, input_size=None, state_is_tuple=True, activation=tf.tanh)
Related
I am new to PyTorch but have a lot of experience with TensorFlow.
I would like to modify the gradient of just a tiny piece of the graph: just the derivative of activation function of a single layer. This can be easily done in Tensorflow using tf.custom_gradient, which allows you to supply customized gradient for any functions.
I would like to do the same thing in PyTorch and I know that you can modify the backward() method, but that requires you to rewrite the derivative for the whole network defined in the forward() method, when I would just like to modify the gradient of a tiny piece of the graph. Is there something like tf.custom_gradient() in PyTorch? Thanks!
You can do this in two ways:
1. Modifying the backward() function:
As you already said in your question, pytorch also allows you to provide a custom backward implementation. However, in contrast to what you wrote, you do not need to re-write the backward() of the entire model - only the backward() of the specific layer you want to change.
Here's a simple and nice tutorial that shows how this can be done.
For example, here is a custom clip activation that instead of killing the gradients outside the [0, 1] domain, simply passes the gradients as-is:
class MyClip(torch.autograd.Function):
#staticmethod
def forward(ctx, x):
return torch.clip(x, 0., 1.)
#staticmethod
def backward(ctx, grad):
return grad
Now you can use MyClip layer wherever you like in your model and you do not need to worry about the overall backward function.
2. Using a backward hook
pytorch allows you to attach hooks to different layer (=sub nn.Modules) of your network. You can register_full_backward_hook to your layer. That hook function can modify the gradients:
The hook should not modify its arguments, but it can optionally return a new gradient with respect to the input that will be used in place of grad_input in subsequent computations.
I want to use a separate loss function in the DNNClassifier as the data is highly imbalanced i want to use
tf.nn.weighted_cross_entropy_with_logits as the loss function but i guess i need to build a new estimator for it?
Is it possible to change the loss function in the existing pre baked DNNClassifier by tensorflow Estimator API?
You can set the classifier's optimizer and the activation function in the hidden layers, but I don't think you can define a custom loss function.
Since your input data is "highly imbalanced," you can set custom weights by assigning your weights to the constructor's weight_column argument. The documentation is here.
When using the GANEstimator in the tensorflow library, I can only pass the mode to the generator function but not the discriminator function. This means I can not perform batch normalisation or dropout in the discriminator. Is there a reason why this is not allowed? Will this be changed in the future?
Link to GANEstimator:
https://www.tensorflow.org/versions/r1.6/api_docs/python/tf/contrib/gan/estimator/GANEstimator
This feature has been added already:
https://github.com/tensorflow/tensorflow/commit/9560054b18ad3fc9faa296d344e062876858817a
I am stuck with the nce_loss activation function in the word2vec model. I want to figure out what activation function it uses among all these listed here:
These include smooth nonlinearities (sigmoid, tanh, elu, softplus,
and softsign), continuous but not everywhere differentiable functions
(relu, relu6, crelu and relu_x), and random regularization (dropout).
I have searched for it in this function and somewhere else but failed to get any ideas.
I suppose it is the relu* series. Any hints please?
None of those. It uses CrossEntropy.
I couldn't find a way to change activation function in DNNClassifier. The document is not well written. I want to do something like:
classifier = learn.DNNClassifier(hidden_units=[8,16,8], n_classes=2, activation_fn=relu)
But there is no activation_fn in the fucntion, so I can hardly change it.
Can anyone help? Thanks,
So there are a bunch of different activation functions out there. The dictionary below just gives you the more common ones. You can find out about all activation function here: https://www.tensorflow.org/versions/r0.11/api_docs/python/nn.html
activation = {'relu': tf.nn.relu,
'tanh': tf.nn.tanh,
'sigmoid': tf.nn.sigmoid,
'elu': tf.nn.elu,
'softplus': tf.nn.softplus,
'softsign': tf.nn.softsign,
'relu6': tf.nn.relu6,
'dropout': tf.nn.dropout}