I would like to perform a convolution with a complex input and complex kernel.
I do not need back propagation, so I don't care whether gradients can be calculated or not.
Is this possible in Tensorflow?
Related
I have a TensorFlow model where I can apply the pruner.prune_low_magnitude layer to the output of my Dense layers. This seems to work according to the instructions, and I get almost the same results down to 95% sparsity. The Processing time in GPU and CPU seems to be the same. It seems the pruning layer is calculating all the weights, then setting the smallest weights to zero to fit the sparsity figure. Is this right?
It would be nice to get a speed-up, of course, but for my present purposes, this is fine.
I would like to prune the weights and have them stay zero thereafter. I would prune some weights; then continue training to allow the model to recover from the pruning; then prune a bit more. I feel this should be a bit closer to what real neurones do. Is there some way of doing this?
My solution (which does not work yet) is to add a custom layer with Trainable=false. This has a mask array that starts off as all ones, and is set to zero if the corresponding weight is zero. The layer multiplies the weights by this mask array, so once a weight goes to zero, it will stay zero. Should this work? Is there a better way?
It would be nice to get a speed-up, of course, but for my present
purposes, this is fine.
To get a reduction in inference time, the op (Dense in your example) implementation needs to be able to take advantage of the sparsity and the particular hardware on which it runs.
The Tensorflow runtime on CPU/GPU does not support this yet, but TFLite does. The
Pruning for on-device inference w/ XNNPACK tutorial demonstrates that.
I would like to prune the weights and have them stay zero thereafter.
I would prune some weights; then continue training to allow the model
to recover from the pruning; then prune a bit more. I feel this should
be a bit closer to what real neurones do. Is there some way of doing
this?
This is the responsibility of the pruning schedule that is passed as argument to prune_low_magnitude().
Existing implementations, such as ConstantSparsity or PolynomialDecay perform what you described: prune only at some steps and let the model recover in between. The begin_step, end_step, and frequency arguments let you control when and how frequently the pruning is applied during training.
My solution (which does not work yet) is to add a custom layer with
Trainable=false. This has a mask array that starts off as all ones,
and is set to zero if the corresponding weight is zero. The layer
multiplies the weights by this mask array, so once a weight goes to
zero, it will stay zero. Should this work? Is there a better way?
This is basically how Tensorflow Model Optimization does it under the hood: pruning_impl.py. You just need to apply prune_low_magnitude() as per the Pruning with Keras tutorial.
Lately I've been reading up on the memory consumed by convolutional neural networks (ConvNets). During training, each convolutional layer has several parameters required for back-propagation of the gradients. These lecture notes suggest that these parameters can in principle be discarded at test time. A quote from the linked notes:
Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.
Is there any way (using TensorFlow) to make use of this "clever implementation" for inference of large batches? Is there some flag that specifies whether or not the model is in its training phase? Or is this already handled automatically based on whether the optimiser function is called?
I'd like to apply stemming to my training data set. I can do this outside of tensorflow as part of training data prep, but I then need to do the same process on prediction request data before calling the (stored) model.
Is there a way of implementing this transformation in tensorflow itself so the transformation is used for both training and predictions?
This problem becomes more annoying if the transformation requires knowledge of the whole dataset, normalisation for example.
Can you easily express your processing (e.g. stemming) as a tensorflow operation? If yes, then you can build your graph in a way that both your inputs and predictions can make use of the same set of operations. Otherwise, there isn't much harm in calling the same (non tensorflow) function for both pre-processing and for predictions.
Re normalisation: you would find the dataset statistics (means, variance, etc. depending on how exactly you are normalizing) and then hardcode them for the pre/post-processing so I don't think that's really an annoying case.
In Tensorflow, you can either perform either classification or linear regression to train your inputs against the labels. Is it possible to perform some classification for your inputs (as pre-processing but not necessarily to use Tensorflow) and determine if you want to run the linear regression using Tensorflow?
For example in image denoising task, you have found that your linear regression algorithm can provide a good smoothing effect against the edges but in the meantime also remove the details for the texture objects. Therefore you would like to perform a binary classification to determine if an input is a texture object, and run the linear regression algorithm using Tensorflow; otherwise do nothing for texture object.
I understand Tensorflow supports transfer learning so I guess one of the possible solutions is to perform binary classification using Tensorflow, and transfer the "texture classification" knowledge to instruct Tensorflow to apply linear regression algorithm only when the input is a texture object? Please correct me if I am wrong as I am not too sure if the above task is do-able in Tensorflow (it would be great if you can describe how to do this in details if this is do-able :-) ).
I guess an alternative solution is to use some binary classification without Tensorflow, and filter out (remove) the texture inputs before passing them to Tensorflow.
Please kindly tell me if which of the above solution (or any other solution) is better (if do-able) for the above scenario? Any suggestions are welcome.
I know tensorflow has provided some approaches to deal with sparse tensor. For a example, the tf.sparse_tensor_dense_matmul is faster than the tf.matmul when there is a sparse matrix.
In a deep convolution network, I get sparse convolution kernels after a training process. I want to know how to save the convolution kernels so that the tensorflow knows the kernels are sparse?
I have read some papers. The papers propose the sparse convolution will bring more efficient calculation than traditional convolution. But the tf.nn.conv2d doesn't indicate that it will calculate with a sparse convolution kernel faster than a dense convolution kernel. How do I obtain the advantages from sparse kernels?
Yes, tf.nn.conv2d does not work with sparse kernel. If you think that sparse convolution will bring you the benefits of speed and feel comfortable writing efficient cpu/gpu code, you can write your own op, the way it is described in their docs