I want to be able to make inferential analysis on my neural network by accessing the weights and making decision based on what I've found. In other words, If I've got my list of weights of a particular neuron in a hidden layer, then I want to be able to manipulate that neuron in any way I want. I want to mess with the Neuron's output. I'm using tensorflow for my neural network.
Related
I have been playing around with neural networks for quite a while now, and recently came across the terms "freezing" & "unfreezing" the layers before training a neural network while reading about transfer learning & am struggling with understanding their usage.
When is one supposed to use freezing/unfreezing?
Which layers are to freezed/unfreezed? For instance, when I import a pre-trained model & train it on my data, is my entire neural-net except the output layer freezed?
How do I determine if I need to unfreeze?
If so how do I determine which layers to unfreeze & train to improve model performance?
I would just add to the other answer that this is most commonly used with CNNs and the amount of layers that you want to freeze (not train) is "given" by the amount of similarity between the task that you are solving and the original one (the one that the original network is solving).
If the tasks are very similar, let's say that you are using CNN pretrained on imagenet and you just want to add some other "general" objects that the network should recognize then you might get away with training just the dense top of the network.
The more dissimilar the tasks are, the more layers of the original network you will need to unfreeze during the training.
By freezing it means that the layer will not be trained. So, its weights will not be changed.
Why do we need to freeze such layers?
Sometimes we want to have deep enough NN, but we don't have enough time to train it. That's why use pretrained models that already have usefull weights. The good practice is to freeze layers from top to bottom. For examle, you can freeze 10 first layers or etc.
For instance, when I import a pre-trained model & train it on my data, is my entire neural-net except the output layer freezed?
- Yes, that's may be a case. But you can also don't freeze a few layers above the last one.
How do I freeze and unfreeze layers?
- In keras if you want to freeze layers use: layer.trainable = False
And to unfreeze: layer.trainable = True
If so how do I determine which layers to unfreeze & train to improve model performance?
- As I said, the good practice is from top to bottom. You should tune the number of frozen layers by yourself. But take into account that the more unfrozen layers you have, the slower is training.
When training a model while transfer layer, we freeze training of certain layers due to multiple reasons, such as they might have already converged or we want to train the newly added layers to an already pre-trained models. This is a really basic concept of Transfer learning and I suggest you go through this article if you have no idea about transfer learning .
What would happen if we do not connect some nodes of the hidden layer to input vector?
We know that The Input layer of a neural network brings the initial data into the system for further processing and the Input layer will pass the input features or data to the 1st hidden layer. If we don't connect some nodes of Input to the 1st layer, it will not get all the data or features of Input for training.
Look at picture 1 & 2. Picture-2 is the Simple schematic of Predicting the Earthquake Magnitude Using the Multilayer Perceptron Neural Network. If we don't connect some of the features of Input to 1st hidden layer, it will not get all features of the Input and can not give the proper result or prediction of the Earthquake Magnitude.
But when we have Large neural nets trained on relatively small datasets, we can do this skip connection in the deeper layer by randomly dropping out nodes during training to get rid of the overfitting problem. and you can see that here Dropout for Regularizing Deep Neural Networks .
I have a large data set of 2D objects which I want to classify. I have set up a 2D convolutional neural network using keras and everything works acceptably.
However, at the moment I do not use all available prior information, specifically each input object can only belong to a small subsets of all output classes. However, this is different for each object, meaning I cannot simply train multiple networks.
Therefore, is there a way to add this information to the neural network, i.e. for each object exclude certain output classes during training and testing?
Is it possible to freeze one LSTM layer and build another LSTM layer on the top of it?
The idea is quite simple but mechanism in Tensorflow makes it harder. All I need to do is to build one LSTM layer and save the model. Then, I restore this model to get kernel (weight) matrix and biases using:
tf.get_default_graph().get_tensor_by_name("rnn/multi_rnn_cell/cell_0/LSTM_cell/kernel:0") tf.get_default_graph().get_tensor_by_name("rnn/multi_rnn_cell/cell_0/LSTM_cell/bias:0")
Then, I want to get these two tensors and keep them untrained and build another LSTM layer on the top of this then fetch the variables of the second layer.
However, the only topic I came up related to what I want to do is here and it says that it is not possible, because I cannot set weights manually.
Doing this is super simple in feed forward neural network.
Does anyone have any idea?
I am using tensorflow to train two instances of the same neural network with two different datasets. the network itself is quite simple with an input and output layer and 6 hidden layers (each layer is a 20 meurons followed by a non-linear activation function).
I can train the network with two different datasets and that is fine. Now, what i want to do is basically create a new network which is a combination of these two trained networks. In particular, I want the input and the first 3 layers to be from one of the trained network and the last 3 layers and the output layer to be from the other network. I am very new to tensorflow and have not found a way to do this. Can someone point me to the API or some way to do this sort of hybrid networks?
Constructing your network with Keras will make this easy; see the keras documentation for how to reuse layers across networks.
You might be asking about multitask learning aspect,well it can be simplified by seperating the weight matrix of each individual variables trained with different datasets and sum there weight layers individually to a sharable_weight_layer variable after a, b trained networks and finally evaluate your model as summed network in multitasking method.