I have read that convolution networks scale very well across multiple gpus, but what about deep neural networks that don't use convolutions? The Tensorflow website provides a multiple gpu example, but it uses convolutions.
Related
Suppose we have a simple TensorFlow model with a few convolutional layers. We like to train this model on a cluster of computers that is not equipped with GPUs. Each computational node of this cluster might have 1 or multiple cores. Is it possible out-of-the-box?
If not, which packages are able to do that? Are those packages able to perform data and model parallelism?
According to Tensorflow documentation
tf.distribute.Strategy is a TensorFlow API to distribute training across multiple GPUs, multiple machines or TPUs.
As mentioned above it supports CPU for distributed training by considering all devices should be in the same network.
Yes, you can use multiple devices for training model and need to have cluster and worker configuration to be done on couple of devices as shown below.
tf_config = {
'cluster': {
'worker': ['localhost:1234', 'localhost:6789']
},
'task': {'type': 'worker', 'index': 0}
}
To know about the configuration and training model, please refer Multi-worker training with Keras.
According to this SO answer
tf.distribute.Strategy is integrated to tf.keras, so when model.fit is
used with tf.distribute.Strategy instance and then using
strategy.scope() for your model allows to create distributed
variables. This allows it to equally divide your input data on your
devices.
Note: One can be benefited using distributed training when dealing with huge data and complex models (i.e. w.r.t performance).
I'm using the Tensorflow (using the Keras API) in Python 3.0. I'm using the VGG19 pre-trained network to perform style transfer on an Nvidia RTX 2070.
The largest input image that I have is 4500x4500 pixels (I have removed the fully-connected layers in the VGG19 to allow for a fully-convolutional network that handles arbitrary image sizes.) If it helps, my batch size is just 1 image at a time currently.
1.) Is there an option for parallelizing the evaluation of the model on the image input given that I am not training the model, but just passing data through the pre-trained model?
2.) Is there any increase in capacity for handling larger images in going from 1 GPU to 2 GPUs? Is there a way for the memory to be shared across the GPUs?
I'm unsure if larger images make my GPU compute-bound or memory-bound. I'm speculating that it's a compute issue, which is what started my search for parallel CNN evaluation discussions. I've seen some papers on tiling methods that seem to allow for larger images
I have taken a few deeplearning.ai courses, all of which focus on fully-connected network topologies which are all interconnected while neglecting sparsely connected ones.
I am wondering if I can use Tensorflow to calculate both the forward and backward propagation of a sparse neural network, like the following:
I'm assuming traditional matrix operations won't work for this type of topology, since not all nodes are connected equally.
Is the TensorFlow designed only for implementing neural networks? Can it be used as a general machine learning library -- for implementing all sorts of supervised as well as unsupervised techniques (naive baysian, decision trees, k-means, SVM to name a few) ? Whatever TensorFlow literature I am coming across is generally talking about neural networks. Probably graph based architecture of TensorFlow makes it suitable candidate for neural nets. But can it be also used as a general Machine Learning framework?
Tensorflow does include additional machine learning algorithms such as:
K-means clustering
Random Forests
Support Vector Machines
Gaussian Mixture Model clustering
Linear/logistic regression
The above list is taken from here, so you can read this link for more details.
I am using tensorflow to train two instances of the same neural network with two different datasets. the network itself is quite simple with an input and output layer and 6 hidden layers (each layer is a 20 meurons followed by a non-linear activation function).
I can train the network with two different datasets and that is fine. Now, what i want to do is basically create a new network which is a combination of these two trained networks. In particular, I want the input and the first 3 layers to be from one of the trained network and the last 3 layers and the output layer to be from the other network. I am very new to tensorflow and have not found a way to do this. Can someone point me to the API or some way to do this sort of hybrid networks?
Constructing your network with Keras will make this easy; see the keras documentation for how to reuse layers across networks.
You might be asking about multitask learning aspect,well it can be simplified by seperating the weight matrix of each individual variables trained with different datasets and sum there weight layers individually to a sharable_weight_layer variable after a, b trained networks and finally evaluate your model as summed network in multitasking method.