Hi i am trying to train the keras model after split. simulating the keras model are placed on different devices and each device have their weights. that devices passes their intermediate output to host and host device receive intermediate output and send back gradient.How can we train the splited model here is the example of splited model for mnist dataset
model1 = keras.Sequential(
[
keras.Input(shape=input_shape),
layers.Conv2D(32, kernel_size=2, activation="relu",strides=1, padding="same"),
layers.MaxPooling2D(pool_size=2),
]
)
model2 = keras.Sequential(
[
layers.Conv2D(64, kernel_size=2, activation="relu", strides=1, padding="same"),
layers.MaxPooling2D(pool_size=2),
layers.Flatten(),
layers.Dropout(0.5),
layers.Dense(num_classes, activation="softmax"),
]
)
I am expecting the way to train these models on different devices where guest device will have images and model1 and send the intermediate output to host after predict with moodel1.
Host contain model2 and label for images. host 2 will receive intermediate output and host will train model2 and send the gradient to guest which will train the model1.
Related
I am following the official distributed training documentation but find that only 1 server is used in databricks ganglia
https://www.tensorflow.org/tutorials/distribute/keras
strategy = tf.distribute.MirroredStrategy()
print('Number of devices: {}'.format(strategy.num_replicas_in_sync))
outputs 2
why is only one server in used ? When there is multiple (2) server available?
with strategy.scope():
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10)
])
model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(),
metrics=['accuracy'])
Is there a way I can force the number of server to split work in training?
Given a pre-trained well-performing auto-encoder. When I train a classifier on encodings (produced by the auto-encoder) the classifier does very poorly. In particular, it does much worse than training a classifier on normal inputs (i.e. unencoded inputs).
However, when I fine-tune the encoder based on classification loss, the classifier does quite well.
Why are encoded representations bad for classification?
Details: I’m working on CIFAR-100 and trying to classify coarse image labels, i.e. 20 classes (but I think I had the same problem when doing classification on CIFAR-10). The classifier has 5 layers and I’m using dropout:
classifier = tf.keras.Sequential([
tf.keras.layers.Dense(512,
activation='relu',
name='classifier_hidden_1'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(256,
activation='relu',
name='classifier_hidden_2'),
tf.keras.layers.Dense(128,
activation='relu',
name='classifier_hidden_3'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(64,
activation='relu',
name='classifier_hidden_4'),
tf.keras.layers.Dense(num_classes,
activation=None,
name='classifier_out'),
], name='classifier')
I am starting off my deep learning journey using the IMDB movie review dataset. I am not sure how the training data is loaded and the input_shape is specified.
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)
If I understand it correctly, there are 25000 reviews from different movies and the words are encoded as list of integers.
train_data.shape
(25000,)
Since each review has different lengths, how is it possible to store such data in a matrix (i.e., 25,000 rows but columns with different lengths)?
model = models.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
model.add(layers.Dense(16, activation='relu', input_shape=(16,)))
model.add(layers.Dense(1, activation='sigmoid', input_shape=(16,)))
After reshaping the input to (25000,10000) using one-hot encoding, why is the first argument to input_shape not 25000 (number of samples) rather than 10000 since the Dense layer will compute the outputs according to relu(dot(w, inputs) + b)? ==> (25000, 10000) dot (10000, 16) = (25000, 16) Why don't we specify the input_shape as [None, 10000] as in Tensorflow core?
You can pad using keras pad_sequence
data = pad_sequences(data, maxlen=max_length, padding='post')
I recommend you to use an Embedding layer also
model = Sequential()
model.add(Embedding(num_words, 32, input_length=maxlen))
So I have this assignment to train a very simple neural network. Our dataset has 6 features that are fed into the network and we are required to train it and then predict one output number. The professor gave us the code and basically told us to learn by ourselves lol. So my doubt is, in the following code, in which the layers for the neural network are defined, does the first dense layer defined (the one with 50 nodes) corresponds to the input layer, or is it the first hidden layer?
If it's the first hidden layer, how are input layers defined?
Thanks in advance!
def get_compiled_model():
model = tf.keras.Sequential([
tf.keras.layers.Dense(50, activation='relu', input_shape=(6,)),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(1, activation='linear'),
])
The first dense layer is the first hidden layer. Keras automatically provides an input layer in Sequential objects, and the number of units is defined by input_shape or input_dim.
You can also explicitly state the input layer as follows:
def get_compiled_model():
model = tf.keras.Sequential([
tf.keras.layers.InputLayer((6,)),
tf.keras.layers.Dense(50, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(1, activation='linear'),
])
It is the first hidden layer. The input layer isn't defined as a separate layer; it simply consists of the input data, and its size is defined by input_shape=(6,).
I'm trying to define a fully connected neural network in keras using tensorflow backend, I have a sample code but I dont know what it means.
model = Sequential()
model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(50, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(20, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
model.add(Dense(y.shape[1],activation='softmax'))
From the above code I want to know what is the number of inputs to my network, number of outputs, number of hidden layers and number of neurons in each layer. And what is the number coming after model.add(Dense ? assuming x.shape[1]=60.
What is the name of this network exacly? Should I call it a fully connected network or convolutional network?
That should be quite easy.
For knowing about the model's inputs and outputs use,
input_tensor = model.input
output_tensor = model.output
You can print these tf.Tensor objects to get the shape and dtype.
For fetching the Layers of a model use,
layers = model.layers
print( layers[0].units )
With these tricks you can easily get the input and output tensors for a model or its layer.