I am just getting into Keras and Tensor flow.
Im having a lot of problems adding an input normalization layer in a sequential model.
Now my model is ;
model = tf.keras.models.Sequential()
model.add(keras.layers.Dense(256, input_shape=(13, ), activation='relu'))
model.add(tf.keras.layers.LayerNormalization(axis=-1 , center=True , scale=True))
model.add(keras.layers.Dense(128, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(1))
model.summary()
My doubts are whether I should first perform an adapt function and how to use it in the sequential model.
Thanks to all!!
I'm trying to figure this out as well. According to this example, adapt is not necessary.
model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tf.keras.layers.LayerNormalization(axis=3 , center=True , scale=True),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)
Also, make sure you want a LayerNormalization. If I understand correctly, that normalizes every input on its own. Batch normalization may be more appropriate. See this for more info.
Related
I have a imbalanced dataset which has 57000 zeros and 2500 ones. I gave class weights as an input to my model, tried to change optimizers, tried to resize number of layers and neurons. Finally I stick to ;
because it was the only one that seems systematic, tried to change layer weight regularization rules but nothing helped me yet. I am not talking about just for my validation AUC score, even train success doesn't rise satisfyingly.
Here is how I declared my model, don't mind if you think the problem is layer and node sizes. I think I tried everything that sounds sensible.
class_weight = {0: 23.59,
1: 1.}
model=Sequential()
model.add(Dense(40, input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(33, activation='relu',kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),bias_regularizer=regularizers.l2(1e-4),activity_regularizer=regularizers.l2(1e-5)))
model.add(Dense(28, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(15, activation='relu'))
model.add(Dense(9, activation='relu'))
model.add(Dense(5, activation='relu'))
model.add(Dense(1, activation='sigmoid',kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),bias_regularizer=regularizers.l2(1e-4),activity_regularizer=regularizers.l2(1e-5)))
opt = keras.optimizers.SGD(learning_rate=0.1)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['AUC'])
model.fit(x_train,y_train,epochs=600,verbose=1,validation_data=(x_test,y_test),class_weight=class_weight)
After approximate 100 epoch, it was stuck at 0.73-0.75 auc, doesn't rise anymore. I couldn't even overfit my model
i have searched through the internet to solve this problem, but no one seems to have a solution to it. I would like to improve my BiLSTM-model with a CRF layer, which is no longer supported by Keras obviously. How can i add this? I dont want to change my whole Keras and Tensorflow version for it. Please help me :) I have the latest Keras and Tensorflow versions.
#vocab_size=4840, embedding is glove6B, max_seq_length=100
model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_vectors], input_length=max_seq_length, trainable= False))
model.add(Bidirectional(LSTM(units=50, dropout=0.5, recurrent_dropout=0.5, return_sequences=True)))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(32, activation='relu'))
model.add(Dense(n_tags, activation='softmax'))
model.compile(loss="categorical_crossentropy", optimizer="rmsprop", metrics= ["categorical_accuracy"])
model.summary()
So I have this assignment to train a very simple neural network. Our dataset has 6 features that are fed into the network and we are required to train it and then predict one output number. The professor gave us the code and basically told us to learn by ourselves lol. So my doubt is, in the following code, in which the layers for the neural network are defined, does the first dense layer defined (the one with 50 nodes) corresponds to the input layer, or is it the first hidden layer?
If it's the first hidden layer, how are input layers defined?
Thanks in advance!
def get_compiled_model():
model = tf.keras.Sequential([
tf.keras.layers.Dense(50, activation='relu', input_shape=(6,)),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(1, activation='linear'),
])
The first dense layer is the first hidden layer. Keras automatically provides an input layer in Sequential objects, and the number of units is defined by input_shape or input_dim.
You can also explicitly state the input layer as follows:
def get_compiled_model():
model = tf.keras.Sequential([
tf.keras.layers.InputLayer((6,)),
tf.keras.layers.Dense(50, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(30, activation='relu'),
tf.keras.layers.Dense(1, activation='linear'),
])
It is the first hidden layer. The input layer isn't defined as a separate layer; it simply consists of the input data, and its size is defined by input_shape=(6,).
I have a trained classifier neural network in Keras. Let the neural network be f(x). I want to find the vectors x such that when ||x||^2 = 1, f(x) is maximized. I currently have trained my neural network with Keras
model = Sequential()
model.add(Dense(500, activation='sigmoid'))
model.add(Dense(500, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', auc])
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=2, verbose = 1, callbacks=[earlyStopping])
I want to know if there is a way to solve this constrained optimization problem once my Neural network has already been trained. There is a scipy optimize which can do this for general functions. Is there a way to do this for a neural network. Please include a code sample.
If I understand you correctly you have finished training your neural network and would like to find the input x which maximizes the probability of it being in a certain class (where the output is close to 1.0)
You could write a small function to assess the performance of your network using the predict_proba() method to get classification probabilities on test data, and then optimise this function using scipy:
model = Sequential()
model.add(Dense(500, activation='sigmoid'))
model.add(Dense(500, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', auc])
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=2, verbose = 1, callbacks=[earlyStopping])
def f(x):
prediction = model.predict_proba(x)
return -prediction
a = scipy.optimize.minimize(f, x0=np.random.randn(500))
optimal_x = a.x
optimal_x will be the input x which maximises the certainty with which your classifier puts it in one specific class.
I'm trying to define a fully connected neural network in keras using tensorflow backend, I have a sample code but I dont know what it means.
model = Sequential()
model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(50, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(20, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(10, input_dim=x.shape[1], kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
model.add(Dense(y.shape[1],activation='softmax'))
From the above code I want to know what is the number of inputs to my network, number of outputs, number of hidden layers and number of neurons in each layer. And what is the number coming after model.add(Dense ? assuming x.shape[1]=60.
What is the name of this network exacly? Should I call it a fully connected network or convolutional network?
That should be quite easy.
For knowing about the model's inputs and outputs use,
input_tensor = model.input
output_tensor = model.output
You can print these tf.Tensor objects to get the shape and dtype.
For fetching the Layers of a model use,
layers = model.layers
print( layers[0].units )
With these tricks you can easily get the input and output tensors for a model or its layer.