i have searched through the internet to solve this problem, but no one seems to have a solution to it. I would like to improve my BiLSTM-model with a CRF layer, which is no longer supported by Keras obviously. How can i add this? I dont want to change my whole Keras and Tensorflow version for it. Please help me :) I have the latest Keras and Tensorflow versions.
#vocab_size=4840, embedding is glove6B, max_seq_length=100
model = Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_vectors], input_length=max_seq_length, trainable= False))
model.add(Bidirectional(LSTM(units=50, dropout=0.5, recurrent_dropout=0.5, return_sequences=True)))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(32, activation='relu'))
model.add(Dense(n_tags, activation='softmax'))
model.compile(loss="categorical_crossentropy", optimizer="rmsprop", metrics= ["categorical_accuracy"])
model.summary()
Related
I am just getting into Keras and Tensor flow.
Im having a lot of problems adding an input normalization layer in a sequential model.
Now my model is ;
model = tf.keras.models.Sequential()
model.add(keras.layers.Dense(256, input_shape=(13, ), activation='relu'))
model.add(tf.keras.layers.LayerNormalization(axis=-1 , center=True , scale=True))
model.add(keras.layers.Dense(128, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(1))
model.summary()
My doubts are whether I should first perform an adapt function and how to use it in the sequential model.
Thanks to all!!
I'm trying to figure this out as well. According to this example, adapt is not necessary.
model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tf.keras.layers.LayerNormalization(axis=3 , center=True , scale=True),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)
Also, make sure you want a LayerNormalization. If I understand correctly, that normalizes every input on its own. Batch normalization may be more appropriate. See this for more info.
I have a imbalanced dataset which has 57000 zeros and 2500 ones. I gave class weights as an input to my model, tried to change optimizers, tried to resize number of layers and neurons. Finally I stick to ;
because it was the only one that seems systematic, tried to change layer weight regularization rules but nothing helped me yet. I am not talking about just for my validation AUC score, even train success doesn't rise satisfyingly.
Here is how I declared my model, don't mind if you think the problem is layer and node sizes. I think I tried everything that sounds sensible.
class_weight = {0: 23.59,
1: 1.}
model=Sequential()
model.add(Dense(40, input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(33, activation='relu',kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),bias_regularizer=regularizers.l2(1e-4),activity_regularizer=regularizers.l2(1e-5)))
model.add(Dense(28, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(15, activation='relu'))
model.add(Dense(9, activation='relu'))
model.add(Dense(5, activation='relu'))
model.add(Dense(1, activation='sigmoid',kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),bias_regularizer=regularizers.l2(1e-4),activity_regularizer=regularizers.l2(1e-5)))
opt = keras.optimizers.SGD(learning_rate=0.1)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['AUC'])
model.fit(x_train,y_train,epochs=600,verbose=1,validation_data=(x_test,y_test),class_weight=class_weight)
After approximate 100 epoch, it was stuck at 0.73-0.75 auc, doesn't rise anymore. I couldn't even overfit my model
I have this cnn model:
model = Sequential()
model.add(Convolution2D(32, (3, 3), activation='relu', input_shape=(n_rows,n_cols,1)))
model.add(Convolution2D(32, (3, 3), activation='relu'))
model.add(AveragePooling2D(pool_size=(1,3)))
model.add(Flatten())
model.add(Dense(1024, activation='relu')) #needed?
model.add(Dense(3)) #default linear activation
I can train it and obtain related weights.
After I want to load the weights up to Flatten (the dense part is not useful for the second stage) and pass the Flatten to an LSTM.
Of course, it is also suggested to use TimeDistributed on the CNN net.
How to do all this: load weights, take only CNN part, TimeDistribute it, and finally add LSTM?
Thanks!
You can use model.save_weights("filename.h5") to save the weights, and model.load_weights("filename.h5") to load them back into the model.
Source: https://keras.io/getting-started/faq/#savingloading-only-a-models-weights
Is there a way to use the native tf Attention layer with keras Sequential API?
I'm looking to use this particular class. I have found custom implementations such as this one. What I'm truly looking for is the use of this particular class with the Sequential API
Here's a code example of what I'm looking for
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Embedding(vocab_length,
EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH,
weights=[embedding_matrix], trainable=False))
model.add(tf.keras.layers.Dropout(0.3))
model.add(tf.keras.layers.Conv1D(64, 5, activation='relu'))
model.add(tf.keras.layers.MaxPooling1D(pool_size=4))
model.add(tf.keras.layers.CuDNNLSTM(100))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Attention()) # Doesn't work this way
model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
I ended up using a custom class I found on this repository by tsterbak. It's the AttentionWeightedAverage class. It is compatible with the Sequential API
Here's my model for reference :
model = Sequential()
model.add(Embedding(input_dim=vocab_length,
output_dim=EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH,
weights=[embedding_matrix], trainable=False))
model.add(Conv1D(64, 5, activation='relu'))
model.add(MaxPooling1D(pool_size=4))
model.add(Bidirectional(GRU(100, return_sequences=True)))
model.add(AttentionWeightedAverage())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
Note that it's what is called "soft attention" or "attention with weighted average", as described in "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention". The details are more understandable here
I have a trained classifier neural network in Keras. Let the neural network be f(x). I want to find the vectors x such that when ||x||^2 = 1, f(x) is maximized. I currently have trained my neural network with Keras
model = Sequential()
model.add(Dense(500, activation='sigmoid'))
model.add(Dense(500, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', auc])
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=2, verbose = 1, callbacks=[earlyStopping])
I want to know if there is a way to solve this constrained optimization problem once my Neural network has already been trained. There is a scipy optimize which can do this for general functions. Is there a way to do this for a neural network. Please include a code sample.
If I understand you correctly you have finished training your neural network and would like to find the input x which maximizes the probability of it being in a certain class (where the output is close to 1.0)
You could write a small function to assess the performance of your network using the predict_proba() method to get classification probabilities on test data, and then optimise this function using scipy:
model = Sequential()
model.add(Dense(500, activation='sigmoid'))
model.add(Dense(500, activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy', auc])
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=2, verbose = 1, callbacks=[earlyStopping])
def f(x):
prediction = model.predict_proba(x)
return -prediction
a = scipy.optimize.minimize(f, x0=np.random.randn(500))
optimal_x = a.x
optimal_x will be the input x which maximises the certainty with which your classifier puts it in one specific class.