I have this error when I add TimeDistributed function to LSTM:
AttributeError: Exception encountered when calling layer 'lstm' (type LSTM). 'LSTMCell' object has no attribute 'kernel'
for this code :
model = Sequential()
# define CNN model
model.add(TimeDistributed(LSTM(64, return_sequences=True, input_shape=(80, 1))))
# model.add(TimeDistributed(LSTM(128, input_shape=(80,1), return_sequences=True)))
# define LSTM model
# model.add(LSTM(64, return_sequences=True))
model.add(LSTM(64))
model.add(TimeDistributed(Dense(100, activation='softmax')))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print('Train...')
historyCNN = model.fit(traininp, trainout, epochs=100 ,validation_data=(validinp, validout))
I upgraded keras and tensorflow, I got sure that I m using correct input and output data.
But I have no idea what can the problem be
LSTM layer is already time distributed layer, you don't need to add TimeDistributed to it. Please try just:
model.add(LSTM(64, return_sequences=True, input_shape=(80, 1)))
Related
I load the model in this way:
model = EfficientNetB7(weights='imagenet', include_top=False, input_shape=input_shape)
What i am trying to do is remove the layer in position:
model.layers[1] #Rescaling
model.layers[2] #Normalization
What i tried is:
del_res= Dense(1, activation='relu', name='remove_rescaling')(base_model.layers[1].output)
del_nor= Dense(1, activation='relu', name='remove_normalization')(base_model.layers[2].output)
but the point is that both layers are still there.
I even tried:
model.layer.pop(1)
model.layer.pop(2)
But nothing to do!!
Do you have any recommendation?
My compute platform has two GPUs and I use MirroredStrategy() to share the load between the two GPUs. I run into problems when I use SparseTopKCategoricalAccuracy as the metric. My code is as follows:
with mirrored_strategy.scope():
model=Sequential()
model.add(Dense(512, kernel_initializer= 'he_uniform', input_shape=(X_train.shape[1],),
activity_regularizer=l1(1.e-3)))
model.add(Activation('relu'))
model.add(BatchNormalization(momentum=0.0))
model.add(Dropout(0.2))
model.add(Dense(4096, kernel_initializer= 'he_uniform', activation='relu',
activity_regularizer=l1(1.e-4)))
model.add(BatchNormalization(momentum=0.0))
model.add(Dropout(0.3))
model.add(Dense(512, kernel_initializer= 'he_uniform', activation='relu',
activity_regularizer=l1(1.e-4)))
model.add(BatchNormalization(momentum=0.0))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
opt=Adam(learning_rate=0.001)
m=SparseTopKCategoricalAccuracy(k=10)
model.compile(optimizer=opt, loss='sparse_categorical_crossentropy', metrics=m)
es = EarlyStopping(monitor='val_sparse_top_k_categorical_accuracy', patience=10, mode='max', verbose=0, restore_best_weights=True)
lr = LearningRateScheduler(lr_scheduler)enter code here
The error message I received is as follows:
ValueError: Metric (<tensorflow.python.keras.metrics.SparseTopKCategoricalAccuracy object at 0x7fee452d4f90>) passed to model.compile was created inside of a different distribution strategy scope than the model. All metrics must be created in the same distribution strategy scope as the model (in this case <tensorflow.python.distribute.mirrored_strategy.MirroredStrategy object at 0x7ff5cd1ff490>). If you pass in a string identifier for a metric to compile the metric will automatically be created in the correct distribution strategy scope.
If I get rid of "with mirrored_strateg.scope():", it works but with only one GPU. What do I need to fix to make it run on two GPUs?
The cause is in the error message:
All metrics must be created in the same distribution strategy scope as the model
Instantiate the metric within the mirrored strategy scope.
with mirrored_strategy.scope():
# Define model ...
m = SparseTopKCategoricalAccuracy(k=10)
model.compile(optimizer=opt, loss='sparse_categorical_crossentropy', metrics=m)
es = EarlyStopping(monitor='val_sparse_top_k_categorical_accuracy', patience=10, mode='max', verbose=0, restore_best_weights=True)
lr = LearningRateScheduler(lr_scheduler)
Tensorflow Version = 2.0.0
num_units = 128
inp = Input(shape = (50,36))
fw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(inp)
bw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(K.reverse(inp, axes=0))
out = layers.Concatenate(axis=-1)([fw, bw])
brnn_model = Model(inputs=inp, outputs=out)
model = tf.keras.Sequential()
model.add(layers.Masking(input_shape=(50,36)))
model.add(brnn_model)
model.add(layers.Dense(10, activation='softmax', activity_regularizer=regularizers.l2(0.001)))
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
model.summary()
model.fit(x_train, y_train, batch_size=32, epochs=14, shuffle=True, validation_data=(x_test, y_test))
With above code I get the same results as with not using Bidirectional RNN and just using a LSTM layer instead. In short the above code fails to act as Bidirectional rather it is giving same result as with a unidirectional LSTM layers.
If i use the pre-built Bidirectional Wrapper I get the expected result.
model = tf.keras.Sequential()
model.add(layers.Masking(input_shape=(50,36)))
model.add(layers.Bidirectional(layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))))
model.add(layers.Dropout(0.3, noise_shape=None, seed=None))
model.add(layers.Dense(10, activation='softmax', activity_regularizer=regularizers.l2(0.001)))model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
model.summary()
model.fit(x_train, y_train, batch_size=32, epochs=14, shuffle=True, validation_data=(x_test, y_test))
I have already checked that replacing
bw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(K.reverse(inp, axes=0))
with
bw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(inp)
has absolutely no effect on the results.
I don't understand how is this even possible.
You seem to be reversing inputs on batch dimension. And yes, that won't have the effect you desire. Instead of,
K.reverse(inp, axes=0)
Try
K.reverse(inp, axes=1)
Because the input to the LSTM is a 3D tensor where each dimension represents [batch, time, input]. So the time dimension (that is, reversing the input sequence) should be done on axis=1.
Solution:
Change axes=0 to axes=1 as pointed out by thushv89.
Concatenate reverse of bw to fw because the bw layer will contain weights for last time step at index 0. (This will correct go_backwards=True case as well
Corrected code:
fw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(inp)
bw = layers.LSTM(num_units, return_sequences=True, activity_regularizer=regularizers.l2(0.001))(K.reverse(inp, axes=1))
out = layers.Concatenate(axis=-1)([fw, K.reverse(bw, axes=1)])
I'm using tf.data.experimental.make_csv_dataset function to create the input to a Keras sequential model. My first layer is a DenseFeature that receives a list of tf.feature_column (indicator, bucketized, numeric etc). The following layers are Dense using relu activation. When I run the fit function I get the error: "Cannot create a tensor proto whose content is larger than 2GB.". What do I need to change to make this model train?
The below is the main part of the code:
train_input = tf.data.experimental.make_csv_dataset(["df_train.csv"], batch_size=64, label_name="loss_rate", num_epochs=1)
eval_input = tf.data.experimental.make_csv_dataset(["df_val.csv"], batch_size=64, label_name="loss_rate", shuffle=False, num_epochs=1)
#all_features is generated by a function (it has 87 tf.feature_column objects)
feature_layer = layers.DenseFeatures(all_features)
def deep_sequential_model():
model = tf.keras.Sequential([
feature_layer,
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss='mse',
optimizer=optimizer,
metrics=['mae', 'mse'])
return model
model = deep_sequential_model()
model.fit(train_input,
validation_data=eval_input,
epochs=10)
I'm getting the error:
/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in __init__(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def)
1696 "Cannot create a tensor proto whose content is larger than 2GB.")
1697 if not _VALID_OP_NAME_REGEX.match(node_def.name):
-> 1698 raise ValueError("'%s' is not a valid node name" % node_def.name)
1699 c_op = None
1700 elif type(node_def).__name__ == "SwigPyObject":
ValueError: '_5' is not a valid node name```
I've just found the problem. The csv that I was loading had an index column that was creating a Tensor without any name and I believe this was causing issues. I just removed the index from the csv and it worked.
I get the following error for the code snippet below:
You must feed a value for placeholder tensor
'bidirectional_1/keras_learning_phase' with dtype bool
If I add the dropout layer model.add(Dropout(dropout)), it works. Anyone knows why? The back-end is Tensorflow, Keras 2.0.1
def prep_model1(embedding_layer1, embedding_layer2, dropout=0.5):
model0 = Sequential()
model0.add(embedding_layer1)
model0.add(Bidirectional(LSTM(128, return_sequences=False, dropout=dropout)))
model1 = Sequential()
model1.add(embedding_layer2)
model1.add(Bidirectional(LSTM(128, return_sequences=False, dropout=dropout)))
model = Sequential()
model.add(Merge([model0, model1], mode='concat', concat_axis=1))
#model.add(Dropout(dropout))
model.add(Dense(1, activation='sigmoid'))
return model
Try to import K and set learning phase before your model.
from keras import backend as K
K.set_learning_phase(1) #set learning phase
From this issue