I'm building an RNN and am having trouble passing in the data. The csv file I'm pulling data from has a sentence column, and a label column that's filled with a binary classification value (1 or 0). This is how I'm preprocessing right now:
data = pd.read_csv(r'/cybersecurity-sqlinjection/sqli.csv', encoding='utf-8')
vectorizer = TfidfVectorizer(norm = False, smooth_idf = False, analyzer='word', stop_words=stopwords.words('english'))
sentence_vectors = vectorizer.fit_transform((data['Sentence'].values.astype('U')))
df = pd.DataFrame(sentence_vectors.toarray())
X=df[df.columns]
y=data['Label']
X_train, X_test, y_train, y_test =train_test_split(X,y, train_size=0.8, test_size=0.2, random_state=42)
X.head()
Next I was passing in X_train to an LSTM model. At this point I was receiving errors about the shape of the data being passed in to the model, so I used the first response on this issue. I added this to the end of my code, before inputting the data into the model.
X_train_shape = X_train.shape #outputs (19327, 15016)
X_train = X_train.values.reshape(-1, 1, 15016)
model = keras.models.Sequential()
model.add(keras.layers.LSTM(15, input_shape=(1, 15016), return_sequences=True))
Now this error is being returned ValueError: Shapes (None, 1) and (None, 1, 10) are incompatible
I'm not sure what the issue is and would appreciate any help!
Related
I am using the keras functional api, but i'm getting an error about the input shape of the model -
ValueError: Input 0 is incompatible with layer financial_model: expected shape=(None, 1, 62), found shape=(1, 62)
samples = np.array(samples, dtype=np.float64)
labels = np.array(labels, dtype=np.uint8)
x_train, x_test, y_train, y_test = train_test_split(samples, labels, test_size=0.33,
random_state=42)
min_max = MinMaxScaler()
x_train = min_max.fit_transform(x_train)
lstm_input = np.expand_dims(x_train, axis=1).shape
inputs = keras.Input(shape=(lstm_input[1],lstm_input[2]))
hidden = keras.layers.LSTM(lstm_input[2], activation='tanh')(inputs)
output = keras.layers.Dense(2)(hidden)
model = keras.Model(inputs=inputs, outputs=output, name="financial_model")
model.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.Adam(learning_rate=0.001),
metrics=["accuracy"],
)
model.summary()
history = model.fit(x_train, y_train, batch_size=1, epochs=5, validation_split=0.2)
I've learnt from similar questions that the batch size is omitted in the input shape dimensions. How do I feed a 3 dimensional input shape into the lstm layer when the batch size is left out in the input object?
Since I have less than 50 reputation, I cannot comment. I'm not sure of this, but as the error says, your input shape is wrong. You have to add another dimension to it. Try something like this:
inputs = keras.Input(shape=(lstm_input[1],lstm_input[2], 1))
So I have this neural network and I am feeding examples "X" and labels "Y" whose shapes are:
X.shape = (10,10,2)
Y.shape = (10,10,2)
The code for the model looks like:
import tensorflow as tf
from convert import process
import numpy as np
X, Y, rate = process('songs/song1.wav')
X = np.array(X[:10])
Y = np.array(Y[:10])
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(128))
model.add(tf.keras.layers.Dense(128))
model.add(tf.keras.layers.Dense(20))
model.compile(optimizer='adam', loss='categorical_crossentropy')
model.fit(X, Y, epochs=2)
Now for some reason once I run this i get the error:
ValueError: Shapes (None, 10, 2) and (None, 20) are incompatible
I am confused because I fed it data where each example of both "X" and "Y" have shapes (10, 2). So why is it saying that I passed it (None, 10, 2) and (None, 20)
Your last layer uses linear activation whereas you choose categorical_crossentropy loss. Set either
model.add(tf.keras.layers.Dense(20, activations='softmax'))
....loss='categorical_crossentropy')
or,
model.add(tf.keras.layers.Dense(20))
....loss='mse')
Also check your data shape, especially the label (y).
when feeding a tf.data.Dataset to train EfficientnetB0 model I get the following error:
ValueError: in converted code:
C:\Users\fconrad\AppData\Local\Continuum\anaconda3\envs\venv_spielereien\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py:677 map_fn
batch_size=None)
C:\Users\fconrad\AppData\Local\Continuum\anaconda3\envs\venv_spielereien\lib\site-packages\tensorflow_core\python\keras\engine\training.py:2410 _standardize_tensors
exception_prefix='input')
C:\Users\fconrad\AppData\Local\Continuum\anaconda3\envs\venv_spielereien\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:573 standardize_input_data
'with shape ' + str(data_shape))
ValueError: Error when checking input: expected efficientnet-b0_input to have 4 dimensions, but got array with shape (224, 224, 3)
I realy wonder why this happens, since when I create a batch from my Dataset:
train_generator = (tf.data.Dataset
.from_tensor_slices((train_imgs, train_labels))
.map(read_img)
.map(flip_img)
.map(brightness)
.map(blur)
.map(noise)
.map(rotate_90)
.repeat()
.shuffle(512)
.batch(BATCH_SIZE)
.prefetch(True))
validation_generator = (tf.data.Dataset
.from_tensor_slices((validation_imgs, validation_labels))
.map(read_img)
)
print(train_generator.__iter__().__next__()[0].shape)
I get the expected result (64, 224, 224, 3).
But after creating the model the error above raises when I start training:
effn = tfkeras.EfficientNetB0(include_top=False, input_shape=img_shape, classes=4)
effn_model = tf.keras.Sequential()
effn_model.add(effn)
effn_model.add(tf.keras.layers.GlobalAveragePooling2D())
effn_model.add(tf.keras.layers.Dense(4, 'softmax'))
effn_model.compile(optimizer= 'adam', loss='categorical_crossentropy', metrics= ['categorical_accuracy'])
effn_model.fit(train_generator,
epochs=20,
steps_per_epoch=train_imgs.shape[0] // BATCH_SIZE,
validation_data= validation_generator)
Does anyone know why the slices from dataset have shape (64,224,224,3) but the model doesnt recognize the batch dimension? when I try to train a keras.application model, everything works fine.
I use tensorflow 2.1 and the pip install of efficientnet. Thanks
as explained here keras.io/api/applications/efficientnet/
input_shape: Optional shape tuple, only to be specified if include_top is False. It should have exactly 3 inputs channels.
as so try this->
from tensorflow.keras.applications.efficientnet import EfficientNetB0, EfficientNetB5
mm = EfficientNetB0(include_top=True, weights=None, input_tensor=None, input_shape=(128, 128, 3), pooling=None, classes=2, classifier_activation="sigmoid")
mm.summary()
note the input_shape=(128, 128, 3) It has 3 channels.
I have a seq2seq model which is working fine. I want to add an embedding layer in this network which I faced with an error.
this is my architecture using pretrained word embedding which is working fine(Actually the code is almost the same code available here, but I want to include the Embedding layer in the model rather than using the pretrained embedding vectors):
LATENT_SIZE = 20
inputs = Input(shape=(SEQUENCE_LEN, EMBED_SIZE), name="input")
encoded = Bidirectional(LSTM(LATENT_SIZE), merge_mode="sum", name="encoder_lstm")(inputs)
encoded = Lambda(rev_ent)(encoded)
decoded = RepeatVector(SEQUENCE_LEN, name="repeater")(encoded)
decoded = Bidirectional(LSTM(EMBED_SIZE, return_sequences=True), merge_mode="sum", name="decoder_lstm")(decoded)
autoencoder = Model(inputs, decoded)
autoencoder.compile(optimizer="sgd", loss='mse')
autoencoder.summary()
NUM_EPOCHS = 1
num_train_steps = len(Xtrain) // BATCH_SIZE
num_test_steps = len(Xtest) // BATCH_SIZE
checkpoint = ModelCheckpoint(filepath=os.path.join('Data/', "simple_ae_to_compare"), save_best_only=True)
history = autoencoder.fit_generator(train_gen, steps_per_epoch=num_train_steps, epochs=NUM_EPOCHS, validation_data=test_gen, validation_steps=num_test_steps, callbacks=[checkpoint])
This is the summary:
Layer (type) Output Shape Param #
=================================================================
input (InputLayer) (None, 45, 50) 0
_________________________________________________________________
encoder_lstm (Bidirectional) (None, 20) 11360
_________________________________________________________________
lambda_1 (Lambda) (512, 20) 0
_________________________________________________________________
repeater (RepeatVector) (512, 45, 20) 0
_________________________________________________________________
decoder_lstm (Bidirectional) (512, 45, 50) 28400
when I change the code to add the embedding layer like this:
inputs = Input(shape=(SEQUENCE_LEN,), name="input")
embedding = Embedding(output_dim=EMBED_SIZE, input_dim=VOCAB_SIZE, input_length=SEQUENCE_LEN, trainable=True)(inputs)
encoded = Bidirectional(LSTM(LATENT_SIZE), merge_mode="sum", name="encoder_lstm")(embedding)
I received this error:
expected decoder_lstm to have 3 dimensions, but got array with shape (512, 45)
So my question, what is wrong with my model?
Update
So, this error is raised in the training phase. I also checked the dimension of the data being fed to the model, it is (61598, 45) which clearly do not have the number of features or here, Embed_dim.
But why this error raises in the decoder part? because in the encoder part I have included the Embedding layer, so it is totally fine. though when it reached the decoder part and it does not have the embedding layer so it can not correctly reshape it to three dimensional.
Now the question comes why this is not happening in a similar code?
this is my view, correct me if I'm wrong. because Seq2Seq code usually being used for Translation, summarization. and in those codes, in the decoder part also there is input (in the translation case, there is the other language input to the decoder, so the idea of having embedding in the decoder part makes sense).
Finally, here I do not have seperate input, that's why I do not need any separate embedding in the decoder part. However, I don't know how to fix the problem, I just know why this is happening:|
Update2
this is my data being fed to the model:
sent_wids = np.zeros((len(parsed_sentences),SEQUENCE_LEN),'int32')
sample_seq_weights = np.zeros((len(parsed_sentences),SEQUENCE_LEN),'float')
for index_sentence in range(len(parsed_sentences)):
temp_sentence = parsed_sentences[index_sentence]
temp_words = nltk.word_tokenize(temp_sentence)
for index_word in range(SEQUENCE_LEN):
if index_word < sent_lens[index_sentence]:
sent_wids[index_sentence,index_word] = lookup_word2id(temp_words[index_word])
else:
sent_wids[index_sentence, index_word] = lookup_word2id('PAD')
def sentence_generator(X,embeddings, batch_size, sample_weights):
while True:
# loop once per epoch
num_recs = X.shape[0]
indices = np.random.permutation(np.arange(num_recs))
# print(embeddings.shape)
num_batches = num_recs // batch_size
for bid in range(num_batches):
sids = indices[bid * batch_size : (bid + 1) * batch_size]
temp_sents = X[sids, :]
Xbatch = embeddings[temp_sents]
weights = sample_weights[sids, :]
yield Xbatch, Xbatch
LATENT_SIZE = 60
train_size = 0.95
split_index = int(math.ceil(len(sent_wids)*train_size))
Xtrain = sent_wids[0:split_index, :]
Xtest = sent_wids[split_index:, :]
train_w = sample_seq_weights[0: split_index, :]
test_w = sample_seq_weights[split_index:, :]
train_gen = sentence_generator(Xtrain, embeddings, BATCH_SIZE,train_w)
test_gen = sentence_generator(Xtest, embeddings , BATCH_SIZE,test_w)
and parsed_sentences is 61598 sentences which are padded.
Also, this is the layer I have in the model as Lambda layer, I just added here in case it has any effect ever:
def rev_entropy(x):
def row_entropy(row):
_, _, count = tf.unique_with_counts(row)
count = tf.cast(count,tf.float32)
prob = count / tf.reduce_sum(count)
prob = tf.cast(prob,tf.float32)
rev = -tf.reduce_sum(prob * tf.log(prob))
return rev
nw = tf.reduce_sum(x,axis=1)
rev = tf.map_fn(row_entropy, x)
rev = tf.where(tf.is_nan(rev), tf.zeros_like(rev), rev)
rev = tf.cast(rev, tf.float32)
max_entropy = tf.log(tf.clip_by_value(nw,2,LATENT_SIZE))
concentration = (max_entropy/(1+rev))
new_x = x * (tf.reshape(concentration, [BATCH_SIZE, 1]))
return new_x
Any help is appreciated:)
I tried the following example on Google colab (TensorFlow version 1.13.1),
from tensorflow.python import keras
import numpy as np
SEQUENCE_LEN = 45
LATENT_SIZE = 20
EMBED_SIZE = 50
VOCAB_SIZE = 100
inputs = keras.layers.Input(shape=(SEQUENCE_LEN,), name="input")
embedding = keras.layers.Embedding(output_dim=EMBED_SIZE, input_dim=VOCAB_SIZE, input_length=SEQUENCE_LEN, trainable=True)(inputs)
encoded = keras.layers.Bidirectional(keras.layers.LSTM(LATENT_SIZE), merge_mode="sum", name="encoder_lstm")(embedding)
decoded = keras.layers.RepeatVector(SEQUENCE_LEN, name="repeater")(encoded)
decoded = keras.layers.Bidirectional(keras.layers.LSTM(EMBED_SIZE, return_sequences=True), merge_mode="sum", name="decoder_lstm")(decoded)
autoencoder = keras.models.Model(inputs, decoded)
autoencoder.compile(optimizer="sgd", loss='mse')
autoencoder.summary()
And then trained the model using some random data,
x = np.random.randint(0, 90, size=(10, 45))
y = np.random.normal(size=(10, 45, 50))
history = autoencoder.fit(x, y, epochs=NUM_EPOCHS)
This solution worked fine. I feel like the issue might be the way you are feeding in labels/outputs for MSE calculation.
Update
Context
In the original problem, you are attempting to reconstruct word embeddings using a seq2seq model, where embeddings are fixed and pre-trained. However you want to use a trainable embedding layer as a part of the model it becomes very difficult to model this problem. Because you don't have fixed targets (i.e. targets change every single iteration of the optimization because your embedding layer is changing). Furthermore this will lead to a very unstable optimization problem, because the targets are changing all the time.
Fixing your code
If you do the following you should be able to get the code working. Here embeddings is the pre-trained GloVe vector numpy.ndarray.
def sentence_generator(X, embeddings, batch_size):
while True:
# loop once per epoch
num_recs = X.shape[0]
embed_size = embeddings.shape[1]
indices = np.random.permutation(np.arange(num_recs))
# print(embeddings.shape)
num_batches = num_recs // batch_size
for bid in range(num_batches):
sids = indices[bid * batch_size : (bid + 1) * batch_size]
# Xbatch is a [batch_size, seq_length] array
Xbatch = X[sids, :]
# Creating the Y targets
Xembed = embeddings[Xbatch.reshape(-1),:]
# Ybatch will be [batch_size, seq_length, embed_size] array
Ybatch = Xembed.reshape(batch_size, -1, embed_size)
yield Xbatch, Ybatch
I have time series data input 72 value by separate last 6 value for test prediction. I want to use CONV1D with LSTM.
This is my code.
df = pd.read_csv('D://data.csv',
engine='python')
df['DATE_'] = pd.to_datetime(df['DATE_']) + MonthEnd(1)
df = df.set_index('DATE_')
df.head()
split_date = pd.Timestamp('03-01-2015')
train = df.loc[:split_date, ['COLUMN3DATA']]
test = df.loc[split_date:, ['COLUMN3DATA']]
sc = MinMaxScaler()
train_sc = sc.fit_transform(train)
test_sc = sc.transform(test)
X_train = train_sc[:-1]
y_train = train_sc[1:]
X_test = test_sc[:-1]
y_test = test_sc[1:]
################### Convolution #######################
X_train_t = X_train[None,:]
print(X_train_t.shape)
X_test_t = X_test[:, None]
K.clear_session()
model = Sequential()
model.add(Conv1D(6, 3, activation='relu', input_shape=(12,1)))
model.add(LSTM(6, input_shape=(1,3), return_sequences=True))
model.add(LSTM(3))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam' )
model.summary()
model.fit(X_train_t, y_train, epochs=400, batch_size=10, verbose=1)
y_pred = model.predict(X_test_t)
When I run it show error like this
ValueError: Error when checking input: expected conv1d_1_input to have shape (None, 12, 1) but got array with shape (1, 64, 1)
How to use conv1D with lstm
The problem is between your input data and your input shape.
You said in the model that your input shape is (12,1) (= batch_shape=(None,12,1))
But your data X_train_t has shape (1,64,1).
Either you fix the input shape of the model, or you fix your data if this is not the expected shape.
For variable lengths/timesteps, you can use input_shape=(None,1).
You don't need an input_shape in the second layer.