I have a dataset:
100 timesteps
10 variables
for example,
dataset = np.arange(1000).reshape(100,10)
The 10 variables are related to each other. So I want to reduce its dimension from 10 to 1.
Also, 100 time steps are related.
Which deep learning architecture is suitable for it guys?
edit:
from keras.models import Sequential
from keras.layers import LSTM, Dense
X = np.arange(1000).reshape(100,10)
model = Sequential()
model.add(LSTM(input_shape = (100, 10), return_sequences=False))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam')
model.fit(???, epochs=50, batch_size=5)
In order to compress your data, the best course of action is to use an autoencoder.
Autoencoder architecture:
Input ---> Encoder (reduces dimensionality of the input) ----> Decoder (tries to recreate input) ---> Lossy version of input
By extracting the trained encoder, we can find a way to represent your data using fewer dimensions.
from keras.layers import Input, Dense
from keras.models import Model
input = Input(shape=(10,)) #will take an input in shape of (num_samples, 10)
encoded = Dense(1, activation='relu')(input) #returns a 1D vector from input
decoded = Dense(10, activation='sigmoid)(encoded) #tries to recreate input from 1D vector
autoencoder = Model(input, decoded) #input image ---> lossy reconstruction from decoded
Now that we have the autoencoder, we need to extract what you really want- the part encoder that reduces the input's dimensionality:
encoder = Model(input, encoded) #maps input to reduced-dimension encoded form
Compile and train the autoencoder:
autoencoder.compile(optimizer='adam', loss='mse')
X = np.arange(1000).reshape(100, 10)
autoencoder.fit(X, X, batch_size=5, epochs=50)
Now you can use the encoder to reduce dimensionality:
encoded_form = encoder.predict(<something with shape (samples, 10)>) #outs 1D vector
You probably also want the decoder as well. If you are going to use it put this block of code right before you compile and fit the autoencoder:
encoded_form = Input(shape=(1,))
decoder_layer = autoencoder.layers[-1]
decoder = Model.(encoded_form, decoder_layer(encoded_form))
Related
Short Version:
I am trying to feed my data in the form of sparse matrix (of the type scipy.sparse._csr.csr_matrix') into a Tensorflow Keras Neural Network model. I highly appreciate any guidance. todense() and toarray() are not options for me. Also feeding in mini batches is not preferred.
Long version (including my efforts):
The problem is about a deep learning model with text, categorical and numerical features. My TfidfVectorizer creates a huge matrix which cannot be fed into a model as dense format.
text_cols = ['ca_name']
categorical_cols = ['cua_name','ca_category_modified']
numerical_cols = ['vidim1', 'vidim2', 'vidim3', 'vim', 'vid']
title_transformer = TfidfVectorizer()
numerical_transformer = MinMaxScaler()
categorical_transformer = OneHotEncoder(handle_unknown='ignore')
preprocessor = ColumnTransformer(
transformers=[
('title', title_transformer, text_cols[0]),
('num', numerical_transformer, numerical_cols),
('cat', categorical_transformer, categorical_cols)
])
# df['dur_linreg] is my numerical target
X_train, X_test, y_train, y_test = train_test_split(df[text_cols+categorical_cols+numerical_cols], df['dur_linreg'], test_size=0.2, random_state=42)
# fit_transform the preprocessor on X_train, only transform X_test
X_train_transformed = preprocessor.fit_transform(X_train)
X_test_transformed = preprocessor.transform(X_test)
I can build and compile a model as following:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(64, activation='relu', input_shape=(X_train_transformed.shape[1],)))
modeladd(tf.keras.layers.Dense(1))
model.compile(optimizer='adam', loss='mean_squared_error')
But cannot fit it:
history = model.fit(X_train_transformed, y_train, epochs=20, batch_size=32, validation_data=(X_test_transformed, y_test))
InvalidArgumentError: Graph execution error: TypeError: 'SparseTensor' object is not subscriptable
Obviously because I am feeding the model with a sparse scipy.sparse._csr.csr_matrix matrix.
The size of my matrix and my resources restrict me to transform it to
dense format:
X_train_transformed.todense()
MemoryError: Unable to allocate 205. GiB for an array with shape (275189, 100074) and data type float64
2) (obviously) array:
X_train_transformed.toarray()
MemoryError: Unable to allocate 205. GiB for an array with shape (275189, 100074) and data type float64
According to a post "https://stackoverflow.com/questions/41538692/using-sparse-matrices-with-keras-and-tensorflow" I there are two approaches
" Keep it as a scipy sparse matrix, then, when giving Keras a minibatch, make it dense
Keep it sparse all the way through, and use Tensorflow Sparse Tensors"
The second approach is preferred for me as well. Therefore, I tried the following as well:
However, again I could only build and compile the model without a problem:
from tensorflow.keras.layers import Input, Dense, Dropout
from tensorflow.keras.models import Model
input_layer = Input(shape=(X_train_transformed.shape[1],), sparse=True)
dense1 = Dense(64, activation='relu')(input_layer)
dropout1 = Dropout(0.2)(dense1)
dense2 = Dense(64, activation='relu')(dropout1)
dropout2 = Dropout(0.2)(dense2)
output_layer = Dense(1)(dropout2)
model = Model(input_layer, output_layer)
model.compile(optimizer='adam', loss='mean_squared_error')
But cannot fit it:
history = model.fit(X_train_transformed, y_train, validation_data=(X_test_transformed, y_test), epochs=5, batch_size=32)
InvalidArgumentError: Graph execution error:TypeError: 'SparseTensor' object is not subscriptable
Lastly, in case it is relevant I am using Tensorflow version 2.11.0 installed January 2023.
Many Thanks in advance for your help.
I'm trying to build a Sequential model with tensorflow.
import tensorflow as tf
import keras
from tensorflow.keras import layers
from keras import optimizers
import numpy as np
model = keras.Sequential (name="model")
model.add(keras.Input(shape=(786,)))
model.add(layers.Dense(2048, activation="relu", name="layer1"))
model.add(layers.Dense(786, activation="relu", name="layer2"))
model.add(layers.Dense(786, activation="relu", name="layer3"))
output = model.add(layers.Dense(786, activation="relu", name="output"))
model.summary()
model.compile(
optimizer=tf.optimizers.Adam(), # Optimizer
loss=keras.losses.CategoricalCrossentropy(),
metrics=[keras.metrics.SparseCategoricalAccuracy()],
)
history = model.fit(
x_train,
y_train,
batch_size=1,
epochs=5,
)
The input shape is a vector with length of 768 (so the input shape is (768,) right?), representing a chess board:
def get_dataset():
container = np.load('/content/drive/MyDrive/test_data_vector.npz')
b, v = container['arr_0'], container['arr_1']
v = np.asarray(v / abs(v).max() / 2 + 0.5, dtype=np.float32) # normalization (0 - 1)
return b, v
xtrain, ytrain = get_dataset()
print(xtrain.shape)
print(ytrain.shape)
>> (37, 786) #there are 37 samples
>> (37, 786)
But I always get the error:
ValueError: Input 0 of layer model is incompatible with the layer: expected axis -1 of input shape to have value 786 but received input with shape (1, 1, 768)
I tried with np.expand_dims(), which ended in the same Error.
The error is just a typo, as the user mentioned the issue is resolved by changing the output shape from 786 to 768 and the issue is resolved.
One suggestion based on the model structure.
The number of units are not related to your input shape, you don't have to match that number.
The number of units like 2048 and 786 in dense layer is too large and this may not help the model to learn better.
Try with smaller numbers like 32,64 etc, you can refer some of the examples in the tensorflow document.
I am implementing an autoencoder using the Fashion Mnsit dataset. The code for the encoder-
class MNISTClassifier(Model):
def __init__(self):
super(MNISTClassifier, self).__init__()
self.encoder = Sequential([
layers.Dense(128, activation = "relu"),
layers.Dense(64, activation = "relu"),
layers.Dense(32, activation = "relu")
])
self.decoder = Sequential([
layers.Dense(64, activation = "relu"),
layers.Dense(128, activation= "relu"),
layers.Dense(784, activation= "relu")
])
def call(self, x):
encoded = self.encoder(x)
decoded = self.decoder(encoded)
return decoded
autoencoder = MNISTClassifier()
now I want to train an SVM classifier on the image representations extracted from the above autoencoder mean
Once the above fully-connected autoencoder is trained, for each image, I want to extract the 32-
dimensional hidden vector (the output of the ReLU after the third encoder layer) as the
image representation and then train a linear SVM classifier on the training images of fashion mnist based on the 32-
dimensional features.
How to extract the output 32-
dimensional hidden vector??
Thanks in Advance!!!!!!!!!!!!
I recommend to use Functional API in order to define multiple outputs of your model because of a more clear code. However, you can do this with Sequential model by getting the output of any layer you want and add to your model's output.
Print your model.summary() and check your layers to find which layer you want to branch. You can access each layer's output by it's index with model.layers[index].output .
Then you can create a multi-output model of the layers you want, like this:
third_layer = model.layers[2]
last_layer = model.layers[-1]
my_model = Model(inputs=model.input, outputs=(third_layer.output, last_layer.output))
Then, you can access the outputs of both of layers you have defined:
third_layer_predict, last_layer_predict = my_model.predict(X_test)
I have a custom keras model built:
def create_model(input_dim,
filters,
kernel_size,
strides,
padding,
rnn_units=256,
output_dim=30,
dropout_rate=0.5,
cell=GRU,
activation='tanh'):
"""
Creates simple Conv-Bi-RNN model used for word classification approach.
:params:
input_dim - Integer, size of inputs (Example: 161 if using spectrogram, 13 for mfcc)
filters - Integer, number of filters for the Conv1D layer
kernel_size - Integer, size of kernel for Conv layer
strides - Integer, stride size for the Conv layer
padding - String, padding version for the Conv layer ('valid' or 'same')
rnn_units - Integer, number of units/neurons for the RNN layer(s)
output_dim - Integer, number of output neurons/units at the output layer
NOTE: For speech_to_text approach, this number will be number of characters that may occur
dropout_rate - Float, percentage of dropout regularization at each RNN layer, between 0 and 1
cell - Keras function, for a type of RNN layer * Valid solutions: LSTM, GRU, BasicRNN
activation - String, activation type at the RNN layer
:returns:
model - Keras Model object
"""
keras.losses.custom_loss = 'categorical_crossentropy'
#Defines Input layer for the model
input_data = Input(name='inputs', shape=input_dim)
#Defines 1D Conv block (Conv layer + batch norm)
conv_1d = Conv1D(filters,
kernel_size,
strides=strides,
padding=padding,
activation='relu',
name='layer_1_conv',
dilation_rate=1)(input_data)
conv_bn = BatchNormalization(name='conv_batch_norm')(conv_1d)
#Defines Bi-Directional RNN block (Bi-RNN layer + batch norm)
layer = cell(rnn_units, activation=activation,
return_sequences=True, implementation=2, name='rnn_1', dropout=dropout_rate)(conv_bn)
layer = BatchNormalization(name='bt_rnn_1')(layer)
#Defines Bi-Directional RNN block (Bi-RNN layer + batch norm)
layer = cell(rnn_units, activation=activation,
return_sequences=True, implementation=2, name='final_layer_of_rnn')(layer)
layer = BatchNormalization(name='bt_rnn_final')(layer)
layer = Flatten()(layer)
#squish RNN features to match number of classes
time_dense = Dense(output_dim)(layer)
#Define model predictions with softmax activation
y_pred = Activation('softmax', name='softmax')(time_dense)
#Defines Model itself, and use lambda function to define output length based on inputs
model = Model(inputs=input_data, outputs=y_pred)
model.output_length = lambda x: cnn_output_length(x, kernel_size, padding, strides)
#Adds categorical crossentropy loss for the classification model
model = add_categorical_loss(model , output_dim)
#compile the model with choosen loss and optimizer
model.compile(loss={'categorical_crossentropy': lambda y_true, y_pred: y_pred},
optimizer=keras.optimizers.RMSprop(), metrics=['accuracy'])
print("\r\ncompile the model with choosen loss and optimizer\r\n")
print(model.summary())
return model
and after training model:
checkpointer = ModelCheckpoint(filepath=save_path+'tst_model.hdf5')
#Train the choosen model with the data generator
hist = model.fit_generator(generator=generator.next_train(), #Calls generators next_train function which generates new batch of training data
steps_per_epoch=steps_per_epoch, #Defines how many training steps are there
epochs=epochs, #Defines how many epochs does a training process takes
validation_data=generator.next_valid(), #Calls generators next_valid function which generates new batch of validation data
validation_steps=validation_steps, #Defines how many validation steps are theere
callbacks=[checkpointer], #Defines all callbacks (In this case we only have molde checkpointer that saves the model)
verbose=verbose)
Adter thet I am trying to load the latest checkpoint model as follows:
from keras.models import load_model
model = load_model(filepath=save_path+'tst_model.hdf5')
and get:
NameError: name 'categorical_crossentropy' is not defined
What i doing wrong?
Using:
Ubuntu 18.04
Python 3.6.8
TensorFlow 2.0
TensorFlow backend 2.3.1
You must import the library.
from tensorflow.keras.losses import categorical_crossentropy
When you load your model, tensorflow will automatically try to compile it (see the compile arguments of tf.keras.load_model). There's 2 ways to give away this warning:
If you provided a custom loss for the model you must include it in the tf.keras.load_model() function (see custom_objects argument; it is a dict object).
Set the compile argument to False.
I want to build a Keras model with two inputs and two outputs which both use the same architecture/weights. Both outputs are then used to compute a single loss.
Here is a picture of my desired architecture.
This is my pseudo code:
model = LeNet(inputs=[input1, input2, input3],outputs=[output1, output2, output3])
model.compile(optimizer='adam',
loss=my_custom_loss_function([output1,outpu2,output3],target)
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
Can this approach work?
Do I need to use a different Keras API?
The architecture is fine. Here is a toy example with training data of how it can be defined using keras' functional API:
from keras.models import Model
from keras.layers import Dense, Input
# two separate inputs
in_1 = Input((10,10))
in_2 = Input((10,10))
# both inputs share these layers
dense_1 = Dense(10)
dense_2 = Dense(10)
# both inputs are passed through the layers
out_1 = dense_1(dense_2(in_1))
out_2 = dense_1(dense_2(in_2))
# create and compile the model
model = Model(inputs=[in_1, in_2], outputs=[out_1, out_2])
model.compile(optimizer='adam', loss='mse')
model.summary()
# train the model on some dummy data
import numpy as np
i_1 = np.random.rand(10, 10, 10)
i_2 = np.random.rand(10, 10, 10)
model.fit(x=[i_1, i_2], y=[i_1, i_2])
Edit given that you want to compute the losses together you can use Concatenate()
output = Concatenate()([out_1, out_2])
Any loss function you pass into model.compile will be applied to output in it's combined state. After you get the output from a prediction you can just split it back up into it's original state:
f = model.predict(...)
out_1, out_2 = f[:n], f[n:]