I am attempting to make a multilayer model with the functional API from keras. The idea is to take an vector that describes an image (comes from a pretrained model) and concatenate it with another vector that comes from the text associated with that image. The first branch processes the input coming from the images, the second branch takes as input the text. Both are concatenated and then processed together with a few more layers.
When I train the model with:
model = Model(inputs=[xa.input, xb.input], outputs=z)
and this fit call:
model.fit([oraciones_img, X], y, epochs=2, batch_size=5, verbose=1)
everything works out fine.
The problem comes when I want to do a prediction. I provided two arrays with the same dimensions as oraciones_img and X to the model.predict function but it throws this error:
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays: [array([[1.1081828 , 0.4828459 , 1.6334529 , ..., 0.03270344, 0.17258629,
0.03792314],
[0.8138524 , 0. , 0.65887845, ..., 0.6183274 , 0.00807555,
0.6506448 ],
[0.8...
I double checked that the dimensions from the inputs during training and testing are the same but did not work out. I tried the training data on the predict function, assuming that at least that should work out, but weirdly that throws the same error. In theory, I should be providing this:
model.predict([ (n_samples, 405504) , (n_samples, 50) ])
where:
(n_samples, 405504) are the vectors coming from n images
(n_samples, 50) are the vectors coming from n texts associated to the images
but somehow the model is asking me for a single array. I provided one to follow up, and the predict function is asking for a single array of dim4 which is a bit more confusing.
I have seen other posts regarding similar issues on the shape of the data, but all of them have problems running the fit function. I already passed that, this is about the predict function.
Any idea of what is going on?
define model
seq_length = 50
seq_length_img = (200704 + 2048)*2
InputA = Input(shape=(seq_length_img,))
InputB = Input(shape=(seq_length,))
xa = Dense(512, activation='relu')(InputA)
xa = Model(inputs = InputA, outputs= xa)
xb = Embedding(input_dim=vocab_size, output_dim=512, input_length=seq_length)(InputB)
xb = Flatten()(xb)
xb = Dense(512, activation='relu')(xb)
xb = Model(inputs = InputB, outputs= xb)
combined = concatenate([xa.output, xb.output], axis=-1)
combined = Reshape((1024, 1))(combined)
z = LSTM(1024, input_shape=(1024, 1), return_sequences=True)(combined)
z = LSTM(1024)(z)
z = Dense(100, activation='relu')(z)
z = Dense(vocab_size, activation='softmax')(z)
model = Model(inputs=[xa.input, xb.input], outputs=z)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
model.fit([oraciones_img, X], y, epochs=2, batch_size=5, verbose=1)
yhat = model.predict([feat, pred_sentence])
Related
I am trying to create a neural network using the Flux package, that takes as input an 100x3 matrix of random points, and outputs True or False. I have the corresponding labels in Y which is an 100 element array of Boolean Values (1 or 0)
This is my code so far.
# model
model = Chain(Dense(100, 32, relu), Dense(32, 1), sigmoid)
# loss function and the optimizer
loss(x, y) = Flux.binarycrossentropy(model(x), y)
opt = Flux.ADAM()
# Train
Flux.train!(loss, Flux.params(model), [(X, Y)], opt)
What I've noticed is that currently, if I call model(x) it outputs a 3 element array with probabilities, which is not the thing I want, since it should output at least a 2 element array with the probability for it to be True or False. Also if I change my model to
model = Chain(Dense(100, 32, relu), Dense(32, 100), sigmoid) it outputs a 100x3 matrix of probabilities, which is again, not correct as it should be 100x2 I believe.
In Flux.jl, the observation dimension is is always the last one.
I think in your problem, each row of the matrix is an observation. Am I right?
If that's the case, I think this will solve your problem:
using Flux
X = rand(3, 100) # each column has one observation
bools = rand(Bool, 100)
Y = hcat([[b, !b] for b in bools]...) # each column has one label
dataset = [(X, Y)]
model = Chain(
Dense(3 => 32, relu),
Dense(32 => 2),
sigmoid
)
opt = Flux.setup(Adam(), model)
loss(model, x, y) = Flux.binarycrossentropy(model(x), y)
Flux.train!(loss, model, dataset, opt)
Note: this code is using the latest version of Flux.jl
All the documentation for Keras pre-processing seems to assume a single Input. If you have a model with multiple Inputs:
x_norm = preprocessing.Normalization()
y_norm = preprocessing.Normalization()
x = layers.Input(shape=(1,))
x = x_norm(x)
y = layers.Input(shape=(1,))
y = y_norm(y)
concated = layers.Concatenate()([x, y])
output = layers.Dense(1)(concated)
model = keras.Model(inputs=[x, y], outputs=output)
It's unclear how to use adapt() on a Dataset to "train" each preprocessing layer (i.e. x_norm and y_norm). With a single Input and preprocessing layer (e.g. preprocessing_layer) you simply do:
preprocessing_layer.adapt(dataset)
But in the case of multiple inputs how do I select the right input feature to use in adapt()?
The best I've come up with so far is:
normalization_layers = {
'x': preprocessing.Normalization(),
'y': preprocessing.Normalization(),
}
for batch in dataset:
for name, layer in normalization_layers.items():
layer.adapt(batch[0][name])
I don't know if this is efficient and TensorFlow gives a warning about a tf.function (inside adapt()) being called in a loop.
I have a string of letters that follow a "grammar." I also have boolean labels on my training set of whether the string follows "the grammar" or not. Basically, my model is trying to learn determine if a string of letters follows the rules. It's a fairly simple problem (I got it out of a textbook).
I am generating my dataset like this:
def generate_dataset(size):
good_strings = [string_to_ids(generate_string(embedded_reber_grammar))
for _ in range(size // 2)]
bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))
for _ in range(size - size // 2)]
all_strings = good_strings + bad_strings
X = tf.ragged.constant(all_strings, ragged_rank=1)
# X = X.to_tensor(default_value=0)
y = np.array([[1.] for _ in range(len(good_strings))] +
[[0.] for _ in range(len(bad_strings))])
return X, y
Notice the line X = X.to_tensor(default_value=0). If this line is commented out, my model learns just fine. However, if it is not commented out, it fails to learn and the validation set performs the same as chance (50-50).
Here is my actual model:
np.random.seed(42)
tf.random.set_seed(42)
embedding_size = 5
model = keras.models.Sequential([
keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),
keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS) + 1, output_dim=embedding_size),
keras.layers.GRU(30),
keras.layers.Dense(1, activation="sigmoid")
])
optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)
model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(X_train, y_train, epochs=5, validation_data=(X_valid, y_valid))
I am using 0 as the default value for the dense tensors. The strings_to_ids doesn't use 0 for any of the values but instead starts at 1. Also, when I switch to using a Dense tensor I change ragged=True to False. I have no idea why using a dense tensor causes the model to fail, as I've used dense tensors before in similar exercises.
For additional details, see the solution from the book (exercise 8) or my own colab notebook.
So turns out the answer was that the shape of the dense tensor was different across the training set and validation set. This was because the longest sequence differed in length between the two sets (same with the test set).
I have created four 3D-CNN models and each was trained on a different (but related) set of images, such that each set has images of a different perspective of the same objects. (i.e: n objects have images from 4 different perspectives, each model is associated to a single perspective).
def get_model(width=128, height=128, depth=4):
inputs = Input((width, height, depth, 3))
x = Conv3D(filters=64, kernel_size=8,padding='same', activation="relu")(inputs)
x = MaxPool3D(pool_size=2,data_format= "channels_first", padding='same')(x)
x = BatchNormalization()(x)
x = Conv3D(filters=256, kernel_size=3,padding='same', activation="relu")(x)
x = MaxPool3D(pool_size=2,data_format= "channels_first", padding='same')(x)
x = BatchNormalization()(x)
x = GlobalAveragePooling3D()(x)
x = Dense(units=512, activation="relu")(x)
x = Dropout(0.3)(x)
outputs = Dense(units=2, activation="sigmoid")(x)
# Define the model.
model = keras.Model(inputs, outputs)
return model
I have now four pre-trained models, and I would like to combine them by removing the last dense layer (sigmoid) and instead, concatenating the dense layers of all the four models followed by an activation function (i.e: sigmoid). I would like to keep four input layers such that each will take an image of an object from one perspective. I have seen examples of concatenating an output layer of model_1 to the input layer of model_2, however, I am not sure how to deal with four separate input layers and concatenating towards the end of the model.
Let's assume that you have your pretrained model files named "A.h5" and "B.h5". You can simply load them in TensorFlow, access the layer that interrest you with the layers attribute, and merge them with the Functional API. One example could be the following :
import tensorflow as tf
pretrainedmodel_files = ["A.h5", "B.h5"]
A,B = [tf.keras.models.load_model(filename) for filename in pretrainedmodel_files]
# Skipping the last dense layer and the dropout means accessing the layer at the index -3
concat = tf.keras.layers.Concatenate()([A.layers[-3].output, B.layers[-3].output])
out = tf.keras.layers.Dense(2,activation="sigmoid")(concat)
model = tf.keras.Model(inputs=[A.input, B.input], outputs=out)
I've created two simple model with the following code:
tf.keras.Sequential(
[
tf.keras.layers.Dense(10, activation="relu", input_shape=(5,)),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(2, activation="sigmoid")
]
)
And then merged them together with my sample code.
A and B have the following architecture (visualization with netron):
And the merged network:
I'm building a CNN model to catagorise data, and am looking at cross validation with Scikit-Learn. I'm using two inputs, and would like to know how to cross validate with two inputs.
From what I can see it only accepts an X input and y output, and have tried using a list of inputs but they are read as a single list.
I've used the following code to iterate through one model and cross validate it, however I don't see any way of cross validating with two inputs.
np.random.seed(seed)
kfold = StratifiedKfold(n_splits=10m shuffle=True, random_state=seed)
for train, test in kfold.split(x, y):
input = Input(shape=(100, 150, 1))
dense = Dense(2, activation='relu')(input)
output = Dense(1, activation='relu')(dense)
model = Model(input, output)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X[train], Y[train], epochs=150, batch_size=10, verbose=0)
scores = model.evaluate(X[test], Y[test], verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
cvscores.append(scores[1] * 100)
print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvscores), numpy.std(cvscores)))
I know it's kinda late but hopefully it will be useful for someone having the same problem.
One possible solution is to concatenate both inputs and then split them accordingly once inside the loop.
As an example:
I already concatenated both inputs into 'x_agg', the first input has 2001 columns.
for train_index, valid_index in kfold.split(x_agg, y):
X_train_fold = x_agg[train_index]
X_valid_fold = x_agg[valid_index]
y_train_fold = y[train_index]
y_valid_fold = y[valid_index]
x_train_global = X_train_fold[0:, :2001] # Split First Input (Training)
x_train_local = X_train_fold[0:, 2001:]
x_valid_global = X_valid_fold[0:, :2001] # Split First Input (Validation)
x_valid_local = X_valid_fold[0:, 2001:]
model.fit([x_train_global, x_train_local], y_train_fold,...]
score = model.evaluate([x_valid_global, x_valid_local], y_valid_fold, verbose=0)[1]