I'm trying to train an svm classifier to do prediction. When I try to use the trained model, I get this error: test data does not match model. I am not why this is happening. This is my code
# to prepare the training and testing data
dat = data.frame(x = rbind(tmp1, tmp2), y = as.factor(c(rep(1, 300), rep(-1, 300))))
set.seed(1)
train_ind = sample(seq_len(nrow(dat)), size = 500)
train = dat[train_ind, ]
test = dat[-train_ind, ]
# training and prediction
library('e1071')
svmfit = svm(y ~ ., data = train, kernel ='linear', cost = 10, scale = FALSE)
ypred = predict(svmfit, test)
table(predict=ypred, truth = test$y)
The reason behind this error is that I included the ids of the observations in the training and testing data which has confused the svm classifier. The ids of the observations are in the first column. So when I removed the first column from the training and testing, it worked.
If you have a categorical predictor variable (independent variable) in your training dataset, only the categories present in training dataset can be present in test dataset. If it is the case, check if all categories in your test dataset are present into training dataset. Some times SVM assumes as categorical one a integer variable with a short range like the months represented as numbers [1:12]
Related
I am a newbie in ML. I have a regression problem. The input layer consists of 7 Parameter. Output layer consists of 128 parameters (which are frequency bands) So I have trained my model and saved the model as well as input- and outputScaler (which is MinMaxScaler for scaling a new input array and retransforming the prediction)
After training, I want to give totally new input arrays without a pause. Just imagine like : we are having (1,7) shape of new inputs in every second. And I am forward them by one by to our loaded model.
loaded_model = tf.keras.models.load_model("model")
inputScaler = joblib.load(input_sc)
outputScaler = joblib.load(output_sc)
new_input = inputScaler.fit_transform(new_input )
prediction = loaded_model.predict(new_input )
inversed_predictData = outputScaler.inverse_transform(prediction)
sum_pred = np.sum(inversed_predictData, axis=1)
sum_pred
We are getting "new_input" in every second and aiming to get a new prediction (sum_pred). But it returns always the first one.
I think the model has to be reset in some way.
I expect to get a different prediction values because I am feeding the model always with a new dataset. (Btw: some parameters have very similar values but still they are different.)
I'm learning how to train RNN model on Keras and I was expecting that training a model to predict the Moving Average of the last N steps would be quite easy.
I have a time series with thousands of steps and I'm able to create a model and train it with batches of data.
If I train it with the following model though, the test set predictions differ a lot from real values. (batch = 30, moving average window = 10)
inputs = tf.keras.Input(shape=(batch_length, num_features))
x = tf.keras.layers.LSTM(10, return_sequences=False)(inputs)
outputs = tf.keras.layers.Dense(num_labels)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs, name="test_model")
To be able to get good predictions, I need to add another layer of TimeDistributed, getting 2D predictions instead of 1D ones (I get one prediction per each time step)
inputs = tf.keras.Input(shape=(batch_length, num_features))
x = tf.keras.layers.LSTM(10, return_sequences=True)(inputs)
x = tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(num_labels))(x)
outputs = tf.keras.layers.Dense(num_labels)(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs, name="test_model")
I suggest that if your goal is to give as input the last 10 timesteps and have as a prediction the moving average to try a regressor model with Densely Connected layers rather than an RNN. (Linear activation with regularization might work well enough)
That option would be cheaper to train and run than an LSTM
I've been trying for several hours to complete this task with no success.
I have a very large dataset which is comprised of the following structure:
I want to split this data into X and Y (and pass Y to tf.to_categorical) as in the picture using the tf.data.Dataset API, but unfortunately every attempt of me trying to use it has ended up with some kind of error.
How do I use tf.data.Dataset to:
Split each row to x and y.
Convert Y to categorical with tf.to_categorical.
Split the dataset into batches.
Feed my model with the dataset.
My current attempt:
def map_sequence():
for sequence in input_sequences:
yield sequence[:-1], keras.utils.to_categorical(sequence[-1], total_words)
dataset = tf.data.Dataset.from_generator(map_sequence,
(tf.int32, tf.int32),
(tf.TensorShape(title_length-1), tf.TensorShape(total_words)))
But when I try to train my model with the following code:
inputs = keras.layers.Input(shape=(title_length-1, ))
x = keras.layers.Embedding(total_words, 32)(inputs)
x = keras.layers.Bidirectional(keras.layers.LSTM(64, return_sequences=True))(x)
x = keras.layers.Bidirectional(keras.layers.LSTM(64))(x)
predictions = keras.layers.Dense(total_words, activation='softmax')(x)
model = keras.Model(inputs=inputs, outputs=predictions)
model.compile('Adam', 'categorical_crossentropy', metrics=['acc'])
model.fit(dataset)
I am getting this error: ValueError: Shapes (32954, 1) and (65, 32954) are incompatible
I think you have a similar problem as in this question. Keras expects the dataset that you give to produce batches, not individual examples. Since you are giving it two one-dimensional vectors at a time, Keras interprets that each of these is a batch of examples with one feature. So, your X data, which has 65 elements, is interpreted as a batch of 65 examples with a single feature (a 65x1 tensor). This fixes the batch size to 65. The output of the model has then shape 65x32,954 (which I assume is the value of total_words). But your Y vector, with 32,954 elements, is again interpreted as a batch of 32,954 with one features (32,954x1 tensor). These two things don't match, hence the error. You should be able to fix it by simply making a new dataset with batch before passing it to fit.
In any case, if you input_sequences is a NumPy array, as it seems to be, your method to produce the dataset is not really good, as using a generator will be really slow. This is a better way to do the same:
def map_sequence(sequence):
# Using tf.one_hot instead of keras.utils.to_categorical
# because we are working with TensorFlow tensors now
return sequence[:-1], tf.one_hot(sequence[-1], total_words)
dataset = tf.data.Dataset.from_tensor_slices(input_sequences)
dataset = dataset.map(map_sequence)
dataset = dataset.batch(batch_size)
I am generating imagenet tags for all keyframes in a video with a single call and have this code:
# all keras/tf/mobilenet imports
model_imagenet = MobileNetV2(weights='imagenet')
frames_list = []
for frame in frame_set:
frame_img = frame.to_image()
frame_pil = frame_img.resize((224,224), Image.ANTIALIAS)
ts = int(frame.pts)
x = image.img_to_array(frame_pil)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
frames_list.append(x)
print(len(frames_list))
preds_list = model_imagenet.predict_on_batch(frames_list)
print("[*]",preds_list)
The result appears thus:
frames_list count: 125
and the predictions thus, one row of 1000 dimensions (imagenet classes), shouldn't it be 125 rows?:
[[1.15425530e-04 1.83317825e-04 4.28701424e-05 2.87547664e-05
:
7.91769926e-05 1.30803732e-04 4.81895368e-05 3.06891889e-04]]
This is generating prediction for a single row in the batch. I have tried both predict and predict_on_batch with the same result.
How can I get a bulk prediction for say 200 frames at one go with Keras/Tensorflow/Mobilenet?
ImageNet is a popular database which consists of 1000 different categories.
The dimension of 1000 is natural and to be expected, since for one image the softmax outputs a probability for each of the 1000 classes.
EDIT: For multiple image predictions, you should use predict_generator(). In addition, as of TensorFlow 2.0, if you use the Keras backend, predict_generator() has been deprecated in favor of simple predict, which also allows input data as generators.
E.g. : (from How to use predict_generator with ImageDataGenerator?) :
test_datagen = ImageDataGenerator(rescale=1./255)
#Modify the batch size here
test_generator = test_datagen.flow_from_directory(
test_dir,
target_size=(200, 200),
color_mode="rgb",
shuffle = False,
class_mode='categorical',
batch_size=1)
filenames = test_generator.filenames
nb_samples = len(filenames)
predict = model.predict_generator(test_generator,steps = nb_samples)
Please bear in mind that it will be highly unlikely to have a lot of predictions at once, since it is constrained to the memory of the video card.
Also, note the difference between predict and predict_on_batch: What is the difference between the predict and predict_on_batch methods of a Keras model?
OK, here is how I solved it, hope this helps someone else:
preds_list = model_imagenet.predict(np.vstack(frames_list),batch_size=32)
print("[*]",preds_list)
Please note the np.vstack and adjust the batch_size to whatever your computer is capable of.
I met with a weird periodical pattern for train loss in a label classification problem using AdamOptimzer in tensorflow.
As shown in the image, train loss increased slightly with a sudden drop in every epoch. What is the possible reason of this?
AdamOptimizer is used with a warm-up+polynomial decay learning rate.
opt = tf.train.AdamOptimizer(learning_rate=learning_rate, epsilon=0.00001)
train_op = opt.optimize(cross_entropy_loss, lr)
Data pipeline is described below.
# data pipeline
dataset = tf.data.Dataset.list_files(tfrecords_names)
# shuffle filenames
dataset = dataset.shuffle(buffer_size=config.num_parallel_readers)
# repeat filename lists
dataset = dataset.repeat(config.epoch)
dataset = dataset.interleave(tf.data.TFRecordDataset,
cycle_length=config.num_parallel_readers)
# parse samples
dataset = dataset.map(map_func=parse_record,
num_parallel_calls=config.num_parallel_calls)
# prefetch samples
dataset = dataset.prefetch(buffer_size=config.prefetch_buffer_size)
# shuffle samples
dataset = dataset.shuffle(buffer_size=config.shuffle_buffer_size)
# make a batch
dataset = dataset.batch(config.batch_size)