I'm trying to use libsvm to train a classifier on the following dataset:
(1) The number of class -1 are the same as the number of class 1
(2) The feature value has been scaled to [-1 1]
(3) I've tried
model = svmtrain(labels, examples, '-t 0');
model = svmtrain(labels, examples);
svm_parameter_selection(labels, examples); //this function is wrote by myself which searches the logc and logg and return the best.
All of these returned models are used to classify the test data. The label of the test data is always 1.
I use these models to classify the train data (examples in the above functions), the label is still always 1.
Related
I am a newbie in ML. I have a regression problem. The input layer consists of 7 Parameter. Output layer consists of 128 parameters (which are frequency bands) So I have trained my model and saved the model as well as input- and outputScaler (which is MinMaxScaler for scaling a new input array and retransforming the prediction)
After training, I want to give totally new input arrays without a pause. Just imagine like : we are having (1,7) shape of new inputs in every second. And I am forward them by one by to our loaded model.
loaded_model = tf.keras.models.load_model("model")
inputScaler = joblib.load(input_sc)
outputScaler = joblib.load(output_sc)
new_input = inputScaler.fit_transform(new_input )
prediction = loaded_model.predict(new_input )
inversed_predictData = outputScaler.inverse_transform(prediction)
sum_pred = np.sum(inversed_predictData, axis=1)
sum_pred
We are getting "new_input" in every second and aiming to get a new prediction (sum_pred). But it returns always the first one.
I think the model has to be reset in some way.
I expect to get a different prediction values because I am feeding the model always with a new dataset. (Btw: some parameters have very similar values but still they are different.)
I'm training a network with temporal data, and determine which of ~60 outputs are "active" at any given timestep (classified as 1 or 0 in the label data) - so I have an output of 60x1 floats that should represent a probability.
My input data is shaped as (X, 1, frames, dataPoints) - where X is the number of recorded sequences I have (I'm new to ML, I think this is 'batches'), frames is how long the longest sequence is (the rest are -1 padded and masked), and dataPoints is the actual input data for any given frame.
This is mostly an LTSM layer with return_sequences, but my input data is unbalanced.
For any given timestep, odds are ~85% that AN output is activated - but for any given output it's likely active at most 5% of the time.
When I attempted to apply a class weight of {0: 0.01, 1:0.99} (pending tuning), I get an error stating "class_weight not supported for 3+ dimensional targets". I've done some googling and people are suggesting compiling with sample_weight_mode of temporal and modifying sample weight, but (A) that doesn't seem right for my data (no individual sample is more important, but each 1 classification within all the samples is important), and (B) I don't understand the dimensionality of what that's doing.
How can I apply the class weighting to help balance each 1 classification with this data structure?
Side note: I'm rescaling the output of the LSTM to 0->1 since it uses tanh activation (and must use tanh activation for CUDA acceleration), and from_logits=False in my binary cross entropy loss.
Extra points if I can just use built-in tf/keras stuff and not have to write a custom loss function.
EDIT to include some code:
I have a data generator that outputs x and y in the shape of:
x.shape == (1, frameCount, inputFeatureLength) where frameCount is the number of frames in the temporal sequence, and inputFeatureLength is the size of the input data (around 100).
y.shape == (1, frameCount, outputSize) where outputSize is about 60 features.
I can successfully compile the mode, but when I try to model.fit with class_weight={0:0.01, 1:0.99} as an argument, I get the error ValueError: class_weight not supported for 3+ dimensional targets.
I've looked into sample weights, but as far as I can tell even using sample_weight_mode="temporal" on model.fit it'll let me give sample weights per frame of output, but not per each of the ~60 outputs per frame.
In the spacy's text classification train_textcat example, there are two labels specified Positive and Negative. Hence the cats score is represented as
cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels]
I am working with Multilabel classfication which means i have more than two labels to tag in one text. I have added my labels as
textcat.add_label("CONSTRUCTION")
and to specify cats score I have used
cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels]
I am pretty sure this is not correct. Any suggestions how to specify the scores for cats in multilabel classification and how to train multilabel classification? Does the example from spacy works for multilabel classification too?
If I understood you correctly, you have a list of categories, and your data can have multiple categories at once. In that case you cannot use "POSITIVE": bool(y), "NEGATIVE": not bool(y) to mark your classes. Instead, try writing a function which will return a dictionary with categories based on the classes. For example, consider having a following list of categories: categories = ['POLITICS', 'ECONOMY', 'SPORT']. Now, you can iterate over you train data, calling a function for each training example.
This function can look like this:
def func(categories):
cats = {'POLITICS': 0, 'ECONOMY': 0, 'SPORT': 0}
for category in categories:
cats[category] = 1
return {'cats': cats}
Having a training example with two categories (for example POLITICS and ECONOMY), you can call this function with a list of categories (labels = func(['POLITICS', 'ECONOMY']) and you will get a full dictionary with classes for this example
The example scripts are mainly quick demos for a single use case and you're right that this isn't the right kind of evaluation for a multilabel case.
The underlying spacy Scorer and the spacy evaluate CLI (https://spacy.io/api/cli#evaluate) report the macro-averaged AUC ROC score for multilabel classification.
You can use the Scorer with nlp.evaluate() (https://spacy.io/api/language#evaluate) or through spacy evaluate / spacy train.
If your data is in the simple TRAIN_DATA format from the example script, nlp.evaluate() is probably the easiest way to run the Scorer, since spacy evaluate would require you to convert your data to spacy's internal JSON training format.
The model settings (specified when you initialize the pipeline component) are used to pick an appropriate evaluation metric (obviously these aren't the only possible metrics, just one suitable metric for each configuration):
f-score on positive label for binary exclusive,
macro-averaged f-score for 3+ exclusive,
macro-averaged AUC ROC score for multilabel
I am using an AlexNet from here
The data there has 1000 classes so it has weights according to that. How do I make it work for predicting values for my data that has 3 classes?
I know I have to change the weights but I don't know how.
You need to
add one more layer on top of the pre trained network
. This will be your
output layer
.
This output of the 1000 class layer will be the input of this layer, and it will give your 3 classes as output.
After that train this new network with your images
You just have to set num_classes = 3 which will reduce the number of output classes for both the model output tensor and the separately defined placeholder y.
The number of weights, i.e. parameters, will be adapted accordingly when calling model = AlexNet(....
I want to ask the model to predict the output for some random samples during the learning process. Currently, I built a class which derived from tf.contrib.learn.monitors.EveryN and overwrite every_n_step_end as follows:
def every_n_step_end(self, step, outputs):
# evaluate the model on the validation set
self._estimator.evaluate(
input_fn=input_fn_eval,
metrics=validation_metrics,
steps=None)
# generate some random samples
for _ in range(FLAGS.num_samples):
input_sample, output_sample, label = random.choice(validation_set)
prob = self._estimator.predict(input_fn=lambda: get_features(input_sample, output_sample))
print("{}:{}:{}:{}".format(input_sample, output_sample, prob[0, 0], label))
The problem is that is each iteration, predict function load the model from checkpoints and etc. Is it the proper way?
predict method get an input function(input_fn) to preprocess and feed data into your models. So it's easy to create an input function and then pass it to predict method to get a list of predicted probabilities. See Building Input Functions with tf.contrib.learn, for more information about writing input_fn function.