I'm trying to use XGBoost to predict the rank for a set of features for a given query. I managed to train a model it but I'm confused around the input data when I ask for a prediction.
I'm trying to understand if I'm doing something wrong or this is not the right approach.
What I'm doing:
I parse the training data (see here a sample) and feed it in a DMatrix such that the first column represents the quality-of-the-match and the following columns are the scores on different properties and also send the docIds as labels
I configure the group sizes
The training seems to work fine, I get not errors, and I use the rank:pairwise objective
For prediction, I use a fake entry with fake scores (1 row, 2 columns see here) and I get back a single float value.
I'm trying to understand:
1. Do I need to feed in a label for the prediction ?
My understanding is that labels are similar to "doc ids" so at prediction time I don't see why I need them
2. Do I need to set the group size when doing predictions ? And if so, what does it represent ?
My understanding is that groups are for training data to assist ranking "per query". How does that correlate with predictions? Do I set a group size anyway?
3. How do I correlate the "group" from the training with the prediction?
How do I figure out the pair (score, group) from the result of the prediction, given I only get back a single float value - what group is that prediction for?
Related
So I have been looking at XGBoost as a place to start with this, however I am not sure the best way to accomplish what I want.
My data is set up something like this
Where every value, whether it be input or output is numerical. The issue I'm facing is that I only have 3 input data points per several output data points.
I have seen that XGBoost has a multi-output regression method, however I am only really seeing it used to predict around 2 outputs per 1 input, whereas my data may have upwards of 50 output points needing to be predicted with only a handful of scalar input features.
I'd appreciate any ideas you may have.
For reference, I've been looking at mainly these two demos (they are the same idea just one is scikit and the other xgboost)
https://machinelearningmastery.com/multi-output-regression-models-with-python/
https://xgboost.readthedocs.io/en/stable/python/examples/multioutput_regression.html
I am using item_similarity_recommender from Graphlab and I noticed that predict() function assigns zero to most of the predicted items even though the ratings go from 1-5. This of course results in having very high RMSE.
Since item_similarity_recommender is based on KNN, I assume that the predicted ratings should be from 1-5. Can anyone explains to me why this is happening??
The code is simply as the following:
train, test = graphlab.recommender.util.random_split_by_user(dataset, max_num_users=1000)
m = graphlab.recommender.item_similarity_recommender.create(train, target='target', only_top_k=65)
rmse=m.evaluate_rmse(test, target='target')
print(m.predict(test))
I want to predict stock price.
Normally, people would feed the input as a sequence of stock prices.
Then they would feed the output as the same sequence but shifted to the left.
When testing, they would feed the output of the prediction into the next input timestep like this:
I have another idea, which is to fix the sequence length, for example 50 timesteps.
The input and output are exactly the same sequence.
When training, I replace last 3 elements of the input by zero to let the model know that I have no input for those timesteps.
When testing, I would feed the model a sequence of 50 elements. The last 3 are zeros. The predictions I care are the last 3 elements of the output.
Would this work or is there a flaw in this idea?
The main flaw of this idea is that it does not add anything to the model's learning, and it reduces its capacity, as you force your model to learn identity mapping for first 47 steps (50-3). Note, that providing 0 as inputs is equivalent of not providing input for an RNN, as zero input, after multiplying by a weight matrix is still zero, so the only source of information is bias and output from previous timestep - both are already there in the original formulation. Now second addon, where we have output for first 47 steps - there is nothing to be gained by learning the identity mapping, yet network will have to "pay the price" for it - it will need to use weights to encode this mapping in order not to be penalised.
So in short - yes, your idea will work, but it is nearly impossible to get better results this way as compared to the original approach (as you do not provide any new information, do not really modify learning dynamics, yet you limit capacity by requesting identity mapping to be learned per-step; especially that it is an extremely easy thing to learn, so gradient descent will discover this relation first, before even trying to "model the future").
I use the python implementation of XGBoost. One of the objectives is rank:pairwise and it minimizes the pairwise loss (Documentation). However, it does not say anything about the scope of the output. I see numbers between -10 and 10, but can it be in principle -inf to inf?
good question. you may have a look in kaggle competition:
Actually, in Learning to Rank field, we are trying to predict the relative score for each document to a specific query. That is, this is not a regression problem or classification problem. Hence, if a document, attached to a query, gets a negative predict score, it means and only means that it's relatively less relative to the query, when comparing to other document(s), with positive scores.
It gives predicted score for ranking.
However, the scores are valid for ranking only in their own groups.
So we must set the groups for input data.
For esay ranking, refer to my project xgboostExtension
If I understand your questions correctly, you mean the output of the predict function on a model fitted using rank:pairwise.
Predict gives the predicted variable (y_hat).
This is the same for reg:linear / binary:logistic etc. The only difference is that reg:linear builds trees to Min(RMSE(y, y_hat)), while rank:pairwise build trees to Max(Map(Rank(y), Rank(y_hat))). However, output is always y_hat.
Depending on the values of your dependent variables, output can be anything. But I typically expect output to be much smaller in variance vs the dependent variable. This is usually the case as it is not necessary to fit extreme data values, the tree just needs to produce predictors that are large/small enough to be ranked first/last in the group.
Is there any announcement about when Google will launch continuous prediction. Currently is there any trick to predict stock prices using Google's prediction API?
They announced continuous output for v1.1 today, along with the much requested multiple category output:
training data submitted with only numbers to v1.1 in the leftmost column will be treated as a continuous output problem (unlike v1)
...
numerical values in the leftmost column of all rows will
automatically return regression values. if you intend to do classification,
we recommend encasing those values within double quotes. For example, 5
indicates a regression value of 5 while "5" indicates a category labeled
"5."
Yes, it can be written.
The important factor affecting the accuracy of your predictions would be the input parameters that you give.
So, try to vary the input training data between different moving averages or other statistical figures and see what comes close at predicting the action to be taken (Buy/Sell).