Hyperparameter tuning with XGBRanker - xgboost

I am trying to figure how to tune my hyperparameter through RandomizedSearchCV with an XGBRanker model.
I could split the data into groups, feed it into the model and make predictions. However I am not sure how to set up the Search object, namely 2 specific things - how to inform it about the groups and also what kind of score I need to supply.
model = xg.XGBRanker(
tree_method='exact',
booster='gbtree',
objective='rank:pairwise',
random_state=42,
learning_rate=0.06,
max_depth=5,
n_estimators=700,
subsample=0.75,
#colsample_bytree=0.9,
#subsample=0.75
min_child_weight=0.06
)
model.fit(x_train, y_train, group=train_groups, verbose=True)
This works fine.
This is where I need some help
param_dist = {'n_estimators': stats.randint(40, 1000),
'learning_rate': stats.uniform(0.01, 0.59),
'subsample': stats.uniform(0.3, 0.6),
'max_depth': [3, 4, 5, 6, 7, 8, 9],
'colsample_bytree': stats.uniform(0.5, 0.4),
'min_child_weight': [0.05, 0.1, 0.02]
}
clf = RandomizedSearchCV(model,
param_distributions=param_dist,
cv=5,
n_iter=5,
scoring=???, #
error_score=0,
verbose=3,
n_jobs=-1)
#also what about the groups?

i had tried something similar. for scoring however i used the ndcg_scorer from sklearn.
i added
scoring = sklearn.metrics.make_scorer(sklearn.metrics.ndcg_score, greater_is_better=True)
for groups u can add to the fit_params in RandomizedSearchCV.
fit_params = {"model__groups": group}
clf = RandomizedSearchCV(model,
param_distributions=param_dist,
cv=5,
n_iter=5,
scoring=scoring,
error_score=0,
verbose=3,
n_jobs=-1,fit_params = fit_params)

Related

Keras Sequential with multiple inputs

Given 3 array as input to the network, it should learn what links data in 1st array, 2nd array, and 3rd array.
In particular:
1st array contains integer numbers (eg.: 2, 3, 5, 6, 7)
2nd array contains integer numbers (eg.: 3, 2, 4, 6, 2)
3rd array contains integer numbers that are the results of an operation done between data in 1st and 2nd array (eg.: 6, 6, 20, 36, 14).
As you can see from the example data here above, the operation done is a multiplication so the network should learn this, giving:
model.predict(11,2) = 22.
Here's the code I've used:
import logging
import numpy as np
import tensorflow as tf
primo = np.array([2, 3, 5, 6, 7])
secondo = np.array([3, 2, 4, 6, 2])
risu = np.array([6, 6, 20, 36, 14])
l0 = tf.keras.layers.Dense(units=1, input_shape=[1])
model = tf.keras.Sequential([l0])
input1 = tf.keras.layers.Input(shape=(1, ), name="Pri")
input2 = tf.keras.layers.Input(shape=(1, ), name="Sec")
merged = tf.keras.layers.Concatenate(axis=1)([input1, input2])
dense1 = tf.keras.layers.Dense(
2,
input_dim=2,
activation=tf.keras.activations.sigmoid,
use_bias=True)(merged)
output = tf.keras.layers.Dense(
1,
activation=tf.keras.activations.relu,
use_bias=True)(dense1)
model = tf.keras.models.Model([input1, input2], output)
model.compile(
loss="mean_squared_error",
optimizer=tf.keras.optimizers.Adam(0.1))
model.fit([primo, secondo], risu, epochs=500, verbose = False, batch_size=16)
print(model.predict(11, 2))
My questions are:
is it correct to concatenate the 2 input as I did? I don't understand if concatenating in such a way the network understand that input1 and input2 are 2 different data
I'm not able to make the model.predict() working, every attempt result in an error
Your model has two inputs, each with shape (None,1), so you need to use np.expand_dims:
print(model.predict([np.expand_dims(np.array(11), 0), np.expand_dims(np.array(2), 0)]))
Output:
[[20.316557]]

XGBoost gives nondeterministic predictions in version <1.0. for INT input

I've trained a classification model in 0.9:
param = {
'objective': 'multi:softprob',
'num_class': 9,
'booster': 'dart',
'eta': 0.3,
'gamma': 0,
'max_depth': 6,
'alpha': 0,
'lambda': 1,
'colsample_bylevel':0.8,
'colsample_bynode': 0.8,
'colsample_bytree': 0.8,
'normalize_type': 'tree',
'rate_drop': 1.0,
'min_child_weight': 5,
'subsample': 0.5,
'num_parallel_tree': 1,
'tree_method': 'approx'
}
model = xgb.train(param, D_train, num_boost_round=1)
model.save_model('./model.bst')
Here is the sample training data:
{f1:1, f2:1, label:"1"}
Both f1 and f2 are integers.
During prediction, the results are nondeterministic with the same input. Sometimes (~1 out of 10 times) it gives equal probability for every output class.
This issue is gone when switching to XGB 1.0: Use XGB 1.0 to make predictions on a model trained in 0.9.
imported_model = xgb.Booster(model_file='./model.bst')
encoded=[[[0, 0]]
prediction_input = xgb.DMatrix(
np.array(encoded).reshape(2, -1), missing=None)
for i in range(1000):
outputs = imported_model.predict(prediction_input)
print(outputs)
Does anyone know the root cause?

Tensorflow object detection evaluation

I like to evaluate my object detection model with mAP (mean average precision). In https://github.com/tensorflow/models/tree/master/research/object_detection/utils/ there is object_detection_evaluation.py that I want to use.
I use following for the groundtruth boxes:
pascal_evaluator = object_detection_evaluation.PascalDetectionEvaluator(
categories, matching_iou_threshold=0.1)
groundtruth_boxes = np.array([[10, 10, 11, 11]], dtype=float)
groundtruth_class_labels = np.array([1], dtype=int)
groundtruth_is_difficult_list = np.array([False], dtype=bool)
pascal_evaluator.add_single_ground_truth_image_info(
'img2',
{
standard_fields.InputDataFields.groundtruth_boxes: groundtruth_boxes,
standard_fields.InputDataFields.groundtruth_classes: groundtruth_class_labels,
standard_fields.InputDataFields.groundtruth_difficult: groundtruth_is_difficult_list
}
)
and this for the prediction Boxes:
# Add detections
image_key = 'img2'
detected_boxes = np.array(
[ [100, 100, 220, 220], [10, 10, 11, 11]],
dtype=float)
detected_class_labels = np.array([1,1], dtype=int)
detected_scores = np.array([0.8, 0.9], dtype=float)
pascal_evaluator.add_single_detected_image_info(image_key, {
standard_fields.DetectionResultFields.detection_boxes:
detected_boxes,
standard_fields.DetectionResultFields.detection_scores:
detected_scores,
standard_fields.DetectionResultFields.detection_classes:
detected_class_labels
})
I print the results with
metrics = pascal_evaluator.evaluate()
print(metrics)
And my Question:
if I use this prediction Boxes [100, 100, 220, 220], [10, 10, 11, 11] the result is:
{'PASCAL/Precision/mAP#0.1IOU': 1.0,
'PASCAL/PerformanceByCategory/AP#0.1IOU/face': 1.0}
If I use [10, 10, 11, 11], [100, 100, 220, 220] (other Box sequence)
I get following result:
{'PASCAL/Precision/mAP#0.1IOU': 0.5,
'PASCAL/PerformanceByCategory/AP#0.1IOU/face': 0.5}
Why is that so? Or is it bug?
Cheers Michael
Although you are not so clear about it I think I found the error in your code. You mentioned you get different results for different order of bounding boxes. This seems peculiar and if true then it was surely a bug.
But, since I tested the code myself, you probably did not change the corresponding scores (detected_scores = np.array([0.8, 0.9], dtype=float)) to the bounding boxes. But this way you changes also the problem not just the order of the bounding boxes. If you apply the correct bounding boxes the mAP remains the same in both cases:
{'PascalBoxes_Precision/mAP#0.5IOU': 1.0,
'PascalBoxes_PerformanceByCategory/AP#0.5IOU/person': 1.0}

Tensorflow LinearRegressor evaluate method hangs

Consider the following toy TensorFlow code. The fit method of LinearRegressor works properly and finds the right coefficients (i.e. y = x1 + x2), but evaluate (see the last print statement) hangs. Any idea what's wrong?
import tensorflow as tf
x1 = [1, 3, 4, 5, 1, 6, -1, -3]
x2 = [5, 2, 1, 5, 0, 2, 4, 2]
y = [6, 5,5, 10, 1, 8, 3, -1]
def train_fn():
return {'x1': tf.constant(x1), 'x2':tf.constant(x2)}, tf.constant(y)
features = [tf.contrib.layers.real_valued_column('x1', dimension=1),
tf.contrib.layers.real_valued_column('x2', dimension=1)]
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
estimator.fit(input_fn=train_fn, steps=10000)
for vn in estimator.get_variable_names():
print('variable name', vn, estimator.get_variable_value(vn))
print(estimator.evaluate(input_fn=train_fn))
estimator.evaluate() takes a parameter steps, which defaults to None, which is interpreted as "infinity". It therefore never ends.
To make it end, pass steps=1 explicitly:
estimator.evaluate(input_fn=your_input_fn, steps=1)

Tensorflow apply_gradients ValueError

I get the following error on second to last line of code: Not sure how to proceed. Can anyone give me some insight.
ValueError: Tensor("Variable_20:0", shape=(8, 8, 4, 32), dtype=float32_ref) must be from the same graph as Tensor("Variable_20/RMSProp_1:0", shape=(8, 8, 4, 32), dtype=float32_ref).
The code is as follows:
optimizer = tf.train.RMSPropOptimizer(0.00025, 0.95, 0.95, 0.01)
readout = tf.reduce_mean(tf.reduce_sum(tf.mul(l_readout,a), reduction_indices=1))
cost = tf.reduce_mean(tf.square(tf.sub(y,readout)))
grads = optimizer.compute_gradients(cost, localW)
grad_vals = sess.run([g for (g,v) in grads], feed_dict = {a: val_a, y: val_y, s: val_s})
placeholder_gradients = []
for var in localW:
placeholder_gradients.append( (tf.placeholder('float',shape=var.get_shape()) , var) )
feed_dict = {}
for i in range(len(placeholder_gradients)):
feed_dict[placeholder_gradients[i]] = grad_vals[i]
apply_gradients = optimizer.apply_gradients(placeholder_gradients) #ERROR LINE
apply_gradients.run(feed_dict=feed_dict)
This maybe related to using a thread which is not shown in the example. I will withdraw this question until I look further into how to use threads with the same graph.