I've trained a classification model in 0.9:
param = {
'objective': 'multi:softprob',
'num_class': 9,
'booster': 'dart',
'eta': 0.3,
'gamma': 0,
'max_depth': 6,
'alpha': 0,
'lambda': 1,
'colsample_bylevel':0.8,
'colsample_bynode': 0.8,
'colsample_bytree': 0.8,
'normalize_type': 'tree',
'rate_drop': 1.0,
'min_child_weight': 5,
'subsample': 0.5,
'num_parallel_tree': 1,
'tree_method': 'approx'
}
model = xgb.train(param, D_train, num_boost_round=1)
model.save_model('./model.bst')
Here is the sample training data:
{f1:1, f2:1, label:"1"}
Both f1 and f2 are integers.
During prediction, the results are nondeterministic with the same input. Sometimes (~1 out of 10 times) it gives equal probability for every output class.
This issue is gone when switching to XGB 1.0: Use XGB 1.0 to make predictions on a model trained in 0.9.
imported_model = xgb.Booster(model_file='./model.bst')
encoded=[[[0, 0]]
prediction_input = xgb.DMatrix(
np.array(encoded).reshape(2, -1), missing=None)
for i in range(1000):
outputs = imported_model.predict(prediction_input)
print(outputs)
Does anyone know the root cause?
Related
We are currently facing a problem with a Prophet model which stops unexpectedly. We've used optuna as our HyperParameter model searching framework, and gave 400 trials per each TimeSeries (we are traininig Prophet on ~ 1000 Timeseries, 1 by 1, not all at once). For cross validation, we've used the methode from the Prophet class called cross_validation.
Here is the code we've used:
def get_best_hp_prophet_cv(df_for_cv, weeks_to_forecast):
logging.getLogger('prophet').setLevel(logging.ERROR)
logging.getLogger('fbprophet').setLevel(logging.ERROR)
# print('DF entered for search: ', df_for_cv)
cutoffs = generate_cutoffs(df_for_cv, weeks_to_forecast)
def objective(trial):
# print(cutoffs)
# print(df.tail(30))
param_grid = {
"changepoint_prior_scale": trial.suggest_categorical(
"changepoint_prior_scale", [0.001, 0.01, 0.1, 0.5, 0.05, 0.8, 0.9]
),
"seasonality_prior_scale": trial.suggest_categorical(
"seasonality_prior_scale", [0.01, 0.05, 0.1, 0.5, 1.0, 10]
),
"seasonality_mode": trial.suggest_categorical(
"seasonality_mode", ["multiplicative", "additive"]
),
"growth": trial.suggest_categorical("growth", ["linear"]),
"yearly_seasonality": trial.suggest_categorical(
"yearly_seasonality", [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16]
),
"daily_seasonality": trial.suggest_categorical("daily_seasonality",[False]),
"weekly_seasonality": trial.suggest_categorical("weekly_seasonality",[False]),
"uncertainty_samples": trial.suggest_categorical("uncertainty_samples",[0]),
}
prior_scale_month = trial.suggest_categorical('prior_scale_month', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_week_num = trial.suggest_categorical('prior_scale_week_num', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_avg_month_qty_over_df = trial.suggest_categorical('prior_scale_avg_month_qty_over_df', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
prior_scale_avg_week_nr_qty_over_df = trial.suggest_categorical('prior_scale_avg_week_nr_qty_over_df', [0.001, 0.01, 0.1, 0.5, 0.05, 0.9])
# I ve used this only for testing to see if everything works fine
# param_grid = {
# 'changepoint_prior_scale': trial.suggest_categorical('changepoint_prior_scale', [0.001]),
# 'seasonality_prior_scale': trial.suggest_categorical('seasonality_prior_scale',[0.01, 0.1]),
# 'seasonality_mode' : trial.suggest_categorical('seasonality_mode',['additive']),
# 'growth': trial.suggest_categorical('growth',['linear']),
# 'yearly_seasonality': trial.suggest_categorical('yearly_seasonality',[14,15]),
# 'holidays_prior_scale' : trial.suggest_categorical('holidays_prior_scale',[10])
# }
# all_params = [dict(zip(param_grid.keys(), v)) for v in itertools.product(*param_grid.values())]
# mses = [] # Store the MSEs for each params here
# Use cross validation to evaluate all parameters
# for params in all_params:
m = Prophet(**param_grid)
m.add_regressor('month', prior_scale = prior_scale_month)
m.add_regressor('week_num', prior_scale = prior_scale_week_num)
m.add_regressor('avg_month_qty_over_df', prior_scale = prior_scale_avg_month_qty_over_df)
m.add_regressor('avg_week_nr_qty_over_df', prior_scale = prior_scale_avg_week_nr_qty_over_df)
m.fit(df_for_cv)
df_cv = cross_validation(
m, cutoffs=cutoffs, horizon="{} days".format(weeks_to_forecast*7), parallel="processes"
)
df_p = performance_metrics(df_cv, rolling_window=1)
return df_p["mse"].values[0]
# Find the best parameters
optuna_prophet = optuna.create_study(
direction="minimize", sampler=TPESampler(seed=321)
)
# * n_trials optuna hyperparameter.
#optuna_prophet.optimize(objective, n_trials=400)
optuna_prophet.optimize(objective, n_trials=1)
prophet_trial = optuna_prophet.best_trial
prophet_trial_params = prophet_trial.params
list_of_variables_outside_the_param_grid = ['prior_scale_month','prior_scale_week_num','prior_scale_avg_month_qty_over_df','prior_scale_avg_week_nr_qty_over_df']
params_outside_the_param_grid={}
param_grid = {}
for param_name in prophet_trial_params.keys():
if param_name in list_of_variables_outside_the_param_grid:
params_outside_the_param_grid.update({param_name : prophet_trial_params[param_name]})
else:
param_grid.update({param_name : prophet_trial_params[param_name]})
return param_grid, params_outside_the_param_grid
We've used Prophet before, but on less Timeseries (around 20) and with even more nr_of_trials in Optuna and never faced this issue before. Does anybody knows why is this happening? Has anybody a workaround? It always stops at the begging of the 54 trial from Optuna Search.
We've tried to run with only 1 trial on all 1000 Timeseries and it worked end - 2 - end, but still we can't figure it out why it stops on the 54th trial ?
Versions:
Optuna ---- 3.0.3.
Prophet ---- 1.1
I am trying to figure how to tune my hyperparameter through RandomizedSearchCV with an XGBRanker model.
I could split the data into groups, feed it into the model and make predictions. However I am not sure how to set up the Search object, namely 2 specific things - how to inform it about the groups and also what kind of score I need to supply.
model = xg.XGBRanker(
tree_method='exact',
booster='gbtree',
objective='rank:pairwise',
random_state=42,
learning_rate=0.06,
max_depth=5,
n_estimators=700,
subsample=0.75,
#colsample_bytree=0.9,
#subsample=0.75
min_child_weight=0.06
)
model.fit(x_train, y_train, group=train_groups, verbose=True)
This works fine.
This is where I need some help
param_dist = {'n_estimators': stats.randint(40, 1000),
'learning_rate': stats.uniform(0.01, 0.59),
'subsample': stats.uniform(0.3, 0.6),
'max_depth': [3, 4, 5, 6, 7, 8, 9],
'colsample_bytree': stats.uniform(0.5, 0.4),
'min_child_weight': [0.05, 0.1, 0.02]
}
clf = RandomizedSearchCV(model,
param_distributions=param_dist,
cv=5,
n_iter=5,
scoring=???, #
error_score=0,
verbose=3,
n_jobs=-1)
#also what about the groups?
i had tried something similar. for scoring however i used the ndcg_scorer from sklearn.
i added
scoring = sklearn.metrics.make_scorer(sklearn.metrics.ndcg_score, greater_is_better=True)
for groups u can add to the fit_params in RandomizedSearchCV.
fit_params = {"model__groups": group}
clf = RandomizedSearchCV(model,
param_distributions=param_dist,
cv=5,
n_iter=5,
scoring=scoring,
error_score=0,
verbose=3,
n_jobs=-1,fit_params = fit_params)
I am new in tensorflow. I use tf.feature_column to finish feature_engineering task, including crossed_column operation. After a simple LR model, I can get the feature_weight for every feature, but how do I know the feature's name? For example:
my_features = []
weight = IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='weight', vocabulary_list=(2, 3, 0, 1), dtype=tf.int64, default_value=-1, num_oov_buckets=0))
my_features.append(weight)
vol = IndicatorColumn(categorical_column=VocabularyListCategoricalColumn(key='vol', vocabulary_list=(3, 4, 1, 2), dtype=tf.int64, default_value=-1, num_oov_buckets=0))
my_features.append(vol)
weight_X_vol = tf.feature_column.crossed_column( [feature_weight, feature_vol], has_bucket_size = 50)
my_features.append(weight_X_vol)
in feature_weight, it's one_hot feature size is 4(weight0, 1, 2, 3)
in feature_vol, it's one_hot feature size is 4(vol0, 1, 2, 3)
in feature_weight_X_vol, it's one_hot feature size is 16(weight0_vol0, weight0_vol1, ...., weight3_vol3)
so my features_size is 4 + 4 + 16 = 24
After fit a simple LR model, I can get the 24 feature_weight by model.layers[1].get_weights(), but how can I build a dict to map feature_name(key) to the feature_weight(value)? For example:
my model is :
my_model = tf.keras.Sequential([
tf.keras.layers.DenseFeatures(my_features),
tf.keras.layers.Dense(1, activatio="sigmoid")
])
feature_weight = my_model.layers[0].get_weights()
my weight is :
faeture_weight = [0.1, 0.2, 0.1, ..., 0.8, 0.4. 0.3] //lengh is 24
my feature_name is :
feature_name = [weight0, weight1, weight2, ..., vol0, vol1, ..., weight0_vol0, weight3_vol3] // lengh is 24
How can I map the key to the value? I just want to know every feature's weight for debuging.
I like to evaluate my object detection model with mAP (mean average precision). In https://github.com/tensorflow/models/tree/master/research/object_detection/utils/ there is object_detection_evaluation.py that I want to use.
I use following for the groundtruth boxes:
pascal_evaluator = object_detection_evaluation.PascalDetectionEvaluator(
categories, matching_iou_threshold=0.1)
groundtruth_boxes = np.array([[10, 10, 11, 11]], dtype=float)
groundtruth_class_labels = np.array([1], dtype=int)
groundtruth_is_difficult_list = np.array([False], dtype=bool)
pascal_evaluator.add_single_ground_truth_image_info(
'img2',
{
standard_fields.InputDataFields.groundtruth_boxes: groundtruth_boxes,
standard_fields.InputDataFields.groundtruth_classes: groundtruth_class_labels,
standard_fields.InputDataFields.groundtruth_difficult: groundtruth_is_difficult_list
}
)
and this for the prediction Boxes:
# Add detections
image_key = 'img2'
detected_boxes = np.array(
[ [100, 100, 220, 220], [10, 10, 11, 11]],
dtype=float)
detected_class_labels = np.array([1,1], dtype=int)
detected_scores = np.array([0.8, 0.9], dtype=float)
pascal_evaluator.add_single_detected_image_info(image_key, {
standard_fields.DetectionResultFields.detection_boxes:
detected_boxes,
standard_fields.DetectionResultFields.detection_scores:
detected_scores,
standard_fields.DetectionResultFields.detection_classes:
detected_class_labels
})
I print the results with
metrics = pascal_evaluator.evaluate()
print(metrics)
And my Question:
if I use this prediction Boxes [100, 100, 220, 220], [10, 10, 11, 11] the result is:
{'PASCAL/Precision/mAP#0.1IOU': 1.0,
'PASCAL/PerformanceByCategory/AP#0.1IOU/face': 1.0}
If I use [10, 10, 11, 11], [100, 100, 220, 220] (other Box sequence)
I get following result:
{'PASCAL/Precision/mAP#0.1IOU': 0.5,
'PASCAL/PerformanceByCategory/AP#0.1IOU/face': 0.5}
Why is that so? Or is it bug?
Cheers Michael
Although you are not so clear about it I think I found the error in your code. You mentioned you get different results for different order of bounding boxes. This seems peculiar and if true then it was surely a bug.
But, since I tested the code myself, you probably did not change the corresponding scores (detected_scores = np.array([0.8, 0.9], dtype=float)) to the bounding boxes. But this way you changes also the problem not just the order of the bounding boxes. If you apply the correct bounding boxes the mAP remains the same in both cases:
{'PascalBoxes_Precision/mAP#0.5IOU': 1.0,
'PascalBoxes_PerformanceByCategory/AP#0.5IOU/person': 1.0}
Consider the following toy TensorFlow code. The fit method of LinearRegressor works properly and finds the right coefficients (i.e. y = x1 + x2), but evaluate (see the last print statement) hangs. Any idea what's wrong?
import tensorflow as tf
x1 = [1, 3, 4, 5, 1, 6, -1, -3]
x2 = [5, 2, 1, 5, 0, 2, 4, 2]
y = [6, 5,5, 10, 1, 8, 3, -1]
def train_fn():
return {'x1': tf.constant(x1), 'x2':tf.constant(x2)}, tf.constant(y)
features = [tf.contrib.layers.real_valued_column('x1', dimension=1),
tf.contrib.layers.real_valued_column('x2', dimension=1)]
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
estimator.fit(input_fn=train_fn, steps=10000)
for vn in estimator.get_variable_names():
print('variable name', vn, estimator.get_variable_value(vn))
print(estimator.evaluate(input_fn=train_fn))
estimator.evaluate() takes a parameter steps, which defaults to None, which is interpreted as "infinity". It therefore never ends.
To make it end, pass steps=1 explicitly:
estimator.evaluate(input_fn=your_input_fn, steps=1)