Mean or Sum of objective function matters in optimization? - optimization

Question

I'm using RcppEnsmallen to estimate quantile regression models. I think I have set up everything right, but problems have been constantly emerging. I've tried two strategies:
Strategy 1: in the objective function, return sum of all errors. The corresponding gradient setup is similar. I tried 0.1 quantile in my example. However, occasionally I can get correct answers. Most of the time, it returns NaN.
Strategy 2: The objective function changed to mean of the sum of all errors. The gradient setup is also use the mean version. However, in this scenario, it returns quite different values each time, far from the correct answers.
I was wondering where the problem might be? Thanks.
#include <RcppEnsmallen.h>
// [[Rcpp::depends(RcppEnsmallen)]]
class QuantileRegressionFunction
{
public:
// Construct the QuantileRegressFunction with the given data.
QuantileRegressionFunction(const arma::mat& X,
const arma::colvec& y
) : X(X),y(y){}
// Define the objective function.
double Evaluate(const arma::mat& beta){
// arma::vec residual1 = y - X * beta;
return arma::sum( (y - X * beta) % ( 0.1 - arma::ones(y.n_rows) % ( (y - X * beta) <0 )) );
}
void Gradient(const arma::mat& beta, arma::mat& gradient)
{
gradient = -X.t() * ( 0.1 - arma::ones(y.n_rows) % ( (y - X * beta) <0 ) ) ;
}
private:
const arma::mat& X;
const arma::vec& y;
};
// [[Rcpp::export]]
arma::mat qr_reg(const arma::mat&X, const arma::vec&y){
QuantileRegressionFunction qrf(X,y);
ens::L_BFGS lbfgs;
lbfgs.MaxIterations()= 100000 ;
lbfgs.MaxLineSearchTrials() = 10000;
//lbfgs.ArmijoConstant() = 1e-10;
arma::mat beta(X.n_cols, 1 , arma::fill::randn);
lbfgs.Optimize(qrf,beta);
return beta;
}