Quantile regressing using RcppEnsmallen diverge - optimization

Question

I am trying RcppEnsmallen to estimate quantile regression models. I noticed that the results seemed not convergent. Every time it pops up different results. I set up its first order condition in my code. I guess the divergence may be owing to the fact that the objective function is not differentiable at zero. I am not sure about that.
Here the code
#include <RcppEnsmallen.h>
// [[Rcpp::depends(RcppEnsmallen)]]
class QuantileRegressionFunction
{
public:
// Construct the QuantileRegressFunction with the given data.
QuantileRegressionFunction(const arma::mat& X,
const arma::colvec& y,
const double tau
) : X(X),y(y),tau(tau){}
// Define the objective function.
double EvaluateWithGradient(const arma::mat& beta, arma::mat& gradient ){
const arma::mat fit = y - X * beta;
const arma::vec eval =0.5* arma::abs(fit) + (tau - 0.5 )*fit;
const arma::vec v= tau - arma::ones<arma::vec>(y.n_rows) % ( fit < 0 ) ;
gradient = - (X.t() * v)*(1/X.n_rows) ;
return arma::accu( (1/y.n_rows) *eval ) ;
}
private:
const arma::mat& X;
const arma::vec& y;
const double tau;
};
// [[Rcpp::export]]
arma::mat qr_reg(const arma::mat& X, const arma::vec& y,const double& tau){
QuantileRegressionFunction qrf(X,y,tau);
ens::L_BFGS lbfgs;
lbfgs.MaxIterations()= 1000 ;
//lbfgs.MaxLineSearchTrials() = 10000;
//lbfgs.ArmijoConstant() = 1e-18;
arma::mat beta(X.n_cols, 1 , arma::fill::randn);
lbfgs.Optimize(qrf,beta);
return beta;
}
Here is a simple simulation exercise:
n <- 1000
beta <- c(-2, 1.5, 3, 8.2, 6.6)
p <- length(beta)
X <- cbind(1, matrix(rnorm(n), ncol = p - 1))
y <- X %*% beta + rnorm(n / (p - 1))
qr_reg(X,y,0.1)
Every time qr_reg(X,y,.1) report divergent estimates.