Extremely new to Julia, so please pardon any obvious oversights on my end
I am trying to estimate a piecewise likelihood function through optimization. I have the code functional in R, but have begun translating it to Julia in the hopes of faster estimation, for eventual bootstrapping
Here is the current block of code that I am trying (v and x are already as 1000x1 vectors elsewhere defined elsewhere):
function est(a,b)
function pwll(v,x)
if v>4
ILL=pdf(Poisson(exp(a+b*x)), v)
elseif v==4
ILL=pdf(Poisson(exp(a+b*x)), 4)+pdf(Poisson(exp(a+b*x)),3)+pdf(Poisson(exp(a+b*x)),2)
else v==0
ILL=pdf(Poisson(exp(a+b*x)), 1)+pdf(Poisson(exp(a+b*x)), 0)
end
return(ILL)
end
ILL=pwll.(v, x)
function fixILL(x)
if x==0
x=0.00000000000000001
else
x=x
end
end
ILL=fixILL.(ILL)
LILL=log10.(ILL)
LL=-1*LILL
return(sum(LL))
end
using Optim
params0=[1,1]
optimize(est, params0)
And the error message(s) I am getting are:
ERROR: InexactError: Int64(NaN)
Stacktrace:
[1] Int64(x::Float64)
# Base ./float.jl:788
[2] x_of_nans(x::Vector{Int64}, Tf::Type{Int64}) (repeats 2 times)
# NLSolversBase ~/.julia/packages/NLSolversBase/kavn7/src/NLSolversBase.jl:60
[3] NonDifferentiable(f::Function, x::Vector{Int64}, F::Int64; inplace::Bool)
# NLSolversBase ~/.julia/packages/NLSolversBase/kavn7/src/objective_types/nondifferentiable.jl:11
[4] NonDifferentiable(f::Function, x::Vector{Int64}, F::Int64)
# NLSolversBase ~/.julia/packages/NLSolversBase/kavn7/src/objective_types/nondifferentiable.jl:10
[5] promote_objtype(method::NelderMead{Optim.AffineSimplexer, Optim.AdaptiveParameters}, x::Vector{Int64}, autodiff::Symbol, inplace::Bool, args::Function)
# Optim ~/.julia/packages/Optim/tP8PJ/src/multivariate/optimize/interface.jl:63
[6] optimize(f::Function, initial_x::Vector{Int64}; inplace::Bool, autodiff::Symbol, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
# Optim ~/.julia/packages/Optim/tP8PJ/src/multivariate/optimize/interface.jl:86
[7] optimize(f::Function, initial_x::Vector{Int64})
# Optim ~/.julia/packages/Optim/tP8PJ/src/multivariate/optimize/interface.jl:83
[8] top-level scope
# ~/Documents/Projects/ki_new/peicewise_ll.jl:120
I understand that it seems the error is coming from the function to be optimized being non-differentiable. A fairly direct translation works well in R, using the built in optim() function.
Can anyone provide any insight?
I have tried the above code displayed above, with multiple variations. The function to be optimized is functional, I am struggling with the optimization (the issues of which may stem from function being inefficiently written)
Here's an adapted version of your code which produces a solution:
using Distributions, Optim
function pwll(v, x, a, b)
d = Poisson(exp(a+b*x))
if v > 4
return pdf(d, v)
elseif v == 4
return pdf(d, 4) + pdf(d, 3) + pdf(d, 2)
else
return pdf(d, 1) + pdf(d, 0)
end
end
fixILL(x) = iszero(x) ? 1e-17 : x
est(a, b, v, x) = sum(-1 .* log10.(fixILL.(pwll.(v, x, a, b))))
v = 4; x = 0.5 # Defining these here as they are not given in your post
obj(input; v = v, x = x) = est(input[1], input[2], v, x)
optimize(obj, [1.0, 1.0])
I have no idea whether this is correct of course, check this against some sort of known result if you can.
Related
There is a error in my code, can anyone help me please?
My code:
function funP(u, τ::Float64)
w = (τ*max(u, 0)) + ((1-τ)*max(-u, 0))
return w
end
τ = 0.2
modelquant = Model(with_optimizer(OSQP.Optimizer))
#variable(modelquant, β[i=0:1])
#variable(modelquant, erro[1:T])
#constraint(modelquant,[i=1:T], erro[i] >= contratos[i] - β[0] - β[1]*spot[i])
#constraint(modelquant,[i=1:T], erro[i] >= -contratos[i] + β[0] + β[1]*spot[i])
#objective(modelquant, Min, sum(funP(erro[i], τ) for i=1:T))
optimize!(modelquant)
objective_value(modelquant)
𝐁 = JuMP.value.(β)
The error is:
julia> #objective(modelquant, Min, sum(funP(erro[i], τ) for i=1:T))
ERROR: MethodError: no method matching isless(::Int64, ::VariableRef)
Closest candidates are:
isless(::Missing, ::Any) at missing.jl:87
isless(::Real, ::AbstractFloat) at operators.jl:166
isless(::Integer, ::ForwardDiff.Dual{Ty,V,N} where N where V) where Ty at C:\Users\Fernanda.julia\packages\ForwardDiff\kU1ce\src\dual.jl:140
...
Stacktrace:
[1] max(::VariableRef, ::Int64) at .\operators.jl:417
[2] funP(::VariableRef, ::Float64) at .\REPL[8]:2
[3] macro expansion at C:\Users\Fernanda.julia\packages\MutableArithmetics\bPWR4\src\rewrite.jl:276 [inlined]
[4] macro expansion at C:\Users\Fernanda.julia\packages\JuMP\qhoVb\src\macros.jl:830 [inlined]
[5] top-level scope at .\REPL[20]:1
Thank you so much!
You need to re-engineer your model to replace the max function with a binary variable.
In your case the code will look like this (check for typos):
#variable(modelquant, erro_neg[1:T], Bin)
#variable(modelquant, erro_pos[1:T], Bin)
#constraint(modelquant,for i=1:T,erro_neg[i]+erro_pos[i]==1)
#constraint(modelquant,for i=1:T,erro[i]*erro_pos[i] >= 0)
#constraint(modelquant,for i=1:T,erro[i]*erro_neg[i] <= 0)
#objective(modelquant, Min, sum( τ*erro_neg[i]*erro[i]+ (1-τ)*erro[i]*erro_npos[i] for i=1:T))
Please note that in my version you could actually safely remove the Bin condition from erro_neg and erro_pos and the model will still work (you need to test empirically what your solver prefers)
I am working on parameterizing b-splines knots and control points with deep learning. To do this, I have a model which outputs the knots and control points, and I want to let the model learn by implementing a custom loss function which evaluates the b-spline and then calculates the mean squared error against the actual curve. Hopefully, the model will learn to output the correct knots and control points.
However, Keras custom loss function requires all statements to be wrapped in backend functions. From scipy B-spline docs, this is how the function will look like normally.
def B(x, k, i, t):
if k == 0:
return 1.0 if t[i] <= x < t[i+1] else 0.0
if t[i+k] == t[i]:
c1 = 0.0
else:
c1 = (x - t[i])/(t[i+k] - t[i]) * B(x, k-1, i, t)
if t[i+k+1] == t[i+1]:
c2 = 0.0
else:
c2 = (t[i+k+1] - x)/(t[i+k+1] - t[i+1]) * B(x, k-1, i+1, t)
return c1 + c2
def bspline(x, t, c, k):
n = len(t) - k - 1
assert (n >= k+1) and (len(c) >= n)
return sum(c[i] * B(x, k, i, t) for i in range(n))
I know that for simple statements such as summation, I can use K.sum, and for if-else statements, I can use K.switch. However, I do not know how to implement for-loops in Keras. Is there a straight forward way to compute B-spline curves in Keras? (converting the above code into Keras backend)
I am trying to solve a differential equation using scipy.integrate.solve_ivp
L*Q'' + R*Q' + (1/C)*Q = E(t), E(t) = 230*sin(50*t)
for Q(t) and Q'(t)
C = 0.0014 #F
dQ_0 = 2.6 #A
L = 1.8 #H
n = 575 #/
Q_0 = 1e-06 #C
R = 43 #Ohm
t_f = 2.8 #s
import numpy as np
from scipy.integrate import solve_ivp
t = np.linspace(0, t_f, n)
def E(x):
return 230*np.sin(50*x)
y = E(y)
def Q(t, y, R, L, C):
return (y - L*Q'' - R*Q')*C
init_cond = [Q_0, dQ_0]
y_ivp = solve_ivp(Q, t_span=(0, t_f), y0=init_cond)
I am only trying to understand how to correctly define a function that is passed as an argument 'fun' in scipy.integrate.solve_ivp
The answer below is no longer valid, solve_ode has since implemented the args parameter similar to odeint. So indeed
y_ivp = solve_ivp(Q, t_span=(0, t_f), y0=init_cond, args=(R, L, C))
is now valid (do not forget to set appropriate error tolerances, or at least check that the default values arol=1e-3, rtol=1e-6 are appropriate).
Always available was the use of semi-global variables in a closure or lambda expression
y_ivp = solve_ivp(lambda t,y: Q(t,y,R, L, C), t_span=(0, t_f), y0=init_cond, args=(R, L, C))
(obsolete part) solve_ivp has no parameter passing mechanism, so treat the parameters as global variables. You are formulating an ODE for Q, as it is a second order ODE, the state also contains the first derivative, as you somehow recognized in the composition of the initial state. The ODE function then needs to produce the derivative values at a given state. Identify Q(t)=Q[0] and Q'(t)=Q[1], then
def Q_ode(t, Q):
return [ Q[1], (E(t) - R*Q[1] - (1/C)*Q[0])/L ]
I would continue to name the variables containing Q values with the letter Q.
I am using cvxpy to work on some simple portfolio optimisation problem. The only constraint I can't get my head around is the cardinality constraint for the number non-zero portfolio holdings. I tried two approaches, a MIP approach and a traditional convex one.
here is some dummy code for a working traditional example.
import numpy as np
import cvxpy as cvx
np.random.seed(12345)
n = 10
k = 6
mu = np.abs(np.random.randn(n, 1))
Sigma = np.random.randn(n, n)
Sigma = Sigma.T.dot(Sigma)
w = cvx.Variable(n)
ret = mu.T*w
risk = cvx.quad_form(w, Sigma)
objective = cvx.Maximize(ret - risk)
constraints = [cvx.sum_entries(w) == 1, w>= 0, cvx.sum_smallest(w, n-k) >= 0, cvx.sum_largest(w, k) <=1 ]
prob = cvx.Problem(objective, constraints)
prob.solve()
print prob.status
output = []
for i in range(len(w.value)):
output.append(round(w[i].value,2))
print 'Number of non-zero elements : ',sum(1 for i in output if i > 0)
I had the idea to use, sum_smallest and sum_largest (cvxpy manual) my thought was to constraint the smallest n-k entries to 0 and let my target range k sum up to one, I know I can't change the direction of the inequality in order to stay convex, but maybe anyone knows about a clever way of constraining the problem while still keeping it simple.
My second idea was to make this a mixed integer problem, s.th along the lines of
import numpy as np
import cvxpy as cvx
np.random.seed(12345)
n = 10
k = 6
mu = np.abs(np.random.randn(n, 1))
Sigma = np.random.randn(n, n)
Sigma = Sigma.T.dot(Sigma)
w = cvx.Variable(n)
binary = cvx.Bool(n)
integer = cvx.Int(n)
ret = mu.T*w
risk = cvx.quad_form(w, Sigma)
objective = cvx.Maximize(ret - risk)
constraints = [cvx.sum_entries(w) == 1, w>= 0, cvx.sum_entries(binary) == k ]
prob = cvx.Problem(objective, constraints)
prob.solve()
print prob.status
output = []
for i in range(len(w.value)):
output.append(round(w[i].value,2))
print sum(1 for i in output if i > 0)
for i in range(len(w.value)):
print round(binary[i].value,2)
print output
looking at my binary vector it seems to be doing the right thing but the sum_entries constraint doesn't work, looking into the binary vector values I noticed that 0 isn't 0 it's very small e.g xxe^-20 I assume this will mess things up. Anyone can give me any guidance if this is the right way to go? I can use the standard solvers, as well as Mosek if that helps. I would prefer to have a non MIP implementation as I understand this is a combinatorial problem and will get very slow for larger problems. Ultimately I would like to either constraint on exact number of target holdings or a range e.g. 20-30.
Also the documentation in cvxpy around MIP is very short. thanks
A bit chaotic, this question.
So first: this kind of cardinality-constraint is NP-hard. This means, you can't express it using cvxpy without using Integer-programming (or else it would implicate P=NP)!
That beeing said, it would have been nicer, if there would be a pure version of the code without trying to formulate this constraint. I just assume it's the first code without the sum_smallest and sum_largest constraints.
So let's tackle the MIP-approach:
Your code trying to do this makes no sense at all
You introduce some binary-vars, but they have no connection to any other variable at all (so a constraint on it's sum is useless)!
You introduce some integer-vars, but they don't have any use at all!
So here is a MIP-approach:
import numpy as np
import cvxpy as cvx
np.random.seed(12345)
n = 10
k = 6
mu = np.abs(np.random.randn(n, 1))
Sigma = np.random.randn(n, n)
Sigma = Sigma.T.dot(Sigma)
w = cvx.Variable(n)
ret = mu.T*w
risk = cvx.quad_form(w, Sigma)
objective = cvx.Maximize(ret - risk)
binary = cvx.Bool(n) # !!!
constraints = [cvx.sum_entries(w) == 1, w>= 0, w - binary <= 0., cvx.sum_entries(binary) == k] # !!!
prob = cvx.Problem(objective, constraints)
prob.solve(verbose=True)
print(prob.status)
output = []
for i in range(len(w.value)):
output.append(round(w[i].value,2))
print('Number of non-zero elements : ',sum(1 for i in output if i > 0))
So we just added some binary-variables and connected them to w to indicate if w is nonzero or not.
If w is nonzero:
w will be > 0 because of constraint w>= 0
binary needs to be 1, or else constraint w - binary <= 0. is not fulfilled
So it's just introducing these binaries and this one indicator-constraint.
Now the cvx.sum_entries(binary) == k does what it should do.
Be careful with the implication-direction we used here. It might be relevant when chaging the constraint on k (like <=).
Keep in mind, that the default MIP-solver is awful. I also fear that Mosek's interface (sub-optimal within cvxpy) won't solve this, but i might be wrong.
Edit: Your in-range can easily be formulated using two more indicators for:
(k >= a) <= ind_0
(k <= b) <= ind_1
and adding a constraint which equals a logical_and:
ind_0 + ind_1 >= 2
I've had a similar problem where my weights could be negative and did not need to sum to 1 (but still need to be bounded), so I've modified sascha's example to accommodate relaxing these constraints using the CVXpy absolute value function. This should allow for a more general approach to tackling cardinality constraints with MIP
import numpy as np
import cvxpy as cvx
np.random.seed(12345)
n = 10
k = 6
mu = np.abs(np.random.randn(n, 1))
Sigma = np.random.randn(n, n)
Sigma = Sigma.T.dot(Sigma)
w = cvx.Variable(n)
ret = mu.T*w
risk = cvx.quad_form(w, Sigma)
objective = cvx.Maximize(ret - risk)
binary = cvx.Variable(n,boolean=True) # !!!
maxabsw=2
constraints = [ w>= -maxabsw,w<=maxabsw, cvx.abs(w)/maxabsw - binary <= 0., cvx.sum(binary) == k] # !!!
prob = cvx.Problem(objective, constraints)
prob.solve(verbose=True)
print(prob.status)
output = []
for i in range(len(w.value)):
output.append(round(w[i].value,2))
print('Number of non-zero elements : ',sum(1 for i in output if i > 0))
I'm performing curve fitting with scipy.optimize.leastsq. E.g. for a gaussian:
def fitGaussian(x, y, init=[1.0,0.0,4.0,0.1]):
fitfunc = lambda p, x: p[0]*np.exp(-(x-p[1])**2/(2*p[2]**2))+p[3] # Target function
errfunc = lambda p, x, y: fitfunc(p, x) - y # Distance to the target function
final, success = scipy.optimize.leastsq(errfunc, init[:], args=(x, y))
return fitfunc, final
Now, I want to optionally fix the values of some of the parameters in the fit. I found that suggestions are to use a different package lmfit, which I want to avoid, or are very general, like here.
Since I need a solution which
works with numpy/scipy (no further packages etc.)
is independent of the parameters themselves,
is flexible, in which parameters are fixed or not,
I came up with the following, using a condition on each of the parameters:
def fitGaussian2(x, y, init=[1.0,0.0,4.0,0.1], fix = [False, False, False, False]):
fitfunc = lambda p, x: (p[0] if not fix[0] else init[0])*np.exp(-(x-(p[1] if not fix[1] else init[1]))**2/(2*(p[2] if not fix[2] else init[2])**2))+(p[3] if not fix[3] else init[3])
errfunc = lambda p, x, y: fitfunc(p, x) - y # Distance to the target function
final, success = scipy.optimize.leastsq(errfunc, init[:], args=(x, y))
return fitfunc, final
While this works fine, it's neither practical, nor beautiful.
So my question is: Are there better ways of performing curve fitting in scipy for fixed parameters? Or are there wrappers, which already include such parameter fixing?
Using scipy, there are no builtin options that I am aware of. You will always have to do a work-around like the one you already did.
If you are willing to use a wrapper package however, may I recommend my own symfit? This is a wrapper to scipy with readability and less boilerplate code as its core principles. In symfit, your problem would be solved as:
from symfit import parameters, variables, exp, Fit, Parameter
a, b, c, d = parameters('a, b, c, d')
x, y = variables('x, y')
model_dict = {y: a * exp(-(x - b)**2 / (2 * c**2)) + d}
fit = Fit(model_dict, x=xdata, y=ydata)
fit_result = fit.execute()
The line a, b, c, d = parameters('a, b, c, d') makes four Parameter objects. To fix e.g. the parameter c to its initial value, do the following anywhere before calling fit.execute():
c.value = 4.0
c.fixed = True
So a possible end result might be:
from symfit import parameters, variables, exp, Fit, Parameter
a, b, c, d = parameters('a, b, c, d')
x, y = variables('x, y')
c.value = 4.0
c.fixed = True
model_dict = {y: a * exp(-(x - b)**2 / (2 * c**2)) + d}
fit = Fit(model_dict, x=xdata, y=ydata)
fit_result = fit.execute()
If you want to be more dynamic in your code, you could make the Parameter objects straight away using:
c = Parameter(4.0, fixed=True)
For more info, check the docs: http://symfit.readthedocs.io/en/latest/tutorial.html#simple-example
The above example using symfit would surely simply the syntax of the fitting approach, however, does the example given really constrain the variable c?
If you look at the fit_result.param you get the following:
OrderedDict([('a', 16.374368575343127),
('b', 0.49201249437123556),
('c', 0.5337962977235504),
('d', -9.55593614465743)])
The parameter c is not 4.0.