I am looking for the method or idea to solve the following optimization problem:
min f(x)
s.t. g(xi, yi) <= f(x), i=1,...,n
where x, y are variables in R^n. f(x) is convex function with respect to x. g(xi, yi) is a bunch of convex functions with respect to (xi, yi).
It is the problem of difference of convex functions (DC) optimization due to the DC structure of the constraints. Since I am fairly new to 'DC programming', I hope to know the global optimality condition of DC programs and the efficient and popular approaches for global optimization.
In my specific problem, it is already verified that the necessary optimality condition is g(xi*, yi*)=f(x*) for i=1,...,n.
Any ideas or solution would be appreciated, thanks.
For global methods, I would suggest looking into Branch and Bound, Branch and Cut, and Cutting Plane methods. These methods may be notoriously slow though depending on the problem size. It's because it is non-convex. It would be difficult to get efficient algorithms for global optimization for this problem.
For local methods, look into the convex-concave procedure. Actually, any heuristic might work.
Related
I am having a non-linear optimization model with several variables and a certain function between three of them should be defined as a constraint. (Let us say, that the efficiency of a machine is dependent on the inlet and outlet temperatures). I have calculated some values in a table to visualize the dependency for T_inlets and T_outlets. It gives back a pretty ugly surface. A good fit would be something like a 5 degree polynomial equation if I wanted to define a function directly, but I do not think that would boost my computation speed... So instead I am considering simply having the created table and use it as a lookup table. Is a non-linear solver able to interpret this? I am using ipopt in Pyomo environment.
Another idea would be to limit my feasible temperature range and simplify the connection...maybe with using peace-wise linearization. Is it doable with 3d surfaces?
Thanks in advance!
What's the difference between using
scipy.sparse.linalg.factorized(A)
and
scipy.sparse.linalg.splu(A)
Both of them return objects with .solve(rhs) method and for both it's said in the documentation that they use LU decomposition. I'd like to know the difference in performance for both of them.
More specificly, I'm writing a python/numpy/scipy app that implements dynamic FEM model. I need to solve an equation Au = f on each timestep. A is sparse and rather large, but doesn't depend on timestep, so I'd like to invest some time beforehand to make iterations faster (there may be thousands of them). I tried using scipy.sparse.linalg.inv(A), but it threw memory exceptions when the size of matrix was large. I used scipy.linalg.spsolve on each step until recently, and now am thinking on using some sort of decomposition for better performance. So if you have other suggestions aside from LU, feel free to propose!
They should both work well for your problem, assuming that A does not change with each time step.
scipy.sparse.linalg.inv(A) will return a dense matrix that is the same size as A, so it's no wonder it's throwing memory exceptions.
scipy.linalg.solve is also a dense linear solver, which isn't what you want.
Assuming A is sparse, to solve Au=f and you only want to solve Au=f once, you could use scipy.sparse.linalg.spsolve. For example
u = spsolve(A, f)
If you want to speed things up dramatically for subsequent solves, you would instead use scipy.sparse.linalg.factorized or scipy.sparse.linalg.splu. For example
A_inv = splu(A)
for t in range(iterations):
u_t = A_inv.solve(f_t)
or
A_solve = factorized(A)
for t in range(iterations):
u_t = A_solve(f_t)
They should both be comparable in speed, and much faster than the previous options.
As #sascha said, you will need to dig into the documentation to see the differences between splu and factorize. But, you can use 'umfpack' instead of the default 'superLU' if you have it installed and set up correctly. I think umfpack will be faster in most cases. Keep in mind that if your matrix A is too large or has too many non-zeros, an LU decomposition / direct solver may take too much memory on your system. In this case, you might be stuck with using an iterative solver such as this. Unfortunately, you wont be able to reuse the solve of A at each time step, but you might be able to find a good preconditioner for A (approximation to inv(A)) to feed the solver to speed it up.
I have a two part question based on the optimization problem,
max f(x) s.t. a <= x <= b
where f is a nonlinear function and a and b are finite.
(1) I have heard that if possible, one should try transform this constrained optimization problem to an unconstrained one (I am interested in not finding local maximums but this could also be to speed up the optimization). Is this in general true?
For the specific problem at hand, I am using the "optim" function in R with "Nelder-Mead" that uses non-differentiable optimization.
(2) Is there a "best" transformation to use to transform the constrained to unconstrained problem?
I am using a +(b-a)*(sin(x)+1)/2 because it is onto and continuous (and so I am hoping not to find local maximums by searching the entire interval).
See https://math.stackexchange.com/questions/75077/mapping-the-real-line-to-the-unit-interval for some transformations. The unconstrained problem is then,
max f(a +(b-a)*(sin(x)+1)/2)
Also in the case of a one-sided constraint a < x, I have seen people use the exponential function a + exp(x). Is this the best thing to do?
I want to minimize a function, subject to constraints (the variables are non-negative). I can compute the gradient and Hessian exactly. So I want something like:
result = scipy.optimize.minimize(objective, x0, jac=grad, hess=hess, bounds=bds)
I need to specify a method for the optimization (http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html). Unfortunately I can't seem to find a method that allows for both user-specified bounds and a Hessian!
This is particularly annoying because methods "TNC" and "Newton-CG" seem essentially the same, however TNC estimates Hessian internally (in C code), while Newton-CG doesn't allow for constraints.
So, how can I do a constrained optimization with user-specified Hessian? Seems like there ought to be an easy option for this in scipy -- am I missing something?
I realized a workaround for my problem, which is to transform the constrained optimization into an unconstrained optimization.
In my case, since I have the constraint x > 0, I decided to optimize over log(x) instead of x. This was easy to do for my problem since I am using automatic differentiation.
Still, this seems like a somewhat unsatisfying solution -- I still think scipy should allow some constrained second-order minimization method.
just bumped into exactly this point myself. I think the TNC applies an active set to the line search of the CG, not the direction of the line search. Conversely the Hessian chooses the direction of the line. So, er, could maybe cut the line search out of NCG and drop it into TNC. Problem is when you are at the boundary the Hessian might not take you out of it.
How about using TNC for an extremely sloppy first guess [give it a really large error bound to hit], then use NCG with a small number of iterations, check: if on boundary back to TNC, else continue with NCG. Ugh...
Yes, or use log(x). I'm going to follow your lead.
I am working with a convex QCQP as the following:
Min e'Ie
z'Iz=n
[some linear equalities and inequalities that contain variables w,z, and e]
w>=0, z in [0,1]^n
So the problem has only one quadratic constraint, except the objective, and some variables are nonnegative. The matrices of both quadratic forms are identity matrices, thus are positive definite.
I can move the quadratic constraint to the objective but it must have the negative sign so the problem will be nonconvex:
min e'Ie-z'Iz
The size of the problem can be up to 10000 linear constraints, with 100 nonnegative variables and almost the same number of other variables.
The problem can be rewritten as an MIQP as well, as z_i can be binary, and z'Iz=n can be removed.
So far, I have been working with CPLEX via AIMMS for MIQP and it is very slow for this problem. Using the QCQP version of the problem with CPLEX, MINOS, SNOPT and CONOPT is hopeless as they either cannot find a solution, or the solution is not even close to an approximation that I know a priori.
Now I have three questions:
Do you know any method/technique to get rid of the quadratic constraint as this form without going to MIQP?
Is there any "good" solver for this QCQP? by good, I mean a solver that efficiently finds the global optimum in a resonable time.
Do you think using SDP relaxation can be a solution to this problem? I have never solved an SDP problem in reallity, so I do not know how efficient SDP version can be. Any advice?
Thanks.