I know there is a very similar question, but mine is different. I am running an optimization using Basinhopping, with the Powell method. Within the function I am optimizing, I also store to an external array the parameters and the resulting cost function value for each iteration, so I can afterwards check the results. I've noticed repeatedly that the lowest minimization result which the basinhopping function returns is not actually the set of parameters which resulted in the lowest overall error. I assume this is not an error, but maybe me misunderstanding how the technique works. For example, in an optimization I just ran, I found the result which was returned was actually the 35th-best option, when I check my arrays after completion. The difference in cost is very small (I'm using RMSE as a metric, and the difference is 0.02), but I still don't understand how it selected the minimum.
My first thought was maybe these parameters somehow exceeded the bounds I set, but I checked and that isn't the case.
I don't yet have a shareable reproducible version since I'm using some internal modules in the function call, but I figured I would post my question since it is more about the conceptual aspect of how basinhopping selects its result.
Related
I'm currently working on a project in Julia where I am starting with an input beta which is assumed to be incorrect. I'm running through a sequence of code that updates this beta to be the correct value and checking the error. As beta gets larger, I expect this error to reach 100%. This code ultimately does a minimization of some parameter chi which is why I've chosen to employ the optimize function from Optim.jl. The output I'm getting is below.
When I perform this calculation by hand (using 1st and 2nd derivative to update) I get this
I see that this still has some numerical instability, but it holds up longer than the Optim way does. I would expect it to behave the other way around. My optimize function is set up as
result = optimize(β -> TEfunc(E,nc,onecut,β,pcutoff,μcutoff,N),β/2,2.2*β,Brent(),abs_tol=tempcutoff,rel_tol=sqrt(tempcutoff))
βstar=Optim.minimizer(result)
Is there an argument that I'm missing in the optimize call? I just want to figure out why I have numerical instability so quickly.
In the code examples from the SAT4J documentation, calling the solver multiple times on the same SAT problem always yields the same solution, even if multiple possible solutions exist - that is, the result is deterministic.
I'm looking for a way to get different solutions on multiple runs, that is, a nondeterministic/random result. For each possible solution, there should be a non-zero probability for the solution to be picked. Ideally, every solution should be picked with the same probability, but that's not a strict requirement.
I'm aware of the possibility to (deterministically) iterate over all solutions and just take a random one, but that's not a feasible solution in my case since there are too many solutions to begin with, and calculating them all takes too long.
Yes, Sat4j is by default deterministic: it will always find the same solution if you run it several times on the same problem from the command line.
The way to add some non determinism in the heuristics is to use the RandomWalkDecorator, as found for instance in the GreedySolver in org.sat4j.minisat.SolverFactory.
Note however that if you several times such solver from the command line :
java -jar org.sat4j.core.jar GreedySolver file.cnf
you will still be deterministic, since the pseudo random numbers generator is seeded by a constant.
Thus you need to ask several models within your Java code.
As mentioned in your question, you can use a ModelIterator decorator with a bound for that:
ISolver solver = SolverFactory.newGreedySolver();
ModelIterator mi = new ModelIterator(solver,10); // look for 10 models
I'm using numpy.random for a Monte Carlo simulation where very small acceptance/rejection probabilities are possible. Although I'm seeding the RNG, I'm unable to reproduce the same sequence of random numbers. In numpy 1.15.1's documentation it says:
Compatibility Guarantee: A fixed seed and a fixed series of calls to
‘RandomState’ methods using the same parameters will always produce
the same results up to roundoff error except when the values were
incorrect. Incorrect values will be fixed and the NumPy version in
which the fix was made will be noted in the relevant docstring.
Extension of existing parameter ranges and the addition of new
parameters is allowed as long the previous behavior remains unchanged.
First of all, what do they mean by incorrect values? Second, how is roundoff error handled? Aren't values always rounded in precisely the same way? Is it possible at all that my code is not fully deterministic even though I provide a seed? I am certain that the seed is nowhere else reset because I provide my RNG object to each of my function as an argument.
It appears that I used sets throughout the code and was picking randomly from those sets, throwing a random number to pick the index of an element. The issue was that sets are unordered and the particular order of sets was uncontrollable thus random.
I try to use the z3 solver for a minimization problem. I was trying to get a timeout, and return the best solution so far. I use the python API, and the timeout option "smt.timeout" with
set_option("smt.timeout", 1000) # 1s timeout
This actually times out after about 1 second. However a larger timeout does not provide a smaller objective. I ended up turning on the verbosity with
set_option("verbose", 2)
And I think that z3 successively evaluates larger values of my objective, until the problem is satisfiable:
(opt.maxres [0:6117664])
(opt.maxres [175560:6117664])
(opt.maxres [236460:6117664])
(opt.maxres [297360:6117664])
...
(opt.maxres [940415:6117664])
(opt.maxres [945805:6117664])
...
I thus have the two questions:
Can I on contrary tell z3 to start with the upper bound, and successively return models with a smaller value for my objective function (just like for instance Minizinc annotations indomain_max http://www.minizinc.org/2.0/doc-lib/doc-annotations-search.html)
It still looks like the solver returns a satisfiable instance of my problem. How is it found? If it's trying to evaluates larger values of my objective successively, it should not have found a satisfiable instance yet when the timeout occurs...
edit: In the opt.maxres log, the upper bound never shrinks.
For the record, I found a more verbose description of the options in the source here opt_params.pyg
Edit Sorry to bother, I've beed diving into this recently once again. Anyway I think this might be usefull to others. I've been finding that I actually have to call the Optimize.upper method in order to get the upper bound, and the model is still not the one that corresponds to this upper bound. I've been able to add it as a new constraint, and call a solver (without optimization, just SAT), but that's probably not the best idea. By reading this I feel like I should call Optimize.update_upper after the solver times out, but the python interface has no such method (?). At least I can get the upper bound, and the corresponding model now (at the cost of unneccessary computations I guess).
Z3 finds solutions for the hard constraints and records the current values for the objectives and soft constraints. The last model that was found (the last model with the so-far best value for the objectives) is returned if you ask for a model. The maxres strategy mainly improves the lower bounds on the soft constraints (e.g., any solution must have cost at least xx) and whenever possible improves the upper bound (the optional solution has cost at most yy). The lower bounds don't tell you too much other than narrowing the range of possible optimal values. The upper bounds are available when you timeout.
You could try one of the other strategies, such as the one called "wmax", which
performs a branch-and-prune. Typically maxres does significantly better, but you may have better experience (depending on the problems) with wmax for improving upper bounds.
I don't have a mode where you get a stream of models. It is in principle possible, but it would require some (non-trivial) reorganization. For Pareto fronts you make successive invocations to Optimize.check() to get the successive fronts.
I am building a pymc model which must evaluate a very cpu expensive function (up to 1 sec per call on a very decent hardware). I am trying to limit the explored parameter space to meaningful solutions by means of a potential (the sum of a list of my variables has to stay within a given range). This works but I noticed that even when my potential returns an infinite value and forbids the parameters choice, this function gets evaluated. Is there a way to prevent that? Can one force the sampler to use a given evaluation sequence (pick up the necessary variables, check if the potential is ok and proceed if allowed)
I thought of using the potential inside the function itself and use it to determine whether it must proceed or immediately return, but is there a better way?
Jean-François
I am not aware of a way of ordering the evaluation of the potentials. This might not be the best way of doing so, but you might be able to check if the parameters are within reasonable at the beginning of the simulation. If the parameters are not within reasonable bounds you can return a value that will create your posterior to be zero.
Another option is to create a function for your likelihood. At the beginning of this function you could check if the parameters are within reasonable limits. If they are not you can return -inf without running your simulation. If they are reasonable you can run your model and calculate the log(p).
This is definitely not an elegant solution but it should work.
Full disclosure - I am not by any means a pymc expert.