I've been slowly building up an energy dispatch model using Pyomo, which now has most of the dispatch constraints that I want included.
Now comes for what I think may be the tricky bit. To avoid the problem of perfect foresight, I want to optimise one day at a time. When I optimise each subsequent day, the model will be passed new information: information on the load point and whether each unit is committed, from the previous day's result. At the moment my model just runs each day independently, and churns out a json file with the outputs from each day.
I think I now want to modify the workflow so that:
I optimise for day 'd'
Grab selected outputs from the optimisation of 'd'
These outputs become initial condition for 'd+1'
Optimise 'd+1'
Subsequently, I will add look-aheads into the code, but I think that will be straight-forward once I have cracked the above.
I think I could work this out by writing some code that processes the outputs from 'd' outside of Pyomo, creates a new set of inputs to 'd+1' and then goes back to Pyomo for optimise 'd+1', but that feels like a cumbersome solution that might be quite slow. Is anyone able to point me towards an example or guidance on how I might tackle this in a more efficient way?
I wrote a little something to help post process all parameters and variables of a Pyomo model after solve.
This may help you easily get all your outputs and put them as inputs in your next iteration.
Good luck.
https://github.com/judejeh/PyomoSolverWrapper
Related
I have been working on a simulation model for battery swapping in Anylogic. So far I have developed the simulation model, optimization experiment and parameters variation experiment.
There are no errors in the model but the output values are unsatisfactory. Small changes such as changing the step size of the decision variables results in a drastic change in the best value obtained after every experiment. Though the objective does not change much but I am concerned about the other variables that are changing with each run. Even with multiple optimization runs it is difficult to come to a conclusion.
For reference I am posting an output of parameters variation experiment here. I ran the experiment with an optimized value but I was getting feasible results (percentile > 95%) far off the expected input values. Although, the overall result is correct (decreasing percentile with increasing charging time) but it is difficult to understand the variability.
Can anyone help?enter image description here
When building a model, this is a common problem you will have when looking at high level overall outputs. You could have a model bug, but it is just as likely (if not more likely) that there is some dynamic to your system that was not clear in simple Excel spreadsheets or mental models. The DES may be telling us something truly interesting about the system behavior, but without additional outputs, there is no way to understand what that is.
A few suggestions:
Run this as a simple single scenario, where you manually update inputs. When you run this with the low range of input values and then the high range of input values, what do you see on the animation or additional outputs that is different than you expected or could explain the overall output trend? Try running several intermediate points.
Add additional output metrics. If you look at queue sizes, resource utilizations, turn-around-times, etc; do you see anything at that level that is different than expected?
Add a "replication" log. When you run a set of inputs for multiple scenarios, does any single replication stand out as an outlier? If so, re-run the scenario with that set of inputs and that random seed.
There is no substitute for understanding underlying system behavior, and without understanding those dynamics, looking at overall correlation with optimization or parameter variation experiments will often lead companies to make the wrong policies decisions.
I am wondering if it's possible to treat scheduling problems with tasks with the following property using Optaplanner. Instead of have a fixed duration of 1 hour we have a 1 hour-man, i.e if there is two employees working on that task, it could be done in 1/2 hour.
Otherwise, what are the other solvers that could be used ?
Model wise, the easy approach is to split up that 1 task into 2 smaller tasks that get individually assigned. (When they're both assigned to the same person, sequentially after each other, you can add a soft constraint to reward that.) The downside is that you have to decide in advance for each task into how many pieces they can be split up.
In reality, tasks are rarely arbitrary dividable. Some parts of each task are atomic. For example, taking out the garbage can is a do-or-do-not task. Taking it half the way out, or taking half of it out, and assigning someone else to do the rest, is not allowed because it will increase the time spent on it.
Some tasks need at least 2 persons to execution. For example, someone to hold the ladder while the other is standing on it. In the docs, see the auto delay to last pattern.
Alternatively to the simple model, you can also play with nullable=true and custom moves to allow multiple people to assign to the same tasks, but it's complicated. It can avoid having to tune the number of task pieces in advance too much. Once we support #PlanningVariableCollection, and do so fully, more and better options in this regard will become available.
As i can understand from the documentation the purpose of the "MoveIteratorFactory" is to generate the moves as they are needed to be played out at every step.
Is the "getSize" method how big the subset of moves will be?
What is the difference between "createOriginalMoveIterator" and "createRandomMoveIterator"?
Will the "createOriginalMoveIterator" regenerate a subset of moves again in a later step or not?
I guess that the move is played out after the method next() is called (after its creation) in both the cases?
Is this the best solution if there are a lot moves that need to be generated ,because in my case there are many moves that need to be first of all generated let alone played out?
Is there a smarter way to generate custom combined moves at every step that are based on the current state of the solution?
The MoveIteratorFactory is a more complex alternative to MoveListFactory which avoids creating a List<Move> by creating an Iterator<Move> instead. In Java 8 terms, it has a lot in common with a Stream.
Suppose you have 50k entities and 10k values. A list of all changemoves would contain 500m elements, causing memory issues and a serious delay loss every time a step starts. To avoid that, it generates a changemove just in time, at Iterator.next() or hasNext().
For that to work well, it needs to behave differently if the iterator needs to selects moves randomly (= might select the same move twice, this is not shuffled, it's random!) or original order.
Suppose you have 50k entities and 10k values. A list of all changemoves would contain 500m elements, causing memory issues and a serious delay loss every time a step starts. To avoid that, it generates a changemove just in time, at Iterator.next() or hasNext()
This is an academic rather than practical question. In the Traveling Salesman Problem, or any other which involves finding a minimum optimization ... if one were using a map/reduce approach it seems like there would be some value to having some means for the current minimum result to be broadcast to all of the computational nodes in some manner that allows them to abandon computations which exceed that.
In other words if we map the problem out we'd like each node to know when to give up on a given partial result before it's complete but when it's already exceeded some other solution.
One approach that comes immediately to mind would be if the reducer had a means to provide feedback to the mapper. Consider if we had 100 nodes, and millions of paths being fed to them by the mapper. If the reducer feeds the best result to the mapper than that value could be including as an argument along with each new path (problem subset). In this approach the granularity is fairly rough ... the 100 nodes will each keep grinding away on their partition of the problem to completion and only get the new minimum with their next request from the mapper. (For a small number of nodes and a huge number of problem partitions/subsets to work across this granularity would be inconsequential; also it's likely that one could apply heuristics to the sequence in which the possible routes or problem subsets are fed to the nodes to get a rapid convergence towards the optimum and thus minimize the amount of "wasted" computation performed by the nodes).
Another approach that comes to mind would be for the nodes to be actively subscribed to some sort of channel, or multicast or even broadcast from which they could glean new minimums from their computational loop. In that case they could immediately abandon a bad computation when notified of a better solution (by one of their peers).
So, my questions are:
Is this concept covered by any terms of art in relation to existing map/reduce discussions
Do any of the current map/reduce frameworks provide features to support this sort of dynamic feedback?
Is there some flaw with this idea ... some reason why it's stupid?
that's a cool theme, that doesn't have that much literature, that was done on it before. So this is pretty much a brainstorming post, rather than an answer to all your problems ;)
So every TSP can be expressed as a graph, that looks possibly like this one: (taken it from the german Wikipedia)
Now you can run a graph algorithm on it. MapReduce can be used for graph processing quite well, although it has much overhead.
You need a paradigm that is called "Message Passing". It was described in this paper here: Paper.
And I blog'd about it in terms of graph exploration, it tells quite simple how it works. My Blogpost
This is the way how you can tell the mapper what is the current minimum result (maybe just for the vertex itself).
With all the knowledge in the back of the mind, it should be pretty standard to think of a branch and bound algorithm (that you described) to get to the goal. Like having a random start vertex and branching to every adjacent vertex. This causes a message to be send to each of this adjacents with the cost it can be reached from the start vertex (Map Step). The vertex itself only updates its cost if it is lower than the currently stored cost (Reduce Step). Initially this should be set to infinity.
You're doing this over and over again until you've reached the start vertex again (obviously after you visited every other one). So you have to somehow keep track of the currently best way to reach a vertex, this can be stored in the vertex itself, too. And every now and then you have to bound this branching and cut off branches that are too costly, this can be done in the reduce step after reading the messages.
Basically this is just a mix of graph algorithms in MapReduce and a kind of shortest paths.
Note that this won't yield to the optimal way between the nodes, it is still a heuristic thing. And you're just parallizing the NP-hard problem.
BUT a little self-advertising again, maybe you've read it already in the blog post I've linked, there exists an abstraction to MapReduce, that has way less overhead in this kind of graph processing. It is called BSP (Bulk synchonous parallel). It is more freely in the communication and it's computing model. So I'm sure that this can be a lot better implemented with BSP than MapReduce. You can realize these channels you've spoken about better with it.
I'm currently involved in an Summer of Code project which targets these SSSP problems with BSP. Maybe you want to visit if you're interested. This could then be a part solution, it is described very well in my blog, too. SSSP's in my blog
I'm excited to hear some feedback ;)
It seems that Storm implements what I was thinking of. It's essentially a computational topology (think of how each compute node might be routing results based on a key/hashing function to the specific reducers).
This is not exactly what I described, but might be useful if one had a sufficiently low-latency way to propagate current bounding (i.e. local optimum information) which each node in the topology could update/receive in order to know which results to discard.
Tried doing a bit of research on the following with no luck. Thought I'd ask here in case someone has come across it before.
I help a volunteer-run radio station with their technology needs. One of the main things that have come up is they would like to schedule their advertising programmatically.
There are a lot of neat and complex rule engines out there for advertising, but all we need is something pretty simple (along with any experience that's worth thinking about).
I would like to write something in SQL if possible to deal with these entities. Ideally if someone has written something like this for other advertising mediums (web, etc.,) it would be really helpful.
Entities:
Ads (consisting of a category, # of plays per day, start date, end date or permanent play)
Ad Category (Restaurant, Health, Food store, etc.)
To over-simplify the problem, this will be a elegant sql statement. Getting there... :)
I would like to be able to generate a playlist per day using the above two entities where:
No two ads in the same category are played within x number of ads of each other.
(nice to have) high promotion ads can be pushed
At this time, there are no "ad slots" to fill. There is no "time of day" considerations.
We queue up the ads for the day and go through them between songs/shows, etc. We know how many per hour we have to fill, etc.
Any thoughts/ideas/links/examples? I'm going to keep on looking and hopefully come across something instead of learning it the long way.
Very interesting question, SMO. Right now it looks like a constraint programming problem because you aren't looking for an optimal solution, just one that satisfies all the constraints you have specified. In response to those who wanted to close the question, I'd say they need to check out constraint programming a bit. It's far closer to stackoverflow that any operations research sites.
Look into constraint programming and scheduling - I'll bet you'll find an analogous problem toot sweet !
Keep us posted on your progress, please.
Ignoring the T-SQL request for the moment since that's unlikely to be the best language to write this in ...
One of my favorites approaches to tough 'layout' problems like this is Simulated Annealing. It's a good approach because you don't need to think HOW to solve the actual problem: all you define is a measure of how good the current layout is (a score if you will) and then you allow random changes that either increase or decrease that score. Over many iterations you gradually reduce the probability of moving to a worse score. This 'simulated annealing' approach reduces the probability of getting stuck in a local minimum.
So in your case the scoring function for a given layout might be based on the distance to the next advert in the same category and the distance to another advert of the same series. If you later have time of day considerations you can easily add them to the score function.
Initially you allocate the adverts sequentially, evenly or randomly within their time window (doesn't really matter which). Now you pick two slots and consider what happens to the score when you switch the contents of those two slots. If either advert moves out of its allowed range you can reject the change immediately. If both are still in range, does it move you to a better overall score? Initially you take changes randomly even if they make it worse but over time you reduce the probability of that happening so that by the end you are moving monotonically towards a better score.
Easy to implement, easy to add new 'rules' that affect score, can easily adjust run-time to accept a 'good enough' answer, ...
Another approach would be to use a genetic algorithm, see this similar question: Best Fit Scheduling Algorithm this is likely harder to program but will probably converge more quickly on a good answer.