NOTE: This question has been ported over from Programmers since it appears to be more appropriate here given the limitation of the language I'm using (VBA), the availability of appropriate tags here and the specificity of the problem (on the inference that Programmers addresses more theoretical Computer Science questions).
I'm attempting to build a Discrete Event Simulation library by following this tutorial and fleshing it out. I am limited to using VBA, so "just switch to [insert language here] and it's easy!" is unfortunately not possible. I have specifically chosen to implement this in Access VBA to have a convenient location to store configuration information and metrics.
How should I handle logging metrics in my Discrete Event Simulation engine?
If you don't want/need background, skip to The Design or The Question section below...
Simulation
The goal of a simulation of the type in question is to model a process to perform analysis of it that wouldn't be feasible or cost-effective in reality.
The canonical example of a simulation of this kind is a Bank:
Customers enter the bank and get in line with a statistically distributed frequency
Tellers are available to handle customers from the front of the line one by one taking an amount of time with a modelable distribution
As the line grows longer, the number of tellers available may have to be increased or decreased based on business rules
You can break this down into generic objects:
Entity: These would be the customers
Generator: This object generates Entities according to a distribution
Queue: This object represents the line at the bank. They find much real world use in acting as a buffer between a source of customers and a limited service.
Activity: This is a representation of the work done by a teller. It generally processes Entities from a Queue
Discrete Event Simulation
Instead of a continuous tick by tick simulation such as one might do with physical systems, a "Discrete Event" Simulation is a recognition that in many systems only critical events require process and the rest of the time nothing important to the state of the system is happening.
In the case of the Bank, critical events might be a customer entering the line, a teller becoming available, the manager deciding whether or not to open a new teller window, etc.
In a Discrete Event Simulation, the flow of time is kept by maintaining a Priority Queue of Events instead of an explicit clock. Time is incremented by popping the next event in chronological order (the minimum event time) off the queue and processing as necessary.
The Design
I've got a Priority Queue implemented as a Min Heap for now.
In order for the objects of the simulation to be processed as events, they implement an ISimulationEvent interface that provides an EventTime property and an Execute method. Those together mean the Priority Queue can schedule the events, then Execute them one at a time in the correct order and increment the simulation clock appropriately.
The simulation engine is a basic event loop that pops the next event and Executes it until there are none left. An event can reschedule itself to occur again or allow itself to go idle. For example, when a Generator is Executed it creates an Entity and then reschedules itself for the generation of the next Entity at some point in the future.
The Question
How should I handle logging metrics in my Discrete Event Simulation engine?
In the midst of this simulation, it is necessary to take metrics. How long are Entities waiting in the Queue? How many Acitivity resources are being utilized at any one point? How many Entities were generated since the last metrics were logged?
It follows logically that the metric logging should be scheduled as an event to take place every few units of time in the simulation.
The difficulty is that this ends up being a cross-cutting concern: metrics may need to be taken of Generators or Queues or Activities or even Entities. Consider also that it might be necessary to take derivative calculated metrics: e.g. measure a, b, c, and ((a-c)/100) + Log(b).
I'm thinking there are a few main ways to go:
Have a single, global Stats object that is aware of all of the simulation objects. Have the Generator/Queue/Activity/Entity objects store their properties in an associative array so that they can be referred to at runtime (VBA doesn't support much in the way of reflection). This way the statistics can be attached as needed Stats.AddStats(Object, Properties). This wouldn't support calculated metrics easily unless they are built into each object class as properties somehow.
Have a single, global Stats object that is aware of all of the simulation objects. Create some sort of ISimStats interface for the Generator/Queue/Activity/Entity classes to implement that returns an associative array of the important stats for that particular object. This would also allow runtime attachment, Stats.AddStats(ISimStats). The calculated metrics would have to be hardcoded in the straightforward implementation of this option.
Have multiple Stats objects, one per Generator/Queue/Activity/Entity as a child object. This might make it easier to implement simulation object-specific calculated metrics, but clogs up the Priority Queue a little bit with extra things to schedule. It might also cause tighter coupling, which is bad :(.
Some combination of the above or completely different solution I haven't thought of?
Let me know if I can provide more (or less) detail to clarify my question!
Any and every performance metric is a function of the model's state. The only time the state changes in a discrete event simulation is when an event occurs, so events are the only time you have to update your metrics. If you have enough storage, you can log every event, its time, and the state variables which got updated, and retrospectively construct any performance metric you want. If storage is an issue you can calculate some performance measures within the events that affect those measures. For instance, the appropriate time to calculate delay in queue is when a customer begins service (assuming you tagged each customer object with its arrival time). For delay in system it's when the customer ends service. If you want average delays, you can update the averages in those events. When somebody arrives, the size of the queue gets incremented, then they begin service it gets decremented. Etc., etc., etc.
You'll have to be careful calculating statistics such as average queue length, because you have to weight the queue lengths by the amount of time you were in that state: Avg(queue_length) = (1/T) integral[queue_length(t) dt]. Since the queue_length can only change at events, this actually boils down to summing the queue lengths multiplied by the amount of time you were at that length, then divide by total elapsed time.
Related
I have a domain model where I penalize a score at multiple levels within a single rule. Consider a cloud scheduling problem where we have to assign processes to computers, but each process can be split amongst several computers. Each process has a threshold (e.g. 75%), and we can only "win" the process if we can schedule up to its threshold. We get some small additional benefit from scheduling the remaining 25% of the process, but our solver is geared to "winning" as many processes as possible, so we should be scheduling as many processes as possible to their threshold before scheduling the remainder of the process.
Our hard rule counts hard constraints (we can't schedule more processes on a computer than it can handle)
Our medium rule is rewarded for how many processes have been scheduled up to the threshold (no additional reward for going above 75%).
Our soft rule is rewarded for how many processes have been scheduled total (here we do get additional reward for going above 75%).
This scoring implementation means that it is more important to schedule all processes up to their threshold than to waste precious computer space scheduling 100% of a process.
When we used a drools implementation, we had a rule which rewarded the medium and soft levels simultaneously.
when
$process : Process()
$percentAllocated : calculatePercentProcessAllocated($process) //uses an accumulator over all computers
then
mediumReward;
if ($percentAllocated > $process.getThreshold()) {
mediumReward = $process.getThreshold();
}
else {
mediumReward = $percentAllocated;
}
softReward = $percentAllocated;
scoreHolder.addMultiConstraintMatch(0, mediumReward, softReward);
The above pseudo-drools is heavily simplified, just want to show how we were rewarded two levels at once.
The problem is that I don't see any good way to apply multi constraint matches using constraint streams. All the examples I see automatically add a terminator after applying a penalize or reward method, so that no further score modifications can be made. The only way I see to implement my rules is to make two rules which are identical outside of their reward calls.
I would very much like to avoid running the same constraint twice if possible, so is there a way to penalize the score at multiple levels at once?
Also to anticipate a possible answer to this question, it is not possible to split our domain model so that each process is two processes (one process from 0% to the threshold, and another from the threshold to 100%). Part of the accumulation that I have glossed over involves linking the two parts and would be too expensive to perform if they were separate objects.
There is currently no way in the constraint streams API to do that. Constraint weights are assumed to be constant.
If you must do this, you can take advantage of the fact that there really is no "medium" score. The medium part in HardMediumSoftScore is just another level of soft. Therefore 0hard/1medium/2soft would in practice behave the same as 0hard/1000002soft. If you pick a sufficiently high constant to multiply the medium part with, you can have HardSoftScore work just like HardMediumSoftScore and implement your use case at the same time.
Is it a hack? Yes. Do I like it? No. But it does solve your issue.
Another way to do that would be to take advantage of node sharing. I'll use an example that will show it better than a thousand words:
UniConstraintStream<Person> stream = constraintFactory.forEach(Person.class);
Constraint one = stream.penalize(HardSoftScore.ONE).asConstraint("Constraint 1")
Constraint two = stream.penalize(HardSoftScore.TEN).asConstraint("Constraint 2")
This looks like two constraints executing twice. And in CS-Drools, it is exactly that. But CS-Bavet can actually optimize this and will only execute the stream once, applying two different penalties at the end.
This is not a very nice programming model, you lose the fluency of the API and you need to switch to the non-default CS implementation. But again - if you absolutely need to do what you want to do, this would be another way.
In Cocoa, NSEvents have a timestamp property which returns a timeinterval representing "The time when the event occurred in seconds since system startup." I am writing an application which needs exactly this information- precisely when the user presses keys. But I am worried that the timestamp might not be accurate enough. Timeinterval itself has sub-millisecond precision, which is great. Is there any documentation that indicates whether or not the sub-millisecond precision is used to provide accurate timestamps for NSEvents that represent keyboard and mouse inputs?
If I had to guess at apple's implementation, I would hope that the timestamp on the NSEvent might be generated soon after a hardware interrupt, which would be fantastic. However I could also imagine a system which polls for keyboard/mouse inputs and only populates the timestamp field when the next poll interval comes around and reads the inputs.
Thanks very much for any insight.
You may find Apple's Cocoa Event Handling Guide: Event Architecture document a useful read. From that:
Before it dispatches an event to an application, the window server processes it in various ways; it time-stamps it, annotates it with the associated window and process port, and possibly performs other tasks as well.
This seems to indicate the Window Server, which lives between the kernel and the application layer, applies the timestamp... though it is possible it reads metadata that was generated at a lower level in order to do this.
I would think the best way to get a better sense of this would be to write an app that records timestamps of key down/up events and look for exact matches in adjacent events. If you see several events in a row with (nearly) identical timestamps, it's more likely they were queued in some buffer before they got timestamped. If there is enough of a gap between adjacent events, it's more likely getting stamped closer to the hardware event.
This is an example (pseudo code) of how you could simulate and render a video game.
//simulate 20ms into the future
const long delta = 20;
long simulationTime = 0;
while(true)
{
while(simulationTime < GetMilliSeconds()) //GetMilliSeconds = Wall Clock Time
{
//the frame we simulated is still in the past
input = GetUserlnput();
UpdateSimulation(delta, input);
//we are trying to catch up and eventually pass the wall clock time
simulationTime += delta;
}
//since my current simulation is in the future and
//my last simulation is in the past
//the current looking of the world has got to be somewhere inbetween
RenderGraphics(InterpolateWorldState(GetMilliSeconds() - simulationTime));
}
That's my question:
I have 40ms to go through the outer 'while true' loop (means 25FPS).
The RenderGraphics method takes 10ms. So that means I have 30ms for the inner loop. The UpdateSimulation method takes 5ms. Everything else can be ignored since it's a value under 0.1ms.
What is the maximum I can set the variable 'delta' to in order to stay in my time schedule of 40ms (outer loop)?
And why?
This largely depends on how often you want and need to update your simulation status and user input, given the constraints mentioned below. For example, if your game contains internal state based on physical behavior, you would need a smaller delta to ensure that movements and collisions, if any, are properly evaluated and reflected within the game state. Also, if your user input requires fine-grained evaluation and state update, you would also need smaller delta values. For example, a shooting game with analogue user input (e.g. mouse, joystick), would benefit from update frequencies larger than 30Hz. If your game does not need such high-frequency evaluation of input and game state, then you could get away with larger delta values, or even by simply updating your game state once any input by the player was being detected.
In your specific pseudo-code, your simulation would update according to a fixed time-slice of length delta, which requires your simulation update to be processed in less wallclock time than the wallclock time to be simulated. Otherwise, wallclock time would proceed faster, than your simulation time can be updated. This ultimately limits your delta depending on how quick any simulation update of delta simulation time can actually be computed. This relationship also depends on your use case and may not be linear or constant. For example, physics engines often would divide your delta time given internally to what update rate they can reasonably process, as longer delta times may cause numerical instabilities and harder to solve linear systems raising processing effort non-linearly. In other use cases, simulation updates may take a linear or even constant time. Even so, many (possibly external) events could cause your simulation update to be processed too slowly, if it is inherently demanding. For example, loading resources during simulation updates, your operating system deciding to lay your execution thread aside, another process run by the user, anti-virus software kicking in, low memory pressure, a slow CPU and so on. Until now, I saw mostly two strategies to evade this problem or remedy its effects. First, simply ignoring it could work if the simulation update effort is low and it is assumed that the cause of the slowdown is temporary only. This would result in more or less noticeable "slow motion" behavior of your simulation, which could - in worst case - lead to simulation time lag piling up forever. The second strategy I often saw was to simply cap the measured frame time to be simulated to some artificial value, say 1000ms. This leads to smooth behavior as soon as the cause of slow down disappears, but has the drawback that the 'capped' simulation time is 'lost', which may lead to animation hiccups if not handled or accounted for. To choose a strategy, analyzing your use case could consist of measuring the wallclock time it takes to process simulation updates of delta and x * delta time and how changing the delta time and simulation load to process actually reflects in wallclock time needed to compute it, which will hint you to what the maximum value of delta is for your specific hardware and software environment.
I'm reading an article on real-time kernels, and the author explains how to implement a scheduler for tasks with a linked list. He also states that this is not the best way since tasks are inserted and removed based on priority; however, he doesn't explain what those other methods are.
What are the other methods for implementing a scheduler other than a linked list?
Take a good hard look at the Queue Data structure. If you have a queue for each priority level, then you can start at the highest priority queue, and process until the queue is empty, then step to the next priority query, until you have hit all of the priorities.
Having tasks at the same priority level in a queue, allows you to guarentee that each task gets at least one quantum of processing, before it is thrown into the tail of (possibly another) queue.
Of course for real-time processsing, you want quick response to an interrupt. Perhaps some sort of Priority Queue might be applicable.
There's lots, for instance could have been a double linked list, so for inserting a low priority task, could have searched backwards from the tail.
You could implement the schedule as in list of tasks with anything from an array to a B-Tree, which one you use depends on what you are scheduling.
Linked list, if it's fairly short might be the optimal solution.
Many games that are created these days come with their own achievement system that rewards players/users for accomplishing certain tasks. The badges system here on stackoverflow is exactly the same.
There are some problems though for which I couldn't figure out good solutions.
Achievement systems have to watch out for certain events all the time, think of a game that offers 20 to 30 achievements for e.g.: combat. The server would have to check for these events (e.g.: the player avoided x attacks of the opponent in this battle or the player walked x miles) all time.
How can a server handle this large amount of operations without slowing down and maybe even crashing?
Achievement systems usually need data that is only used in the core engine of the game and wouldn't be needed out of there anyway if there weren't those nasty achievements (think of e.g.: how often the player jumped during each fight, you don't want to store all this information in a database.). What I mean is that in some cases the only way of adding an achievement would be adding the code that checks for its current state to the game core, and thats usually a very bad idea.
How do achievement systems interact with the core of the game that holds the later unnecessary information? (see examples above)
How are they separated from the core of the game?
My examples may seem "harmless" but think of the 1000+ achievements currently available in World of Warcraft and the many, many players online at the same time, for example.
Achievement systems are really just a form of logging. For a system like this, publish/subscribe is a good approach. In this case, players publish information about themselves, and interested software components (that handle individual achievements) can subscribe. This allows you to watch public values with specialised logging code, without affecting any core game logic.
Take your 'player walked x miles' example. I would implement the distance walked as a field in the player object, since this is a simple value to increment and does not require increasing space over time. An achievement that rewards players that walk 10 miles is then a subscriber of that field. If there were many players then it would make sense to aggregate this value with one or more intermediate broker levels. For example, if 1 million players exist in the game, then you might aggregate the values with 1000 brokers, each responsible for tracking 1000 individual players. The achievement then subscribes to these brokers, rather than to all the players directly. Of course, the optimal hierarchy and number of subscribers is implementation-specific.
In the case of your fight example, players could publish details of their last fight in exactly the same way. An achievement that monitors jumping in fights would subscribe to this info, and check the number of jumps. Since no historical state is required, this does not grow with time either. Again, no core code need be modified; you only need to be able to access some values.
Note also that most rewards do not need to be instantaneous. This allows you some leeway in managing your traffic. In the previous example, you might not update the broker's published distance travelled until a player has walked a total of one more mile, or a day has passed since last update (incrementing internally until then). This is really just a form of caching; the exact parameters will depend on your problem.
You can even do this if you don't have access to source, for example in videogame emulators. A simple memory-scan tool can be written to find the displayed score for example. Once you have that your achievement system is as easy as polling that memory location every frame and seeing if their current "score" or whatever is higher than their highest score. The cool thing about videogame emulators is that memory locations are deterministic (no operating system).
There are two ways this is done in normal games.
Offline games: nothing as complex as pub/sub - that's massive overkill. Instead you just use a big map / dictionary, and log named "events". Then every X frames, or Y seconds (or, usually: "every time something dies, and 1x at end of level"), you iterate across achievements and do a quick check. When the designers want a new event logged, it's trivial for a programmer to add a line of code to record it.
NB: pub/sub is a poor fit for this IME because the designers never want "when player.distance = 50". What they actually want is "when player's distance as perceived by someone watching the screen seems to have travelled past the first village, or at least 4 screen widths to the right" -- i.e. far more vague and abstract than a simple counter.
In practice, that means that the logic goes at the point where the change happens (before the event is even published), which is a poor way to use pub/sub. There are some game engines that make it easier to do a "logic goes at the point of receipt" (the "sub" part), but they're not the majority, IME.
Online games: almost identical, except you store "counters" (int that goes up), and usually also: "deltas" (circular buffers of what's-happened frame to frame), and: "events" (complex things that happened in game that can be hard-coded into a single ID plus a fixed-size array of parameters). These are then exposed via e.g SNMP for other servers to collect at low CPU cost and asynchronously
i.e. almost the same as 1 above, except that you're careful to do two things:
Fixed-size memory usage; and if the "reading" servers go offline for a while, achievements won in that time will need to be re-won (although you usually can have a customer support person manually go through the main system logs and work out that the achievement "probably" was won, and manually award it)
Very low overhead; SNMP is a good standard for this, and most teams I know end up using it
If your game architecture is Event-driven, then you can implement achievements system using finite-state machines.