I have recorded real-time stock quotes in an SQL database with fields Id, Last, and TimeStamp. Last being the current stock price (as a double), and TimeStamp is the DateTime when the price change was recorded.
I would like to replay this stream in the same way it came in, meaning if a price change was originally 12 seconds apart then the price change event firing (or something similar) should be 12 seconds apart.
In C# I might create a collection, sort it by the DateTime then fire an event using the difference in time to know when to fire off the next price change. I realize F# has a whole lot of cool new stuff relating to events, but I don't know how I would even begin this in F#. Any thoughts/code snippets/helpful links on how I might proceed with this?
I think you'll love the F# solution :-).
To keep the example simple, I'm storing the price and timestamps in a list containing tuples (the first element is the delay from the last update an the second element is the price). It shouldn't be too difficult to turn your input data into this format. For example:
let prices = [ (0, 10.0); (1000, 10.5); (500, 9.5); (2500, 8.5) ]
Now we can create a new event that will be used to replay the process. After creating it, we immediatelly attach some handler that will print the price updates:
let evt = new Event<float>()
evt.Publish.Add(printfn "Price updated: %f")
The last step is to implement the replay - this can be done using asynchronous workflow that loops over the values, asynchronously waits for the specified time and then triggers the event:
async { for delay, price in prices do
do! Async.Sleep(delay)
evt.Trigger(price) }
|> Async.StartImmediate
I'm starting the workflow using StartImmediate which means that it will run on the current thread (the waiting is asynchronous, so it doesn't block the thread). Keeping everything single-threaded makes it a bit simpler (e.g. you can safely access GUI controls).
EDIT To wrap the functionality in some component that could be used from other parts of the application, you could define a type like this:
type ReplyDataStream(prices) =
let evt = new Event<float>()
member x.Reply() =
// Start the asynchronous workflow here
member x.PriceChanged =
evt.Publish
The users can then create an instance of the type, attach their event handlers using stream.PriceChanged.Add(...) and then start the replaying the recorded changes using Reply()
Related
I can't find the answer to an interesting moment.
in akka.net I have the scheduler. It will work in actor which are sort out a number.
here a simple implementation
_statusScheduler = Context.System.Scheduler.ScheduleTellRepeatedlyCancelable(
TimeSpan.FromSeconds(_shedulerInterval),
TimeSpan.FromSeconds(_shedulerInterval),
_reporterActor,
new ProgressReport(requestId, _testedQuantity),
Self);
where
_shedulerInterval - 5-second interval,
_testedQuantity - quantity of tested number all time updated.
and after 5 seconds it is sent 0; always, not a changed number. And here is a question: is it possible to send updated quantity?
I can't send the message to the updating quantity from Recieve<> methods, because my actor is handled the counting message and it is counted the quantity all the time and updated it(when it finished it will receive next message). But all five seconds I should generate a report by a scheduler. Is it possible to fix it?
I think now I need to send all logic because it works fine, and the stone of my problem is scheduler behavior.
The issue you have here is that the message you pass into the scheduler, new ProgressReport(requestId, _testedQuantity), is what is going to be sent each time. Since you're passing in those integer values by value, the object is going to have the original values for those fields at the time you created the message and therefore the message will never update.
If you want to be able to change the content that is sent in the recurring scheduler, do this instead:
var self = Self; // need a closure here, since ActorContext won't be available
_statusScheduler = Context.System.Scheduler.Advanced.ScheduleRepeatedlyCancelable(interval, interval, () => {
_reporterActor.Tell(new ProgressReport(requestId, _testedQuantity), self);
});
This usage of the scheduler will generate a new message each time when it invokes the lambda function and thus you'll be able to include the updated integer values inside your object.
I am building a work-logging app which starts by showing a list of projects that I can select, and then when one is selected you get a collection of other buttons, to log data related to that selected project.
I decided to have a selected_project : Maybe Int in my model (projects are keyed off an integer id), which gets filled with Just 2 if you select project 2, for example.
The buttons that appear when a project is selected send messages like AddMinutes 10 (i.e. log 10 minutes of work to the selected project).
Obviously the update function will receive one of these types of messages only if a project has been selected but I still have to keep checking that selected_project is a Just p.
Is there any way to avoid this?
One idea I had was to have the buttons send a message which contains the project id, such as AddMinutes 2 10 (i.e. log 10 minutes of work to project 2). To some extent this works, but I now get a duplication -- the Just 2 in the model.selected_project and the AddMinutes 2 ... message that the button emits.
Update
As Simon notes, the repeated check that model.selected_project is a Just p has its upside: the model stays relatively more decoupled from the UI. For example, there might be other UI ways to update the projects and you might not need to have first selected a project.
To avoid having to check the Maybe each time you need a function which puts you into a context wherein the value "wrapped" by the Maybe is available. That function is Maybe.map.
In your case, to handle the AddMinutes Int message you can simply call: Maybe.map (functionWhichAddsMinutes minutes) model.selected_project.
Clearly, there's a little bit more to it since you have to produce a model, but the point is you can use Maybe.map to perform an operation if the value is available in the Maybe. And to handle the Maybe.Nothing case, you can use Maybe.withDefault.
At the end of the day is this any better than using a case expression? Maybe, maybe not (pun intended).
Personally, I have used the technique of providing the ID along with the message and I was satisfied with the result.
We are designing a system that will stream event using RabbitMQ (maybe later kafka) and spark streaming. Some or our events have been broken to several event types, so not to have too big of an event. This means that certain events have to wait for the other events (with the same id). We cannot proceed with processing until all events for a specific event have arrived.
Is there a way to delay the processing of an event until the next processing window in spark streaming (if the other event has not arrived)
Thank
Nir
From an architectural perspective, there are questions to consider:
how do you determine that all events have arrived?
what happens if one event gets lost?
what happens if events arrive out of order? Last first and similar?
In principle it would seem that breaking down an event in parts that was originally formed as a whole would increase the complexity and affect the reliability of the system.
To answer to the question in any case, since Spark 1.6.x a new stateful streaming function has been introduced: mapWithState. mapWithState allows you to keep state information per key and issue zero or more events of the same or different type in response to an incoming event.
Applied to this case, we could think of modelling the state as State[PartialEvent]: as events come in, they are assembled in a PartialEvent object. Once the criteria that an event is complete has been fulfilled, mapWithState can generate a WholeEvent object to be processed downstream.
The process would roughly(*) look like this:
val sourceEventDStream:DStream[Event] = ???
def stateUpdateFunction(eventId:String, event: Event, partialEventState: State[PartialEvent]): Option[WholeEvent] = {
val eventState = partialEventState.get() // Get current state of the event
val updatedEvent = merge(eventState, event)
if (updatedEvent.isComplete) {
partialEventState.remove()
Some(WholeEvent(updatedEvent))
} else {
partialEventState.update(updatedEvent)
None
}
}
val wholeEventDStream:DStream[WholeEvent] = sourceEventDStream.mapWithState(StateSpec.function(stateUpdateFunction))
//do stuff with wholeEventDStream ...
As you could observe, with this approach, any PartialEvent that never completes will stay in the state forever. We also need a unique key to identify events that belong together. There're timeout options that must be considered to cover for the failure cases, but the bottom line is that preserving a whole event through the pipeline would be a better approach, if technically possible.
(*) not compiled or tested. Provided only to illustrate the idea.
I'm new to psychopy and python. I'm trying to program a way to quit a script (that I didn't write), by pressing a key for example. I've added this to the while loop:
while n < total
start=time.clock()
if len(event.getKeys()) > 0:
break
# Another while loop here that ends when time is past a certain duration after 'start'.
And it's not working, it doesn't register any key presses. So I'm guessing key presses are only registered during specific times. What are those times? What is required to register key presses? That loop is extremely fast, sending signals every few milliseconds, so I can't just add wait commands in the loop.
If I could just have a parallel thread checking for a key press that would be good too, but that sounds complicated to learn.
Thanks!
Edits: The code runs as expected otherwise (in particular no errors). "core" and "event" are included. There aren't any other "event" command of any kind that would affect the "key press log".
Changing the rest of the loop content to something that includes core.wait statements makes it work. So for anybody else having this difficulty, my original guess was correct: key presses are not registered during busy times (i.e. in my case a while statement that constantly checks the time), or possibly only during specific busy times... Perhaps someone with more knowledge can clarify.
....So I'm guessing key presses are only registered during specific
times. What are those times? What is required to register key
presses?....
To try and answer your specific question, the psychopy api functions/methods that cause keyboard events to be registered are ( now updated to be literally every psychopy 1.81 API function to do this):
event.waitKeys()[1]
event.clearEvents()[1]
event.getKeys()[2]
event.Mouse.getPressed()
win.flip()
core.wait()
visual.Window.dispatchAllWindowEvents()
1: These functions also remove all existing keyboard events from the event list. This means that any future call to a function like getKeys() will only return a keyboard event if it occurred after the last time one of these functions was called.
2: If keyList=None, does the same as *, else removes keys from the key event list that are within the keyList kwarg.
Note that one of the times keyboard events are 'dispatched' is in the event.getKeys() call itself. By default, this function also removes any existing key events.
So, without being seeing the full source of the inner loop that you mention, it seems highly likely that the event.getKeys() is never returning a key event because key events are being consumed by some other call within the inner loop. So the chance that an event is in the key list when the outer getKeys() is called is very very low.
Update in response to OP's comment on Jonas' test script ( I do not have enough rep to add comments to answers yet):
... Strange that you say this ..[jonas example code].. works
and from Sol's answer it would seem it shouldn't. – zorgkang
Perhaps my answer is giving the wrong understanding, as it is intended to provide information that shows exactly why Jonas' example should, and does, work. Jonas' example code works because the only time key events are being removed from the event buffer is when getKeys() is called, and any events that are removed are also returned by the function, causing the loop to break.
This is not really an answer. Here's an attempt to minimally reproduce the error. If the window closes on keypress, it's a success. It works for me, so I failed to reproduce it. Does it work for you?
from psychopy import event, visual, core
win = visual.Window()
clock = core.Clock()
while True:
clock.reset()
if event.getKeys():
break
while clock.getTime() < 1:
pass
I don't have the time module installed, so I used psychopy.core.Clock() instead but it shouldn't make a difference, unless your time-code ends up in an infinite loop, thus only running event.getKeys() once after a few microseconds.
In short, I'm trying to "sort" incoming results of using threadpooling as they finish. I have a functional solution, but there's no way in the world it's the best way to do it (it's prone to huge pauses). So here I am! I'll try to hit the bullet points of what's going on/what needs to happen and then post my current solution.
The intent of the code is to get information about files in a directory, and then write that to a text file.
I have a list (Counter.ListOfFiles) that is a list of the file paths sorted in a particular way. This is the guide that dictates the order I need to write to the text file.
I'm using a threadpool to collect the information about each file, create a stringbuilder with all of the text ready to write to the text file. I then call a procedure(SyncUpdate, inlcluded below), send the stringbuilder(strBld) from that thread along with the name of the path of the file that particular thread just wrote to the stringbuilder about(Xref).
The procedure includes a synclock to hold all the other threads until it finds a thread passing the correct information. That "correct" information being when the xref passed by the thread matches the first item in my list (FirstListItem). When that happens, I write to the text file, delete the first item in the list and do it again with the next thread.
The way I'm using the monitor is probably not great, in fact I have little doubt I'm using it in an offensively wanton manner. Basically while the xref (from the thread) <> the first item in my list, I'm doing a pulseall for the monitor. I originally was using monitor.wait, but it would eventually just give up trying to sort through the list, even when using a pulse elsewhere. I may have just been coding something awkwardly. Either way, I don't think it's going to change anything.
Basically the problem comes down to the fact that the monitor will pulse through all of the items it has in the queue, when there's a good chance the item I am looking for probably got passed to it somewhere earlier in the queue or whatever and it's now going to sort through all of the items again before looping back around to find a criteria that matches. The result of this is that my code will hit one of these and take a huge amount of time to complete.
I'm open to believing I'm just using the wrong tool for the job, or just not using tool I have correctly. I would strongly prefer some sort of threaded solution (unsurprisingly, it's much faster!). I've been messing around a bit with the Parallel Task functionality today, and a lot of the stuff looks promising, but I have even less experience with that vs. threadpool, and you can see how I'm abusing that! Maybe something with queue? You get the idea. I am directionless. Anything someone could suggest would be much appreciated. Thanks! Let me know if you need any additional information.
Private Sub SyncUpdateResource(strBld As Object, Xref As String)
SyncLock (CType(strBld, StringBuilder))
Dim FirstListitem As String = counter.ListOfFiles.First
Do While Xref <> FirstListitem
FirstListitem = Counter.ListOfFiles.First
'This makes the code much faster for reasons I can only guess at.
Thread.Sleep(5)
Monitor.PulseAll(CType(strBld, StringBuilder))
Loop
Dim strVol As String = Form1.Volname
Dim strLFPPath As String = Form1.txtPathDir
My.Computer.FileSystem.WriteAllText(strLFPPath & "\" & strVol & ".txt", strBld.ToString, True)
Counter.ListOfFiles.Remove(Xref)
End SyncLock
End Sub
This is a pretty typical multiple producer, single consumer application. The only wrinkle is that you have to order the results before they're written to the output. That's not difficult to do. So let's let that requirement slide for a moment.
The easiest way in .NET to implement a producer/consumer relationship is with BlockingCollection, which is a thread-safe FIFO queue. Basically, you do this:
In your case, the producer threads get items, do whatever processing they need to, and then put the item onto the queue. There's no need for any explicit synchronization--the BlockingCollection class implementation does that for you.
Your consumer pulls things from the queue and outputs them. You can see a really simple example of this in my article Simple Multithreading, Part 2. (Scroll down to the third example if you're just interested in the code.) That example just uses one producer and one consumer, but you can have N producers if you want.
Your requirements have a little wrinkle in that the consumer can't just write items to the file as it gets them. It has to make sure that it's writing them in the proper order. As I said, that's not too difficult to do.
What you want is a priority queue of some sort onto which you can place an item if it comes in out of order. Your priority queue can be a sorted list or even just a sequential list if the number of items you expect to get out of order isn't very large. That is, if you typically have only a half dozen items at a time that could be out of order, then a sequential list could work just fine.
I'd use a heap because it performs well. The .NET Framework doesn't supply a heap, but I have a simple one that works well for jobs like this. See A Generic BinaryHeap Class.
So here's how I'd write the consumer (the code is in pseudo-C#, but you can probably convert it easily enough).
The assumption here is that you have a BlockingCollection called sharedQueue that contains the items. The producers put items on that queue. Consumers do this:
var heap = new BinaryHeap<ItemType>();
foreach (var item in sharedQueue.GetConsumingEnumerable())
{
if (item.SequenceKey == expectedSequenceKey)
{
// output this item
// then check the heap to see if other items need to be output
expectedSequenceKey = expectedSequenceKey + 1;
while (heap.Count > 0 && heap.Peek().SequenceKey == expectedSequenceKey)
{
var heapItem = heap.RemoveRoot();
// output heapItem
expectedSequenceKey = expectedSequenceKey + 1;
}
}
else
{
// item is out of order
// put it on the heap
heap.Insert(item);
}
}
// if the heap contains items after everything is processed,
// then some error occurred.
One glaring problem with this approach as written is that the heap could grow without bound if one of your consumers crashes or goes into an infinite loop. But then, your other approach probably would suffer from that as well. If you think that's an issue, you'll have to add some way to skip an item that you think won't ever be forthcoming. Or kill the program. Or something.
If you don't have a binary heap or don't want to use one, you can do the same thing with a SortedList<ItemType>. SortedList will be faster than List, but slower than BinaryHeap if the number of items in the list is even moderately large (a couple dozen). Fewer than that and it's probably a wash.
I know that's a lot of info. I'm happy to answer any questions you might have.