According to the Interactive Brokers TWS API documentation on "Requesting Historical Bar Data":
keepUpToDate, Whether a subscription is made to return updates of unfinished real time bars as they are available (True), or all data is returned on a one-time basis (False). Available starting with API v973.03+ and TWS v965+. If True, and endDateTime cannot be specified.
I don't understand exactly what this means. Logically, I had assumed that this function would start a "5 Second Real Time Bars" subscription and close the "Historical Bar Data" subscription after historical bars were sent to the present.
However, this is not what happens. The post "Historical Bar Data" messages received appear randomly, sometimes a couple of seconds apart, sometimes milliseconds apart. The messages almost appear to be Ticks, except that most of these "Ticks" have identical price data.
I think I know what is going here or at least an example that might make sense to folks also trying to figure this out.
Have a look:
The yellow price slider on the Y-Axis moves up and down in sync with the last candle, which mutates with the "live" last price until the candle fully closes and the next one starts. The price updates of the last unfinished candle appear to match the HistoricalDataUpdateMessage data almost exactly.
I have handled the HistoricalDataUpdateMessage to produce similar action as what is seen on the TWS chart for my chart. The results are nearly identical.
Related
I have an alarm set on Lambda iterator age, when it crosses X over 5 minutes, It goes to alarm and I send e-mail to a certain group.
Problem I have is - how to, or where to - setup notification for when alarm is resolved.
We have occasional blips which last only a few minutes more than the alarm itself (e.g. 6-7 minutes) and yes, I could extend and not trigger the alarm, but I'd love to get "Alarm is now resolved" e-mail rather than having people dropping everything and jumping on the problem.
I don't see that option, I tried to copy the same alarm and set it "inverted"
But now this guy is "always on" end "Red" which is not what I want.
And I have the proper Alarm that is currently "not in alarm" and which works as Intended:
So, what are my options here? Do I need composite alarm somehow?
If anyone arrives with the same problem. In late 2021.
AWS UI is not the greatest, so my initial thought was that "Add notification" means, add existing configuration. Sort of "apply".
Turns out, that's the place where you need to add another notification, or multiple ones, e.g. one for OK, one for Insufficient data behaviour.
Just in case anyone ends up with the same problem.
Is there a way to detect a user pausing a run/activity within the strava API?
With Get Activity Streams (getActivityStreams) you can obtain differents StreamSet from your activity: in order to detect pause I think you can analyze CadenceStream or MovingStream.
Pauses are not available in the Strava API and can not be extracted consistently through algorithmic processing of the available fields. Moreover, the data contained in the API's streams collection can not be processed in a way which will arrive at the summary distance or time of the run.
The MovingStream contains a bit field which does not flag pauses, but instead (presumably) flags points where the athlete stopped moving. Although, that said, this field can not be used to arrive at the Moving Time by summing up the time values where this flag is true.
So I'm trying to understand what practical problems Queues solve. By reading all the information from Google, I get the high-level.
Push message to Queue for processing at a later time
So I'm looking at an architecture from Company A and they have different use cases for Job Queueing like for example
chat messages
file conversion
searching
Heavy sql queries
Why process it at a later time?
Here's my best guess...
Let's say I have an application that can process 10 "things" at a time.
My application then maxes out it's processing capacity.
an 11th request came in so app puts it in the Queue for later processing
Assuming this is a valid Use Case, wouldn't adding more servers to process more "things" make sense? Is it because it's more costly to add more servers than employ a Queue and sacrifice response time a little bit?
Given my Use Case examples, what other problems would Queues solve for them?
Have you ever lined up at a bank when it is busy? You would have waited in a queue.
"But," you could say, "wouldn't adding more staff to process more customers make sense? Is it because it's more costly to add more staff than employ a Queue and sacrifice response time a little bit?"
That would be correct. It can be quite costly to staff a bank based on the peak number of customers who would arrive each day. It is cheaper to staff below this level and have some customers wait in a queue.
Also, the number of customers each day are not 100% predictable. A queue allows excess demand to wait without breaking the system.
Queues enable decoupling.
For example, imagine an online store where customers purchase an item. They select the item, provide a credit card number and click 'Purchase'. If the credit card is declined, the online store can immediately prompt them to re-enter the number. This interaction has to take place immediately while the customer is still online.
However, there is no need to have the customer wait while an invoice is generated, a record is added to the accounting system and inventory is pulled off the shelf. This can be decoupled from the ordering process. A good way to do it is to push the order into a queue, which can be handled by the next system.
If that 'next system' happens to be offline at the moment, there is no reason to cancel the whole sale. The transaction can be processed when the 'next system' comes back online. This is much better than failing the whole process just because one component (which is not required immediately) has a failure.
Bottom line: Queues are excellent. They enable better handling of failures. They makes things more resilient (just wait a few minutes and try again!). They should be used at all times when the process is compatible with a queuing architecture.
Let's do scenarios
Scenario 1 without queue:
you request an endpoint /blabla/do-eveything/
this request do
download an image from very slow FTP
e.g 1.5 sec (can error, retry ? add +X sec)
attach the image to an email
send an email (3 sec)
e.g 1 sec (can error, retry ? add +X sec)
confirmation received > store confirmation to a third company tracking stuff
e.g 1.5 (can error, retry ? add +X sec)
when tracking confirm, update your data from another third company for big data purpose
e.g 2 sec (can error, retry ? add +X sec)
... you get the idead
return the response e.g 11 sec later (this is to slow) or more or timeout when everything failed
End user said internet was faster 20 years ago, maybe I need to change my internet connection or change my 16 threads
Scenario 2 queue everything you can:
you request an endpoint /blabla/do-eveything/
this request do
Queue job "DO_EVERYTHING"
e.g 0.02 sec
Return the response less then 0.250 sec
End user said that is website/app is too fast, I can keep my 56K internet connection
on queue/event system one failed job can be retry later without affeting the end user
you can pause job, add a unlimited number a task/step after the original message
better fault tolerance
Working with queue will allow you a better micro/nano service architecture, better testing because, you can test a single job, intead of a full controller that do everything...
Ye, is maybe more work, more thinking, but a the end no need to think about the work when holidays
I have two queues that both have distinct data types that affect one another as they're being processed by my application, therefore processing messages from the two queues asynchronously would cause a data integrity issue.
I'm curious as to the best practice for making sure only one consumer is consuming at any given time. Here is a summary of what I have so far:
EventMessages receive information about external events that may or may not have an impact on the enqueued/existing PurchaseOrderMessages.
Since we anticipate we'll be consuming more PurchaseOrderMessage than EventMessage, maybe we should just ensure the EventMessage Queue is empty (via the API) before we process anything in PurchaseOrderMessage Queue - but that gets into the question of wait times, etc. and this all needs to happen as close to real time as possible.
If there's a way to simply pause a Consumer A until Consumer B is at rest that might be the simplest solution, I'm just not quite sure which direction I need to go in.
UPDATE
To provide some additional context, a PurchaseOrderMessage will contain a Origin and Destination.
A EventMessage also contains location data.
Each time a PurchaseOrderMessage is processed, it will query the current EventMessage records for any Event locations that match the Origin and Destination of that PurchaseOrder and create an association.
Each time an EventMessage is processed, it will query the current PurchaseOrderMessage records for any Origins of Destinations that match that Event and create an association.
If synchronous queues aren't a good solution, what's an alternative that would insure none of the associations are missed when EventMessages and PurchaseOrderMessages are getting published to the app at the same time?
UPDATE 2
Ultimately this data will serve a UI which will have a list of PurchaseOrders and the events that might be affecting their delivery dates. It would be too slow to do the "Event Check" as the PurchaseOrder data was being rendered/retrieved by the end user which is why we're wanting to do it as they're processed/consumed.
Let me begin with the bottom line up front - on the face of it, what you are asking doesn't make sense.
Queues should never require synchronization. The very thought of doing so entirely defeats the purpose of having a queue. For some background, visit this answer.
Let's consider some common places from real life where we encounter multiple queues:
Movie theaters (box office, concession counter, usher)
Theme parks (snack bars, major attractions)
Manufacturing floors (each station may have a queue waiting to process)
In each of these examples, from the point of view of the object in the queue, it can only wait in one at a time. It cannot wait in one line while it is waiting in another- such a thing is physically impossible.
Your example seems to take two completely unrelated things and merge them together. You have a queue for PurchaseOrder objects - but what is the queue for? That's the equivalent of going to Disney World and waiting in the Customer queue - what is the purpose of such a queue? If the purpose is not clear, it's not a real queue.
Addressing your issue
This particular issue needs to be addressed first by clearly defining the various operations that are being done to a PurchaseOrder, then creating queues for each of those operations. If these operations are truly synchronous, then your business logic should be coded to wait for one operation to complete before starting another. In this circumstance, it would be considered an exception if a PurchaseOrder got to the head of one queue without fulfilling a pre-requisite.
Please remember that a message queue typically serves a stateless operation. Good design dictates that messages in the queue contain all the information needed for the processor to process the message. If you don't adhere to this, then your database becomes a single point of contention for your system - and while this is not an insurmountable problem, it does make the design more complex.
Waiting in Multiple Queues
Now, if you've ever been to Disney World, you'll also know that they have something called a FastPass+ (FP+), which allows the holder to skip the line at the designated attraction. Disney allocates a certain number of slots per hour for each major attraction at the park, and guests are able to request up to three FP+s during each day. FP+ times are allocated for one hour blocks, and guests cannot have two overlapping FP+ time blocks. Once all FP+ slots have been issued for the ride, no more are made available. The FP+ system ensures these rules are enforced, independently of the standby queues for each ride. Essentially, by using FastPass+, guests can wait in multiple lines virtually and experience more attractions during their visit.
If you are unable to analyze your design and come up with an alternative, perhaps the FastPass+ approach could help alleviate some of the bottlenecks.
Disclaimer: I don't work for Disney, but I do go multiple times per month, always getting my FastPass first
I have joined a bot in a group to archive the messages. I'm coding a php code for archiving, till this get complete, I manually getupdates from the browser. and save the results in a text file. and frequently I use offset to get newer msgs. but next time when I getupdates after offset (for example one after that) I see that I have a jump! for example if the last id was 1500, after getupdate I see 1553 or 1540, and I donn't see the msgs in the gap. anyone can help me,please?
I may be because old updates are deleted if not requested in a period of time (about 24 hours).
From https://core.telegram.org/bots/api#getting-updates:
Incoming updates are stored on the server until the bot receives them [...], but they will not be kept longer than 24 hours.
What I am not sure is if the lost updates consume update_id values.