I am using the Binance Websocket API and when I subscribe to depth channels (such as via wss://stream.binance.com:9443/stream?streams=btcusdt#depth/), I occasionally get an update that has too many updates for the given first id ('U') and last id ('u'):
{"stream":"btcusdt#depth","data":{"e":"depthUpdate","E":1536066285499,"s":"BTCUSDT",
"U":238495725,
"u":238495734,
"b":[
["7369.47000000","0.00000000",[]], --238495725
["7369.46000000","0.00000000",[]], --238495726
["7369.45000000","0.00000000",[]], --238495727
["7369.19000000","0.90770800",[]], --238495728
["7365.20000000","0.20000000",[]], --238495729
["7363.71000000","0.54130000",[]], --238495730
["7362.94000000","0.00000000",[]], --238495731
["7358.00000000","0.00000000",[]]] --238495732
,"a":[
["7369.47000000","0.02302600",[]], --238495733
["7369.70000000","0.00000000",[]], --238495734
["7408.90000000","3.93000000",[]]] --238495735 WHICH IS NOT OKAY (>u)
}}
I have also noticed that occasionally I will get more ids than updates such as in the following:
{"stream":"btcusdt#depth","data":{"e":"depthUpdate","E":1536068517031,"s":"BTCUSDT",
"U":238514909,
"u":238514912,
"b":[],
"a":[
["7366.54000000","0.00000000",[]], --238514909
["7367.09000000","0.00000000",[]]] --238514910
}}
I am trying to add these updates to a database so having too many ids is less of a problem for me than having too few. If anyone could explain why this happens or how I might be misunderstanding update ids it would be much appreciated.
Thanks in advance!
So it turns out that "update IDs" do not correspond to order IDs and multiple order book entries can be changed in a single update.
Furthermore, the order of the given bids and asks isn't the actual order that they were received but rather they are descending order for bids and ascending for asks and the state of the order book can not be deterministically gotten from the websocket API :/
Related
How should you handle the fact that events received via webhooks can be received in random order ?
For instance, given the following ordered event:
A: invoiceitem.created (with quantity of 1)
B: invoiceitem.updated (with quantity going from 1 to 3)
C: invoiceitem.updated (with quantity going from 3 to 2)
How do you make sure receiving C-A-B does not result in corrupted data (ie with a quantity of 2 instead of 3)?
You could reject the webhook if the previous_attributes in Event#data do not correspond to the current state, but then you are stuck if your local model was updated already, as you will never find yourself in the state expected by the webhook.
Or you can just use treat any webhook as a hint to retrieve and update an object. You just disregard the data sent by the webhook and always retrieve it.
Even if you receive events ordered as update/delete/create it should work, as update would in fact create the object, delete would delete it, and create would fail to retrieve the object and do nothing.
But it feels like a waste of resources to retrieve data each time when the webhook offers it as event data.
This question was asked before but the answers don't cover the above solutions.
Thanks
If your application is sensitive to changes like this that can occur close in time, you really should just use the event as a signal to retrieve the object, as #koopajah noted in their comment. That's the only way to ensure you have the latest state.
I'm working with bus data where each record in the raw data records a bus reaching a stop and how many people got on or off the bus. The raw data also includes which route the bus is for every record, and by creating an ID of [bus route] + [bus stop], I can then reference a manually maintained stop order table so that the order for the stop in the context of the route is available for sorting. E.g order for stop100.route5 = 4; order for stop100.route6 = 8 , etc.
Example of the same stop having a different order here:
The separate table for stop order I mentioned is set up like this (filtered to show different values for the same stop):
Now that I'm trying out PowerBI I'm hitting a bit of a roadblock. I can't sort the stop column by the stop order column, as there are multiple values for each stop depending on the route in question. I know that I can still use stop order and stop as row values and toggle the 'expand all' setting, but my ideal is to hide the stop order numbers, plus in drill-down situations to stop level, the stops will be sorted alphabetically rather than by order number.
For any experts I have a few avenues I thought might be viable workarounds with enough know-how:
Is there a way to hide portions of a field's values in visuals? This whole thing wouldn't be an issue if I could use the stop.route IDs in place of the stop name field, but I would want to hide the .route portion of the value.
Is there any long-winded way to create a one-to-one sorting that I can use to sort the stop column? Some sort of dynamic calculation that filters a one-to-many to a one-to-one, as every context I plan on using this, there will be only one possible order # for the stop.
Many thank yous to anyone with advice!
Yodlee seems to be returning running balance for each transaction returned in the API response but here i am facing an issue where i am not able to determine the correct order of transactions and hence the running balance displayed in my app does'nt make any sense because the transactions are not in correct order i.e. the order in which they happened # bank.
Is there any solution to this?
I will really appreciate a proper solutions or any work around for this issue
Note
I have already tried to sort by the id returned for each transaction but it results in incorrect order of transactions
The order can be determined by the transactionDate if returned , Yodlee does not know what order the transactions are nor does it store chronologically.
Its dependent on the end Site from where its scraped and most of the banks are missing the transaction Date .
(I tried posting this to the CFWheels Google Group (twice), but for some reason my message never appears. Is that list moderated?)
Here's my problem: I'm working on a social networking app in CF on Wheels, not too dissimilar from the one we're all familiar with in Chris Peters's awesome tutorials. In mine, though, I'm required to display the most recent status message in the user directory. I've got a User model with hasMany("statuses") and a Status model with belongsTo("user"). So here's the code I started with:
users = model("user").findAll(include="userprofile, statuses");
This of course returns one record for every status message in the statuses table. Massive overkill. So next I try:
users = model("user").findAll(include="userprofile, statuses", group="users.id");
Getting closer, but now we're getting the first status record for each user (the lowest status.id), when I want to select for the most recent status. I think in straight SQL I would use a subquery to reorder the statuses first, but that's not available to me in the Wheels ORM. So is there another clean way to achieve this, or will I have to drag a huge query result or object the statuses into my CFML and then filter them out while I loop?
You can grab the most recent status using a calculated property:
// models/User.cfc
function init() {
property(
name="mostRecentStatusMessage",
sql="SELECT message FROM statuses WHERE userid = users.id ORDER BY createdat DESC LIMIT 1,1"
);
}
Of course, the syntax of the SELECT statement will depend on your RDBMS, but that should get you started.
The downside is that you'll need to create a calculated property for each column that you need available in your query.
The other option is to create a method in your model and write custom SQL in <cfquery> tags. That way is perfectly valid as well.
I don't know your exact DB schema, but shouldn't your findAll() look more like something such as this:
statuses = model("status").findAll(include="userprofile(user)", where="userid = users.id");
That should get all statuses from a specific user...or is it that you need it for all users? I'm finding your question a little tricky to work out. What is it you're exactly trying to get returned?
I have a table that has ordernumber, cancelled date and reason.
Reason field is varchar(255) field and it was written by many different sales rep and really hard to group by the reason category I need to generate a report to categorize cancelation reasons. What is the best way to analyse the reasons with TSQL?
Sample of reasons entered by sales rep
cust already has this order going out
cust can not hold for item Called to cancel order
cust doesn't want to pay for shipping
wife ordered same item from different vendor, sent email
cst made a duplicate order, sent email
cst can't hold
Cust doesn't want to go through verification process so is cancelling order
doesn't ant to hold for Bo
doesn't want
Cust called to cancel the order He can no longer get the product he wants
cnt hld
will not comply with export req
cant' hold
Custs request
Cust will not hold for BO
per. cust. request.
BTW I have SQL Server 2005.
part of your problem is that this these aren't truly reason codes. sounds like an issue with your schema to me. if there aren't predefined reason codes to reference and you're allowing free text entry for each reason, then there's really no way to do this directly, outside of pulling distinct reasons back, which is probably not going to be very useful.
just an idea, can you add another column to the table, even if it's in a temp or test environment and then give the business users the ability to assign a code (e.g. 1 for mis-ships, 2 for duplicate orders, 3 for wrong item etc.) to each order cancellation. then perform the analysis on that.
i assume that's what they're expecting from you, but i don't know that i see any better way. you could always perform the analysis yourself if you have the authority/knowledge but this might be painful if you have a ton of cancellations.
edit- i see now that you've tagged this with regex... it would be possible to setup specified keywords to pull out the entries, but there'd have to be some tolerance built in and still manual analysis afterwards for items which don't fall into any specified category due to misspellings etc. /edit
+1 to #jmatthews, you really need to have reason codes that are selected and then possibly allow free-form entry for the full reason.
If this isn't an option you can look into text clustering. Don't expect that to be fast or easy though, it's still an open research topic and is related to both AI and machine learning.
Look at Term Lookup in SSIS, here is an article to read.