I'm using the Autodesk Forge - Model Derivative service to convert a Revit (.rvt) file to IFC and SVF formats. To be notified when these conversions are done I have set up a webhook using the webhooks system Autodesk created for this purpose.
Now, when the conversion is done, the webhook succesfully sends a request to my given callback URL, but it does it more than once. And even the amount of times it fires isn't constant. It just fired the webhook twice in a minute, and then again after ten minutes. What could be a reason for this?
The problem is, that the first time the webhook fires, I can't even access the completed derivatives completely yet (I can't load the hierarchy and properties yet). It seems to be an issue on Autodesk's side, but I was wondering if anyone else encountered this problem.
This is the webhook I've created, and the only one on my account:
"links": {
"next": null
},
"data": [
{
"hookId": "<<hook id>>",
"tenant": "jobstarted",
"callbackUrl": "<<my url>>/callback",
"createdBy": "2UbAPVbvWGfLAPHWLZD7Mld0bVRpI8aJ",
"event": "extraction.finished",
"createdDate": "2019-07-29T09:06:46.971+0000",
"system": "derivative",
"creatorType": "Application",
"status": "active",
"scope": {
"workflow": "jobstarted"
},
"urn": "<<hook urn>>",
"__self__": "/systems/derivative/events/extraction.finished/hooks/<<hook id>>"
}
]
}
What I expect, is for the webhook to fire only once, so I can trust that I can access, download and use the entire derivative's data. Currently it's just unreliable.
Such behavior typically happens when our service did not receive a successful response from your end to the callbacks. Try divert the callbacks to a third party service (e.g. ngrok) and record them to isolate the issue. And see here for more details:
Webhooks guarantees at least once delivery. When the event occurs, the webhooks service sends a payload to the callback URL as an HTTP POST request. The webhook service expects a 2xx response to the HTTP POST request. The response must be received within 6 seconds. A non-2xx response is considered an error. In the event of an error, the webhook service will retry immediately, in 15 minutes, and 45 minutes thereafter. The webhook service retries for 48 hours and disables the webhook if the callback does not succeed during this time. You may need to reconfigure your webhooks if they are disabled.
Related
Reading the docs I found:
XMPP server throttling
We limit the rate that you can connect to FCM XMPP servers to 400 connections per minute per project. This shouldn't be an issue for message delivery, but it is important for ensuring the stability of our system.
For each project, FCM allows 2500 connections in parallel.
https://firebase.google.com/docs/cloud-messaging/concept-options#xmpp_throttling
Also within that page there was a description of different ways to connect to FCM to send messages. HTML and XMPP are different mechanisms, so I am assuming that the admin SDK (Golang in my case) uses HTTP under the hood and not XMPP so please correct me if that's not true.
If the admin SDK uses HTTP, that means there can only be 2500 simultaneous connections.
I'm making a scalable application where users basically define their own schedule for notifications (and the messages) and a server retrieves it, runs on a timer loop every 30 seconds or so to see who needs their message sent.
For all intents and purposes, each notification is different. However the vast majority of these notifications will land on the hour. Meaning my server will have to send possibly many thousands of notifications within the X:00 minute in the hour. It's important these notifications come on time (ie, I cannot space them all out within the hour).
Using workarounds like topics won't work in my case because everyone is individual.
I'm just thinking of options to deal with these limitations (and make sure I understand them). If FCM allows 2500 connections in parallel via the admin SDK in Go, can I do 2500 async connections, wait until they all finish, and do another 2500, rinse and repeat? That way if I have 25,000 subscribed users let's say, and each takes 1 seconds, I could theoretically send all the notifications in 10 seconds, which is acceptable.
Are there any other rate limits that I need to be aware of?
Thanks!
I am assuming that the admin SDK (Golang in my case) uses HTTP under the hood
That is correct. The Admin SDKs use the versioned HTTP API to make calls to FCM.
The key to scaling your FCM usage is to use the resources efficiently. For example in the versioned API (that the Admin SDKs use under the hood) you can pass up to 500 requests over a single HTTP connection, meaning that you can amortize the cost of building the connection over many calls.
You can find an example of the actual HTTP calls in the REST example in the documentation on sending messages to multiple devices:
--subrequest_boundary
Content-Type: application/http
Content-Transfer-Encoding: binary
Authorization: Bearer ya29.ElqKBGN2Ri_Uz...HnS_uNreA
POST /v1/projects/myproject-b5ae1/messages:send
Content-Type: application/json
accept: application/json
{
"message":{
"token":"bk3RNwTe3H0:CI2k_HHwgIpoDKCIZvvDMExUdFQ3P1...",
"notification":{
"title":"FCM Message",
"body":"This is an FCM notification message!"
}
}
}
...
--subrequest_boundary
Content-Type: application/http
Content-Transfer-Encoding: binary
Authorization: Bearer ya29.ElqKBGN2Ri_Uz...HnS_uNreA
POST /v1/projects/myproject-b5ae1/messages:send
Content-Type: application/json
accept: application/json
{
"message":{
"token":"cR1rjyj4_Kc:APA91bGusqbypSuMdsh7jSNrW4nzsM...",
"notification":{
"title":"FCM Message",
"body":"This is an FCM notification message!"
}
}
}
--subrequest_boundary--
In the Go Admin SDK this'd be equivalent to calling sendAll, which:
func (c Client) SendAll(ctx context.Context, messages []*Message) (*BatchResponse, error)
SendAll sends the messages in the given array via Firebase Cloud Messaging.
The messages array may contain up to 500 messages. SendAll employs batching to send the entire array of [messages] as a single RPC call. Compared to the Send() function, this is a significantly more efficient way to send multiple messages. The responses list obtained from the return value corresponds to the order of the input messages. An error from SendAll indicates a total failure -- i.e. none of the messages in the array could be sent. Partial failures are indicated by a BatchResponse return value.
Situation
I am preparing a JMeter test plan to test against AWS Device provisioning MQTT API.
The flow is:
connect to AWS IoT Core endpoint
make CreateKeysAndCertificate request until receiving a success response as the below one (as there is a 10 times requests per second quota limit in aws)
{
"certificateId": "string",
"certificatePem": "string",
"privateKey": "string",
"certificateOwnershipToken": "string"
}
extract certificateOwnershipToken from the success response to make RegisterThing request until receiving a success response (as there is a 10 times requests per second quota limit in aws)
{
"certificateOwnershipToken": "string",
"parameters": {
"string": "string",
...
}
}
If the no. of requests exceeds AWS quota limits, the response would be as the below one
{
"statusCode":412,
"errorCode": "Throttled",
"errorMessage": "Rate limit exceeded"
}
Problems in my test plan
My test plan is as below pics, there are two problems
I want to make 500 success requests, but now the test plan only make 500 requests(including those fails)
I need to use certificateOwnershipToken from each success request for each thread to make RegisterThing request, but now certificateOwnershipToken would be the default value not_found even if CreateKeysAndCertificate request is successful
What are the problems in my test plan and how to fix them? Thank you.
AWS Device provisioning MQTT API docs
JMeter MQTT plugin github page
There are following problems in your Test Plan:
The condition in While Controller should be inverted, currently it repeats the request until certificateOwnershipToken is not equal to not_found and my expectation is that you should repeat the request until it equals and stop repeating the request once the value will differ. Moreover you should be using __jexl3() or __groovy() function for performance reasons instead of __javaScript().
If there is a rate limiting for the certificateOwnershipToken request you shouldn't be sending requests at the rate above this limit, consider using an appropriate JMeter Timer like Precise Throughput Timer so JMeter would send at most 10 requests per second
I have long operation, which called via Web API. Status code 102 says to us:
An interim response used to inform the client that the server has
accepted the complete request, but has not yet completed it.
This status code SHOULD only be sent when the server has a reasonable
expectation that the request will take significant time to complete.
As guidance, if a method is taking longer than 20 seconds (a
reasonable, but arbitrary value) to process the server SHOULD return a
102 (Processing) response. The server MUST send a final response after
the request has been completed.
So, I want to return 102 status code to client, then client waits response about result of operation. How to implement it on .NET?
I read this thread: How To Return Http 102 Processing in Asp.Net Web Api?
This thread has good explanation what is necessary, but no response. I don't understand how it implement on .NET, not theory...
Using HTTP 102 requires that the server send two responses for one request. ASP.NET (Core or not) does not support sending a response to the client without completely ending the request. Any attempt to send two responses will end up in throwing an exception and just not working. (I tried a couple different ways)
There's a good discussion here about how it's not actually in the HTTP spec, so implementing it isn't really required.
There are a couple alternatives I can think of:
Use web sockets (a persistent connection that allows data to be sent back and forth), like with SignalR, for example.
If your request takes a long time because it's getting data from elsewhere, you can try pulling in that data via a stream and send it to the client via a stream. That will send the data as it's coming in, rather than loading it all into memory first before sending it. Here's an example of streaming data from a database to the response: https://stackoverflow.com/a/45682190/1202807
We have implemented a Logic app to call do a POST call to a third-party API which returns a 202 with location header. The Logic app in the backend automatically polls using the location header resulting in GET request to the third-party provider hoping to receive a 200 response once the processing is complete. However, the GET requests are resulting in 404 errors.
We have tried disabling the check location headers but for some reason Logic apps still continues to send the GET requests and at a faster rate.
Is there any way to stop the GET request from Logic Apps or should this be the third-party provider's responsibility to handle the polling and not send 404's?
Yes, you can stop the GET request from your Logic Apps. Basically it totally depends on your workflow. If you are designing a stateful workflow then I would suggest that not to stop the GET request.
For stateful workflow all HTTP-based actions follow the standard asynchronous operation pattern as the default behavior. Where after an HTTP action calls or sends a request to an endpoint or API, the receiver immediately returns a "202 ACCEPTED" response. And the response can include a location header which the caller can use to poll or check the status for the asynchronous request until the receiver stops processing and returns a "200 OK" success response or other non-202 response.
But if you are designing a stateless workflow, then caller doesn't have to wait for the request to finish processing and can continue to run the next action. In this case the receiver return the "202 ACCEPTED" response as-is, and proceed to the next step in the workflow execution. A stateless workflow won't poll the specified URI to check the status.
You can stop the GET request from your logic app by following any of the two approaches mentioned below.
Turn off Asynchronous Pattern setting.
You can achieve this by going to the Logic App Designer, on the HTTP action's title bar, selecting the ellipses (...) button and setting Asynchronous Pattern to Off if enabled.
Disable asynchronous pattern in HTTP action's JSON definition.
In the HTTP action's underlying JSON definition, add the "DisableAsyncPattern" operation option to the action's definition so that the action follows the synchronous operation pattern. Check this document for more information.
Also check this Asynchronous request-response behavior document by Microsoft for more understanding.
I was able to use square's webhook API based on descriptions here, https://docs.connect.squareup.com/api/connect/v1#webhooks-overview
and payment webhook was working fine.
Recently, I noticed that after completing a cash payment my webhook event handler
is not receiving any PAYMENT_UPDATED notifications.
I'm able to get the Test Webhook Notification trigger with my event handler service and I did register the PAYMENT_UPDATED webhook for my location.
This service was working before, is there any new changes for square-connect api?
There is no guarantee that the webhooks notification will successfully go through. If it fails for any reason, Square will not attempt to resend it. You should definitely use alternate methods (such as the ListTransactions endpoint) to fully verify the data.