I have REST backend api, and front end will call api to get data.
I was wondering how REST api handles continuous data update, for example,
in jenkins, we will see that if we execute build job, we can see the continous log output on page until job finishes. How REST accomplish that?
Jenkins will just continue to send data. That's it. It simply carries on sending (at least that's what I'd presume it does). Normally the response contains a header field indicating how much data the response contains (Content-Length). But this field is not necessary. The server can omit it. In such a case the response body ends when the server closes the connection. See RFC 7230:
Otherwise, this is a response message without a declared message body length, so the message body length is determined by the number of octets received prior to the server closing the connection.
Another possibility would be to use the chunked transfer encoding. Then the server sends a chunk of data having its own Content-Length header. The server terminates this by sending a zero-length last chunk.
Websocksts would be a third possibility.
I was searching for an answer myself and then the obvious solution struck me. In order to see what type of communication a service is using, you can simply view it from browser side using Developer Tools.
In Google Chrome it will be F12 -> Network.
In case of Jenkins, front-end is sending AJAX requests to backend for data:
every 5 seconds on Dashboards page
every second during Pipeline run (Console Output page), that you have mentioned.
I have also checked the approach in AWS. When checking the status of instances (example: Initializing... , Booting...), it queries the backend every second. It seems to be a standard interval for its services.
Additional note:
When running an AWS Remote Console though, it first sends requests for remote console instance status (backend answers with { status: "BOOTING" }, etc.). After backend returns status as "RUNNING", it starts a WebSocket session between your browser and AWS backend (you can notice it by applying WS filter in developer tools).
Then it is no longer REST API, but WebSockets, that is a different protocol (stateful).
Related
I have long operation, which called via Web API. Status code 102 says to us:
An interim response used to inform the client that the server has
accepted the complete request, but has not yet completed it.
This status code SHOULD only be sent when the server has a reasonable
expectation that the request will take significant time to complete.
As guidance, if a method is taking longer than 20 seconds (a
reasonable, but arbitrary value) to process the server SHOULD return a
102 (Processing) response. The server MUST send a final response after
the request has been completed.
So, I want to return 102 status code to client, then client waits response about result of operation. How to implement it on .NET?
I read this thread: How To Return Http 102 Processing in Asp.Net Web Api?
This thread has good explanation what is necessary, but no response. I don't understand how it implement on .NET, not theory...
Using HTTP 102 requires that the server send two responses for one request. ASP.NET (Core or not) does not support sending a response to the client without completely ending the request. Any attempt to send two responses will end up in throwing an exception and just not working. (I tried a couple different ways)
There's a good discussion here about how it's not actually in the HTTP spec, so implementing it isn't really required.
There are a couple alternatives I can think of:
Use web sockets (a persistent connection that allows data to be sent back and forth), like with SignalR, for example.
If your request takes a long time because it's getting data from elsewhere, you can try pulling in that data via a stream and send it to the client via a stream. That will send the data as it's coming in, rather than loading it all into memory first before sending it. Here's an example of streaming data from a database to the response: https://stackoverflow.com/a/45682190/1202807
I've built up an API application with ASP.NET Core 2.2.
Everything has been fine. Except one PATCH API, which takes an ID and a list, to replace the list of corresponding item.
This API works fine with POSTMAN too. Simply and fast, works just as expected.
However, to run on browsers, it stalls 1 minute to send that request.
I've tried to make it simple by rewriting the App within only one jQuery function, to check if the problem is on my frontend app; however it still stalls for 1 minute.
I've looked up stalled, people say that it can be a Chrome policy to load maximum 6 requests at the same time; however it's not my case. There's only such request at that time, and every other API works fine except this one.
Also, I've tried with other browsers: Firefox and Edge, but it's still the same.
According to the article Chrome provides:
Queueing. The browser queues requests when:
There are higher priority requests.
There are already six TCP connections open for this origin, which is the limit. Applies to HTTP/1.0 and > HTTP/1.1 only.
The browser is briefly allocating space in the disk cache
Stalled. The request could be stalled for any of the reasons described in Queueing.
It seems that getting "stalled" for long, means that the request wasn't event sent. Does it mean that I can just exclude the possibility to fix backend API?
And also, since that there's no other request at the same time, does it mean that it most likely goes to the reason that "The browser is briefly allocating space in the disk cache", or is there any other reason?
And I also wander why only this API gets this issue. Is there anything special with the method "PATCH"?
At first use stopwatch and evaluate response time of your code in browser and postman and see how take long time in each.
If both is same, don't touch your code because your problem isn't from your method.
If you can, test it with 'post http attribute' till know your problem is because of it or not.
However I guess reason of it is your system.
Of course it's possible ypur problem resolve with changing pipeline (startup.cs) . There are also problems like CORS that occurred only in browsers and not postman.
I am building an application which uses the Amazon MWS API.
The API has limits for how frequently you can hit it.
I am looking for a tool that can act as a reverse-proxy, save the MWS API responses, and eventually masquerade as the MWS API without ever hitting it, returning only responses from the cache.
Some tools do this, but what I need is a bit more complicated.
Say I request a report from Amazon MWS:
I'll call RequestReport
I'll get a ReportRequestId back
I'll start server polling GetReportRequestList to find out what the current status of the report request is. The report request will go likely go through the statuses SUBMITTED then DONE, but it could also be set to ERROR or CANCELLED
When the report request status returned by GetReportRequestList is DONE, I can finally call GetReport and get the data.
The behavior from step 3 is what I'm trying to replicate.
This external API cache should be able to produce different results for the same request: the first response should yield SUBMITTED and then the second response should yield DONE.
I should be able to easily configure these flows as I wish, setting the responses I want for the 1st, 2nd, nth request.
I would like this tool to necessitate minimal configuration, I do not want to configure routes or anything, I want it to automatically cache everything and then return everything from the cache, never flushing it.
Also, I need this level of control over what's returned in a response, depending on the count of requests done up to that point.
I am trying to understand, why does Netty SSL mode work on strange way?
Also, the problem is following, when any SSL client(https browser, java client using ssl, also any ssl client application) connects to Netty server I get on beginning the full message, where I can recognize correctly the protocol used, but as long the channel stays connected, any following messages have strange structure, what is not happening same way with non-ssl mode.
As example on messageReceived method when the https browser connects to my server:
I have used PortUnificationServerHandler to switch protocols.. (without using nettys http handler, it is just example, because i use ssl mode for my own protocol too)
first message is ok, I get full header beginning with GET or POST
than I send response...
second message is only one byte long and contains "G" or "P" only.
third message is than the rest beginning either with ET or OST and the rest of http header and body..
here again follows my response...
fourth message is again one byte long and again contains only one byte..
fifth message again the rest... and on this way the game goes further..
here it is not important, which sub protocol is used, http or any else, after first message I get firstly one byte and on second message the rest of the request..
I wanted to build some art of proxy, get ssl data and send it unencoded on other listener, but when I do it directly without waiting for full data request, the target listener(http server as example) can not handle such data, if the target gets one byte as first only (even if the next message contains the rest), the channel gets immediately closed and request gets abandoned..
Ok, first though would be to do following, cache the first byte temporarily and wait for next message and than join those messages, and only than response, that works fine, but sometimes that is not correct approach, because the one byte is sometimes really the last message byte, and if i cache it and await wrongly next message, i can wait forever, because the https browser expects at this time some response and does not send any data more..
Now the question, is it possible to fix this problem with SSL? May be there are special settings having influence on this behavior?
I want fully joined message at once as is and not firstly first byte and than the rest..
Can you please confirm, that with newer Netty versions you have same behaving by using PortUnificationServerHandler (but without netty http handler, try some own handler.)
Is this behavior Ok so, I do not believe, it was projected so to work..
What you're experiencing is likely to be due to the countermeasures against the BEAST attack.
This isn't a problem. What seems to be the problem is that you're assuming that you're meant to read data in terms of messages/packets. This is not the case: TCP (and TLS/SSL) are meant to be used as streams of continuous data. You should keep reading data while data is available. Where to split incoming data where it's meaningful is guided by the application protocol. For HTTP, the indications are the blank line after the header and the Content-Length or chunked transfer encoding for the entity.
If you define your own protocol, you'll need a similar mechanism, whether you use plain HTTP or SSL/TLS. Assuming you don't need it only works by chance.
I had experienced this issue and found it was caused bu using JDK1.7. Moving back to JDK1.6 solved it. I did not have time to investigate further but have assumed for now that the SSLEngine implementation has changed in the JDK. I will investigate further when time permits.
I am trying to implement an API which uses the long-polling concept in Symfony framework.
Let's say that I have a table 'feeds' which can only grow (assume that users can insert thier feed from other interface).
I want to create a client-side real-time updated page. The idea is the following:
Client send an ajax request with timestamp of last modification (first time sends 0)
Server compares timestamp of client to timestamp, to retrieve all messages with bigger timestamp than the one sent by user
If there are newer messages, return them immediately to the client, with the timestamp of the latest one
On other hand, if there are no new messages, enter into a 2 minutes busy-wait loop, checking every 1-3 seconds (randomly) whether there are new messages.
When client receive servers answer, browser updates view and immediately sends a new ajax request.
In other words, instead of send an AJAX call every x seconds, the server holds the request till it has new information for us.
Having good experience with Symfony I tried to implement a simple demo of this api, and it works great. I had a problem of session blocking (the ajax call is held so access to the server is not possible), so I simply added the following to the action:
public function executeIndex(sfWebRequest $request)
{
session_write_close();
:
:
(see also this link)
Then I testes massive access to the API. 100 users works fine, 1000 everything crashes.
I realized that I have two problems:
For each access a new DB connection is opened
For each access the server executes a new process
For the first problem I tried to put persistent: true In my database.yml Doctrine connetor. When I monitored the server connections I saw that still each access to the API opens a new connection. So basically I am still blocked with the same two problems.
Does anyone have any idea or experience with this issue?? Or maybe I should give-up the idea of implementing my api with Symfony??
I think using symfony for this, is the wrong approach. Using Sockets would be much easier.
For example have a look at nodejs or ape-project (comet)
they both are able to handle much more current users than apache, lighttpd or nginx...
Apache creating different threads for each user and each thread have a separate database connection. that's why the db connection are high