We have a RESTful API built using ASP.NET Web API, and it is hosted as a Azure cloud service.
Recently we had to fix the performance (response time) of an endpoint owing to which we made a few changes. The API request-response needed to remain unchanged.
Thus to test that the changes we made didn't alter the response, we benchmarked the responses by capturing it for different users. We captured the following -
Response times (Postman display)
Response size (Postman display)
Response body
Now that we are testing, oddly we see that although the response body is an exact match (done using file compare) the response sizes are order of magnitude different. For instance what was 562.37KB before is now 52.33KB. In fact we had benchmarked 30 users and all the response sizes reduced by one order. But for all the response body is exactly the same.
What could be the possible reason? Is there anything we are missing?
Size is just the response size when it will be saved inside the memory. This response size is the size of complete response and headers and cookies and everything that has been sent along with the response.
NOTE: The response size that is shown in the Postman is approximate response size and not the exact size.
For details you may refer to
https://www.toolsqa.com/postman/response-in-postman/
https://github.com/postmanlabs/postman-app-support/issues/156
Secondly, it is important to know the difference as detailed in link size and content :
Chrome Dev Tools - "Size" vs "Content"
For easy access, a snapshot of the answer is below:
"Size" is the number of bytes on the wire, and "content" is the actual size of the resource. A number of things can make them different, including:
Being served from cache (small or 0 "size")
Response headers, including cookies (larger "size" than "content")
Redirects or authentication requests
gzip compression (smaller "size" than "content", usually
Related
We have an API endpoint that allows users to upload images; one of its parameters is an IFormFileCollection.
We'd like to validate the file size to make sure that the endpoint isn't being abused so I'm checking the Length property of each IFormFile, but I don't know whether I can trust this property or not, i.e. does this come from the request? Is it considered 'input', much like Content-Length is?
If you have an IFormFileCollection parameter, and you send data using a "form-data" content-type in the request, that parameter will be bound by a whole lot of plumbing that's hard to dig through online, but if you just debug the action method that accepts the IFormFileCollection (or any collection of IFormFile, really)and inspect the collection, you'll see that the uploaded files will already have been saved on your server's disk.
That's because the entire multi-part form request's body has to be read to determine how many files there are, if any, and form parameters, and validate the request body's format while it's reading it.
So yes, by the time your code ends up there, you can trust IFormFile.Length, because it's pointing to a local file that exists and contains that many bytes.
You're too late there to reject the request though, as it's been already entirely read. You better fix rate and size limits lower in the stack, like on the web server or firewall.
Content-Length is compressed number of bytes of data in the body , it is not reliable since it may include extra data ,for example , you are sending multipart request . Just use the IFormFile.length for features like calculation or validation .
I'm trying to create new pages in OneNote using Microsoft Graph REST API (Objective-C). Those new pages should contain a PDF document as an attachment.
The POST operations succeed with PDF files that are under ~4MB. However, the operations for files that are over 4MB fail with the error message request too large (413) and the following response:
{
"error": {
"code": "BadRequest",
"message": "Maximum request length exceeded.",
"innerError": {
"request-id": "269c663c-9289-47cc-a833-d471b7b867f6",
"date": "2019-04-09T09:35:49"
}
}
}
The end point that is used is: https://graph.microsoft.com/v1.0/me/onenote/sections/XXX/pages
Microsoft states in the documentation for Graph and OneNote:
The total POST size limit is ~70 MB, including images, files, and other data. The actual limit is affected by downstream encoding, so there's no fixed byte-count limit. Requests that exceed the limit may produce unreliable results.
The limit for each data part is 25 MB, including the part headers. Data parts that exceed the limit are rejected by Microsoft Graph.
I could not find any limit of 4MB for POST requests in the Microsoft Graph documentation for OneNote. Is there any workaround for my current issue?
The short answer is that this documentation is incorrect. The longer answer requires a little background on Graph itself.
Graph is an API aggregator. It takes incoming requests, routes them to one or more underlying APIs, and then normalizes the responses so the end developer gets consistent results across endpoints.
In this case, the underlying API is the OneNote REST API. The limits mentioned in the docs are accurate in terms of the OneNote API but since this request is being handled by Graph, you are hitting the smaller 4 MB limit of Graph itself before that request ever gets routed to the underlying API.
More than likely the documentation error was unintentionally missed when the original docs were ported over to Graph. I've filed a documentation issue so it can be corrected.
https://github.com/request/request-promise
I want to measure and possibly minimise the payload of response.
So I want to get content-length header. I will not get header unles i specify resolveWithFullResponse to request. The question is, how it works, in detail? Can it make server response larger in terms of network load?
When i get specific page with browser, i got different (3 times smaller, in my case) content-length header value, compared to getting the same page with request-promise with resolveWithFullResponse: true.
So is request-promise just trim response it gets, if you do not specify resoveWithFullResponse, or it requests more data in case it is set?
resolveWithFullResponse just means that instead of receiving the response body in your .then, you'll receive the full IncomingMessage object for the response, with all the headers, status codes, and other attached information instead.
It does not affect the network request being made.
I have an API for a file upload that expects a multipart form submission. But I have a customer writing a client and his system can't properly generate a multipart/form-data request. He's asking that I modify my API to accept the file in a application/x-www-form-urlencoded request, with the filename in one key/value pair and the contents of the file, base64 encoded, in another key/value pair.
In principle I can easily do this (tho I need a shower afterwards), but I'm worried about size limits. The files we expect in Production will be fairly large: 5-10MB, sometimes up to 20MB. I can't find anything that tells me about length limitations on individual key/value pair data inside a form POST, either in specs (I've looked at, among others, the HTTP spec and the Forms spec) or in a specific implementation (my API runs on a Java application server, Jetty, with an Apache HTTP server in front of it).
What is the technical and practical limit for an individual value in a key/value pair in a form POST?
There are artificial limits, configurations, present on the HttpConfiguration class. Both for maximum number of keys, and maximum size of the request body content.
In practical terms, this is a really bad idea.
You'll have a String, which uses 2-bytes per character for the Base64 data.
And you have the typical 33% overhead just being Base64.
They'll also have to utf8 urlencode the Base64 string for various special characters (such as "+" which has meaning in Base64, but is space " " in urlencoded form. So they'll need to encode that "+" to "%2B").
So for a 20MB file you'll have ...
20,971,520 bytes of raw data, represented as 27,892,122 characters in raw Base64, using (on average) 29,286,728 characters when urlencoded, which will use 58,573,455 bytes of memory in its String form.
The decoding process on Jetty will take the incoming raw urlencoded bytes and allocate 2x that size in a String before decoding the urlencoded form. So that's a 58,573,456 length java.lang.String (that uses 117,146,912 bytes of heap memory for the String, and don't forget the 29MB of bytebuffer data being held too!) just to decode that Base64 binary file as a value in a x-www-form-urlencoded String form.
I would push back and force them to use multipart/form-data properly. There are tons of good libraries to generate that form-data properly.
If they are using Java, tell them to use the httpmime library from the Apache HttpComponents project (they don't have to have/use/install Apache Http Client to use the httpmime, its a standalone library).
Alternative Approach
There's nothing saying you have to use application/x-www-form-urlecnoded or multipart/form-data.
Offer a raw upload option via application/octet-stream
They use POST, and MUST include the following valid request headers ...
Connection: close
Content-Type: application/octet-stream
Content-Length: <whatever_size_the_content_is>
Connection: close to indicate when the http protocol is complete.
Content-Type: application/octet-stream means Jetty will not process that content as request parameters and will not apply charset translations to it.
Content-Length is required to ensure that the entire file is sent/received.
Then just stream the raw binary bytes to you.
This is just for the file contents, if you have other information that needs to be passed in (such as filename) consider using either the query parameters for that, or a custom request header (eg: X-Filename: secretsauce.doc)
On your servlet, you just use HttpServletRequest.getInputStream() to obtain those bytes, and you use the Content-Length variable to verify that you received the entire file.
Optionally, you can make them provide a SHA1 hash in the request headers, like X-Sha1Sum: bed0213d7b167aa9c1734a236f798659395e4e19 which you then use on your side to verify that the entire file was sent/received properly.
A POST to a specific end point allows to upload an image except if the image is too large, so I want to return the appropiate http status code response in that case.
A http status code 400 response it does not seem to fit well in this case.
400 Bad Request: "The server cannot or will not process the request due
to something that is perceived to be a client error (e.g., malformed
request syntax, invalid request message framing, or deceptive request
routing).
I think that the image being too large it does not imply that the request is malformed or syntactically incorrect.
Any suggestions?
This seems like it would be an ideal candidate for 413 Payload Too Large. From Section 6.5.11 of RFC 7231:
The 413 (Payload Too Large) status code indicates that the server is
refusing to process a request because the request payload is larger
than the server is willing or able to process.
You can use 420 or even 422, but I would avoid that until you have really good reason to have separate code for it. Usually is better to keep number of different status codes rather small. Check top 10 on that list: http://www.restapitutorial.com/httpstatuscodes.html
You should avoid using more than 10 codes, because your API will become too complex.
So my answer is: use 400 with proper error message returned to the client like: "Image too large, you can upload files up to XX MB"