Android Volley: gzip response - gzip

What type of response listener must we use to handle gzip responses with Android Volley?
If a String listener is used, the response seems to lose its encoding.
How do you handle gzip responses using Volley?
MAJOR EDIT:
HttpUrlConnection automatically adds the gzip header to requests, and if the response is gzipped, it will seamlessly decode it and present to you the response. All the gzip stuff happens behind the scenes and you don't need to do what I posted in a gist as an answer to this question. See the documentation here http://developer.android.com/reference/java/net/HttpURLConnection.html
As a matter of fact, the answer I posted SHOULD NOT be used, because the gzip decoding is extremely slow, and should be left to be handled by HttpUrlConnection.
Here is the exact piece from the documentation:
By default, this implementation of HttpURLConnection requests that
servers use gzip compression. Since getContentLength() returns the
number of bytes transmitted, you cannot use that method to predict how
many bytes can be read from getInputStream(). Instead, read that
stream until it is exhausted: when read() returns -1. Gzip compression
can be disabled by setting the acceptable encodings in the request
header:
urlConnection.setRequestProperty("Accept-Encoding", "identity");

So I figured out how to do this.
Basically, I extended StringRequest so that it handles the network response a different way.
You can just parse the response bytearray using GZipInputStream and return the resultant string.
Here's the gist: https://gist.github.com/premnirmal/8526542

You should use ion ,, it's comes pre-set with
Transparent usage of HTTP features and optimizations:
SPDY and HTTP/2
Caching
Gzip/Deflate Compression
Connection pooling/reuse via HTTP Connection: keep-alive
Uses the best/stablest connection from a server if it has multiple IP addresses
Cookies

Related

is there a size limit to individual fields in HTTP POST?

I have an API for a file upload that expects a multipart form submission. But I have a customer writing a client and his system can't properly generate a multipart/form-data request. He's asking that I modify my API to accept the file in a application/x-www-form-urlencoded request, with the filename in one key/value pair and the contents of the file, base64 encoded, in another key/value pair.
In principle I can easily do this (tho I need a shower afterwards), but I'm worried about size limits. The files we expect in Production will be fairly large: 5-10MB, sometimes up to 20MB. I can't find anything that tells me about length limitations on individual key/value pair data inside a form POST, either in specs (I've looked at, among others, the HTTP spec and the Forms spec) or in a specific implementation (my API runs on a Java application server, Jetty, with an Apache HTTP server in front of it).
What is the technical and practical limit for an individual value in a key/value pair in a form POST?
There are artificial limits, configurations, present on the HttpConfiguration class. Both for maximum number of keys, and maximum size of the request body content.
In practical terms, this is a really bad idea.
You'll have a String, which uses 2-bytes per character for the Base64 data.
And you have the typical 33% overhead just being Base64.
They'll also have to utf8 urlencode the Base64 string for various special characters (such as "+" which has meaning in Base64, but is space " " in urlencoded form. So they'll need to encode that "+" to "%2B").
So for a 20MB file you'll have ...
20,971,520 bytes of raw data, represented as 27,892,122 characters in raw Base64, using (on average) 29,286,728 characters when urlencoded, which will use 58,573,455 bytes of memory in its String form.
The decoding process on Jetty will take the incoming raw urlencoded bytes and allocate 2x that size in a String before decoding the urlencoded form. So that's a 58,573,456 length java.lang.String (that uses 117,146,912 bytes of heap memory for the String, and don't forget the 29MB of bytebuffer data being held too!) just to decode that Base64 binary file as a value in a x-www-form-urlencoded String form.
I would push back and force them to use multipart/form-data properly. There are tons of good libraries to generate that form-data properly.
If they are using Java, tell them to use the httpmime library from the Apache HttpComponents project (they don't have to have/use/install Apache Http Client to use the httpmime, its a standalone library).
Alternative Approach
There's nothing saying you have to use application/x-www-form-urlecnoded or multipart/form-data.
Offer a raw upload option via application/octet-stream
They use POST, and MUST include the following valid request headers ...
Connection: close
Content-Type: application/octet-stream
Content-Length: <whatever_size_the_content_is>
Connection: close to indicate when the http protocol is complete.
Content-Type: application/octet-stream means Jetty will not process that content as request parameters and will not apply charset translations to it.
Content-Length is required to ensure that the entire file is sent/received.
Then just stream the raw binary bytes to you.
This is just for the file contents, if you have other information that needs to be passed in (such as filename) consider using either the query parameters for that, or a custom request header (eg: X-Filename: secretsauce.doc)
On your servlet, you just use HttpServletRequest.getInputStream() to obtain those bytes, and you use the Content-Length variable to verify that you received the entire file.
Optionally, you can make them provide a SHA1 hash in the request headers, like X-Sha1Sum: bed0213d7b167aa9c1734a236f798659395e4e19 which you then use on your side to verify that the entire file was sent/received properly.

msf4j chunked encoding and multipart/form-data

I've been running some of the examples for MSF4J. Im looking into creating a service that allows uploading of files, but they are send using multipart/form-data. Our front sends the data with chunked encoding.
So, the FileServer example shows how to handle chunked streams with the HttpStreamer and the Formparam examples show how to handle multipart/form-data. But when I send a chunked request to the /simpleFormStreaming, it doesn't work (get a HTTP 500 response). When sending an non-chunked request (Content-Lenght is set). It does work ok.
So how can I handle a multipart/form-data request that is send using chunked encoding?
Thanks,
Danny
At the moment MSF4J doesn't support chunked data with FormParam. But you can use the HttpStreamer as in FileServer sample.
HttpStreamer.chunk method will get execute for each and every chunk. You need to implement the multipart/form-data handling logic in your HttpStreamHandler implementation. I think you can use commons-file-upload directly and do the processing.

AWS API Gateway to S3 - PUT Content-Encoding .Z Files

I have run into an issue with using API Gateway as a proxy to S3 (for custom authentication), in that it does not handle binary data well (which is a known issue).
I'm usually uploading either .gz or .Z (Unix compress utility) files. As far as I understand it, the data is not maintained due to encoding issues. I can't seem to figure out a way to decode the data back to binary.
Original leading bytes: \x1f\x8b\x08\x08\xb99\xbeW\x00\x03
After passing through API GW: ��9�W�
... Followed by filename and the rest of the data.
One way of 'getting around this' is to specify Content-Encoding in the header of the PUT request to API GW as 'gzip'. This seems to force API GW to decompress the file before forwarding it to S3.
The same does not work for .Z files compressed with the Unix compress utility. Where you should specify the Content-Encoding as 'compress'.
Does anyone have any insight about what is happening to the data, to help shed some light on my issue? Also, does anyone know any possible work-around's to maintain the encoding of my data while passing through API GW (or to decode it once it's in S3)?
Obviously I could just access the S3 API directly (or have API GW return a pre-signed URL for accessing the S3 API), but there are a few reasons why I don't want to do that.
I should mention that I don't understand very much at all about encoding - sorry if there are some obvious answers to some of my questions.
It's not exactly an "encoding issue" -- it's the fact that API Gateway just doesn't support binary data ("yet")... so it's going to potentially corrupt binary data, depending on the specifics of the data in question.
Uploading as Content-Encoding: gzip probably triggers decoding in a front-end component that is capable of dealing with binary data (gzip, after all, is a standard encoding and is binary) before passing the request body to the core infrastructure... but you will almost certainly find that this is a workaround that does not consistently deliver correct results, depending on the specific payload. The fact that it works at all seems more like a bug than a feature.
For now, the only consistently viable option is base64-encoding your payload, which increases its size on-the-wire by 33% (base64 encoding produces 4 bytes of output for every 3 bytes of input) so it's not much of a solution. Base64 + gzip with the appropriate Content-Encoding: gzip should also work, which seems quite a silly suggestion (converting a compressed file into base64 then gzipping the result to try to reduce its size on the wire) but should be consistent with what API Gateway can currently deliver.

What is http multipart request?

I have been writing iPhone applications for some time now, sending data to server, receiving data (via HTTP protocol), without thinking too much about it. Mostly I am theoretically familiar with process, but the part I am not so familiar is HTTP multipart request. I know its basic structure, but the core of it eludes me.
It seems that whenever I am sending something different than plain text (like photos, music), I have to use a multipart request. Can someone briefly explain to me why it is used and what are its advantages?
If I use it, why is it better way to send photos that way?
An HTTP multipart request is an HTTP request that HTTP clients construct to send files and data over to an HTTP Server. It is commonly used by browsers and HTTP clients to upload files to the server.
What it looks like
See Multipart Content-Type
See multipart/form-data
As the official specification says, "one or more different sets of data are combined in a single body". So when photos and music are handled as multipart messages as mentioned in the question, probably there is some plain text metadata associated as well, thus making the request containing different types of data (binary, text), which implies the usage of multipart.
I have found an excellent and relatively short explanation here.
A multipart request is a REST request containing several packed REST requests inside its entity.

Fiddler: How to use oSession.utilFindInResponse in gzip encoded response

I'm trying to use Fiddler to break when the response contains a specific word and edit the response live.
However, it seems that the oSession.utilFindInResponse function does not match successfully because the response is using GZIP encoding.
Is it possible to get around this ?
I'm new to fiddlers rules but there might be a way to change http compression on the fly ?
Thanks
Calling oSession.utilDecodeResponse() on the session object will remove HTTP chunking and compression from the response.