Nginx + Tornado ( + curl): Inflate gzipped POST request - file-upload

Question

I have setup a server (well... two servers, but I don't think that is too relevant for this question) running Tornado (version 2.4.1) and being proxied by Nginx (version 1.4.4).
I need to periodically upload json (basically text) files to one of them through a POST request. These files would greatly benefit from gzip compression (I get compression ratios of 90% when I compress the files manually) but I don't know how to inflate them in a nice way.
Ideally, Nginx would inflate it and pass it clean an neat to Tornado... but that's not what's happening now, as you'll have probably guessed, otherwise I wouldn't be asking this question :-)
These are the relevant parts of my nginx.conf file (or the parts that I think are relevant, because I'm pretty new to Nginx and Tornado):
user borrajax;
worker_processes 1;
pid /tmp/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
access_log /tmp/access.log main;
error_log /tmp/error.log;
# Basic Settings
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
gzip on;
gzip_disable "msie6";
gzip_types application/json text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
gzip_http_version 1.1;
gzip_proxied expired no-cache no-store private auth;
upstream web {
server 127.0.0.1:8000;
}
upstream input {
server 127.0.0.1:8200;
}
server {
listen 80 default_server;
server_name localhost;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://web;
}
}
server {
listen 81 default_server;
server_name input.localhost;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://input;
}
}
}
As I mentioned before, there are two Tornado servers. The main one is running on localhost:8000 for the web pages and that kind of stuff. The one running on localhost:8200 is the one intended to receive those json files) This setup is working fine, except for the Gzip part.
I'd like for Nginx to inflate the gzipped requests that come to localhost:81, and forward them to the Tornado I have running on localhost:8200 (inflated)
With the configuration like this, the data reaches Tornado, but the body is still compressed, and Tornado throws an exception:
[E 140108 15:33:42 input:1085] Uncaught exception POST
/input/log?ts=1389213222 (127.0.0.1)
HTTPRequest(
protocol='http', host='192.168.0.140:81',
method='POST', uri='/input/log?&ts=1389213222',
version='HTTP/1.0', remote_ip='127.0.0.1', body='\x1f\x8b\x08\x00\x00',
headers={'Content-Length': '1325', 'Accept-Encoding': 'deflate, gzip',
'Content-Encoding': 'gzip', 'Host': '192.168.0.140:81', 'Accept': '*/*',
'User-Agent': 'curl/7.23.1 libcurl/7.23.1 OpenSSL/1.0.1c zlib/1.2.7',
'Connection': 'close', 'X-Real-Ip': '192.168.0.94',
'Content-Type': 'application/json'}
)
I understand I can always get the request's body within the post() Tornado handler and inflate it manually, but that just sounds... dirty.
Finally, this is the curl call I use to upload the gzipped file:
curl --max-time 60 --silent --location --insecure \
--write-out "%{http_code}" --request POST \
--compressed \
--header "Content-Encoding:gzip" \
--header "Content-Type:application/json" \
--data-binary "$log_file_path.gz" \
"/input/log?ts=1389216192" \
--output /dev/null \
--trace-ascii "/tmp/curl_trace.log" \
--connect-timeout 30
The file in $log_file_path.gz is generated using gzip $log_file_path (I mean... is a regular Gzip compressed file)
Is this something doable? It sounds like something that should be pretty straight forward, but nopes...
If this is is something not doable through Nginx, an automated method in Tornado would work too (something more reliable and elegant that having me uncompressing files in the middle of a POST request's handler) Like... something like Django middlewares or something like that?
Thank you in advance!!

Answer 1

You're already calling json.loads() somewhere (Tornado doesn't decode json for you so the exception you're seeing (but did not quote) must be coming from your own code); why not just replace that with a method that examines the Content-Encoding and Content-Type headers and decodes appropriately?

Answer 2

I gave up trying to have Nginx or Tornado automatically expanding the body of the POST request, so I went with what Ben Darnell mentioned in his answer. I compress the file using gzip and POST it as a part of a Form (pretty much as if I were uploading a file).
I'm gonna post the bits of code that take care of it, just in case this helps someone else:
In the client (a bash script using curl):
The path (absolute) to the file to send is in the variable f. The variable TMP_DIR points to /tmp/, and SCRIPT_NAME contains the name of the bash script trying to perform the upload (namely uploader.sh)
zip_f_path="$TMP_DIR/$(basename ${f}).gz"
[[ -f "${zip_f_path}" ]] && rm -f "${zip_f_path}" &>/dev/null
gzip -c "$f" 1> "${zip_f_path}"
if [ $? -eq 0 ] && [[ -s "${zip_f_path}" ]]
then
response=$(curl --max-time 60 --silent --location --insecure \
--write-out "%{http_code}" --request POST \
"${url}" \
--output /dev/null \
--trace-ascii "${TMP_DIR}/${SCRIPT_NAME}_trace.log" \
--connect-timeout 30 \
--form "data=#${zip_f_path};type=application/x-gzip")
else
echo "Attempt to compress $f into $zip_f_path failed"
fi
In the server (in the Tornado handler):
try:
content_type = self.request.files['data'][0]['content_type']
if content_type == 'application/x-gzip':
gzip_decompressor = GzipDecompressor()
file_body = gzip_decompressor.decompress(
self.request.files['data'][0]['body'])
file_body += gzip_decompressor.flush()
else:
file_body = self.request.files['data'][0]['body']
except:
self.send_error(400)
logging.error('Failed to interpret data: %s',
self.request.files['data'])
return