For the purpose of unit testing, I want to configure Apache to gzip a static file and then chunk encode it and send it as a response. I tried following so far:
# Created a file httpd-deflate.conf
# Set to gzip all output
SetOutputFilter DEFLATE
#set compression level
DeflateCompressionLevel 9
# Deflate in chunks
DeflateBufferSize 100
Included this file in httpd.conf
Include httpd-deflate.conf
Send request as follows:
GET / HTTP/1.1
Host: test-server
Accept-Encoding: gzip
Connection: Keep-Alive
Got following response:
HTTP/1.1 200 OK
Date: Wed, 05 Nov 2014 19:24:23 GMT
Server: Apache/2.2.15 (Scientific Linux)
Last-Modified: Wed, 05 Nov 2014 18:17:21 GMT
ETag: "5e28c-5df-5072097390e40"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1186
Connection: close
Content-Type: text/html; charset=UTF-8
<binary-data>
Is there a way by which Apache can start sending the gziped data in chunked format. I thought DeflateBufferSize can help me do that. However, no luck so far.
Any suggestions?
Thanks.
Related
I work at a company that makes a web publication software. Yesterday I've stumbled upon the strange bug with opening links from a flash. The link was to a page on linkedin.com site, but maybe it's not the only case.
Here is a test publication:
http://cdiem.cld.bz/Link-test
(Click the "Product guide" text, there is the link to a page on linkedin.com)
For some reason it opens as a plain text in Chrome and Opera (and maybe other Chromium-based browsers), but works fine in Firefox and IE.
It also works fine from HTML version of the publication (disable Flash plugin to see it). And it also works fine if you just reload the page.
My guess is that it has something to do with the X-Requested-With header field, cause it's the only thing I found that differs between the HTTP request from Flash and HTML versions of publication:
X-Requested-With:ShockwaveFlash/16.0.0.305
Could anyone give any advice on that?
I think that you are right about X-Requested-With.
Take these two tests that I did using hurl.it where you can test HTTP requests :
First test : just request our page.
Request headers :
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: runscope/0.1
Response headers :
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Length: 6156
Content-Type: text/html;charset=utf-8
Date: Thu, 05 Mar 2015 21:10:50 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
...
Here we can see very clear that server has sent a text/html content.
We do the same test but we will just add the X-Requested-With header.
Second test : request our page with X-Requested-With header.
Request headers :
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: runscope/0.1
X-Requested-With: stackoverflow_test
Response headers :
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Encoding: gzip
Content-Language: en-US
Content-Length: 3602
Content-Type: text/plain;charset=UTF-8
Date: Thu, 05 Mar 2015 21:21:06 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
...
This time we can see that the server has sent a text/plain content.
So it's clear that the server is changing the Content Type to text/plain when receiving a X-Requested-With header which is sent by Flash Player PPAPI (used in Chrome and Opera) like you can see here.
Hope that can help.
My current setup is as following:
Apache -> Nginx -> Varnish
running on Ubuntu
apache2.conf: h**p://pastebin.com/A3wehAbe
.htaccess: h**p://pastebin.com/Yre4hdSy (edited to allow deflate)
nginx.conf: h**p://pastebin.com/6X59CTAr (gzip enabled)
varnish: default settings
My problem is, it seems that GZIP only works with html content, not css or js.
I tested GZIP with this tool at:
http://www.gidnetwork.com/tools/gzip-test.php
h**p://rentsites.com.au
result: compressed yes
status HTTP/1.1 200 OK
server nginx/0.7.65
content-type text/html;
charset=UTF-8
x-powered-by PHP/5.3.2-1ubuntu4.22
x-pingback h**p://rentsites.com.au/xmlrpc.php
vary Accept-Encoding
content-encoding gzip
content-length 2281
accept-ranges bytes
date Mon, 13 Jan 201400:50:26 GMT
x-varnish 785049695 785049694
age 13
via 1.1 varnish
connection close
h**p://rentsites.com.au/wp-includes/js/jquery/jquery.js
result: compressed NO
status HTTP/1.1 200 OK
server nginx/0.7.65
content-type application/x-javascript
last-modified Wed, 21 Aug 2013 15:41:10 GMT
expires Mon, 12 Jan 2015 23:17:22 GMT
cache-control max-age=31536000, public
pragma public
content-length 93085
accept-ranges bytes
date Mon, 13 Jan 2014 00:27:30 GMT
x-varnish 785049693 785049631
age 4208
via 1.1 varnish
connection close
h**p://rentsites.com.au/wp-content/themes/twentyfourteen/genericons/genericons.css
result: compressed NO
status HTTP/1.1 200 OK
server nginx/0.7.65
content-type text/css
last-modified Tue, 12 Nov 2013 18:38:10 GMT
expires Mon, 12 Jan 2015 23:16:50 GMT
cache-control max-age=31536000, public
pragma public
content-length 22680
accept-ranges bytes
date Mon, 13 Jan 2014 00:52:38 GMT
x-varnish 785049696 785049621
age 5748
via 1.1 varnish
connection close
Please help. Thanks for reading.
Turns out that we need to add this to .htaccess:
AddType text/css .css
AddType text/javascript .js
I am verifying if my application handles file content delivered through chunked-encoding mode. I am not sure what change to make to the httpd.conf file to force chunked encoding through Apache. Is it even possible to do this with Apache server, if not what would be an easier solution? I am using Apache 2.4.2 and HTTP 1.1.
By default, keep-alive is On in Apache and I do not see the data as chunked when testing with wireshark.
EDIT: Added more info:
Only way I managed to do this was by enabling the deflate module.
Then I configured my client to send "Accept-Encoding: gzip, deflate" header and apache would compress and send the file back in chunked mode.
I had to enable the file type in the module though.
AddOutputFilterByType DEFLATE image/png
See example:
curl --raw -v --header "Accept-Encoding: gzip, deflate" http://localhost/image.png | more
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /image.png HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost
> Accept: */*
> Accept-Encoding: gzip, deflate
>
< HTTP/1.1 200 OK
< Date: Mon, 13 Apr 2015 10:08:45 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Mon, 13 Apr 2015 09:48:53 GMT
< ETag: "3b5306-5139805976dae-gzip"
< Accept-Ranges: bytes
< Vary: Accept-Encoding
< Content-Encoding: gzip
< Transfer-Encoding: chunked
< Content-Type: image/png
<
This resource produces chunked results http://www.httpwatch.com/httpgallery/chunked/
which is very useful for testing clients. You can see this by running
$ curl --raw -i http://www.httpwatch.com/httpgallery/chunked/
HTTP/1.1 200 OK
Cache-Control: private,Public
Transfer-Encoding: chunked
Content-Type: text/html
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 22 Jul 2013 09:41:04 GMT
7b
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2d
<html xmlns="http://www.w3.org/1999/xhtml">
....
I tried this way to get HTTP chunked encoded data in Ubuntu, it might help.
In apache server create a file index.php in your directory where index page is there ( ex : /var/www/html/) and paste below content (should have php installed):
<?php phpinfo(); ?>
Then try to curl the page as below :
root#ubuntu-16:~# curl -v http://10.11.0.230:2222/index.php
* Trying 10.11.0.230...
* Connected to 10.11.0.230 (10.11.0.230) port 2222 (#0)
> GET /index.php HTTP/1.1
> Host: 10.11.0.230:2222
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 01 Jul 2020 07:51:24 GMT
< Server: Apache/2.4.18 (Ubuntu)
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
<
<!DOCTYPE html>
<html>
<body>
...
...
...
I have a problem of Content-Type empty for video files and wanted to update those objects with Content-Type: video/** based on extension of the file. But when i am using the put method for same key/bucket with content-type it is being overridden with '0' content-length.
Can I update an existing Amazon S3 object? link says
put will override the file s3 object which doesn't serve my purpose.
Note: I don't want to use Java SDK. I want to use normal the Java httpput due to memory constraints of the mobile.
What you could try is to use the PUT COPY request with the same source and target.
PUT /yourvideo.flv HTTP/1.1
Host: bucket.s3.amazonaws.com
Date: Wed, 28 Oct 2009 22:32:00 GMT
x-amz-copy-source: /bucket/yourvideo.flv
Authorization: AWS AKIAIOSFODNN7EXAMPLE:0RQf4/cRonhpaBX5sCYVf1bNRuU=
Apparently python boto modifies metadata this way
Yes, I can confirm, copy is working:
PUT /test/123.png HTTP/1.1\r\n
Host: yourbucket.s3.amazonaws.com\r\n
Accept-Encoding: identity\r\n
Content-Length: 0\r\n
x-amz-storage-class: STANDARD\r\n
x-amz-copy-source: yourbucket/test/123.png\r\n
Date: Fri, 11 Jan 2013 15:16:48 GMT\r\n
Content-Type: image/gif\r\n # this is the content-type set
Authorization: AWS AKXXXXXDDRXXXXXX63CA:RcktkZ9nPwsXXXXXd+KXXXXXY=\r\n
x-amz-metadata-directive: REPLACE\r\n
\r\n
S3 Reply:
HTTP/1.1 200 OK
x-amz-id-2: Vf1bRJd+e4ru0y73GB0Ra0xYB2vv6GLWYKsHAC4AzE+a8uZB56Xy8+YTDkJ0/wfN
x-amz-request-id: C0E6A2823F3FB3E9
Date: Fri, 11 Jan 2013 15:16:49 GMT
Content-Type: application/xml
Content-Length: 234
Server: AmazonS3
Verification:
HEAD /test/123.png HTTP/1.1\r\n
Host: yourbucket.s3.amazonaws.com\r\n
Accept-Encoding: identity\r\n
Date: Fri, 11 Jan 2013 15:25:23 GMT\r\n
Content-Length: 0\r\n
Authorization: AWS AXXXIXXXXMDDXXXXX7XXXCA :HoZ4Woxxw6SWlxxxx000Rxxm8ODQ=\r\n
\r\n
S3 replies with the correct content-type ( the one I set wrongly to image/gif instead of image/png )
HTTP/1.1 200 OK
x-amz-id-2: 7Ut6sdm6i+1h7Di5UrB4v9Sn3mVCNjyDnkb4rm1jGzN6wBTgDT2/yCSHdrKG12Jd
x-amz-request-id: 38A1F6EC6ECD1D2D
Date: Fri, 11 Jan 2013 15:25:24 GMT
Last-Modified: Fri, 11 Jan 2013 15:25:24 GMT
ETag: "c09fad0faf4e6bb1148670af78b6de41"
Accept-Ranges: bytes
Content-Type: image/gif
Content-Length: 122311
Server: AmazonS3
varnish don't resolve ESI in my localhost:
i have similar configuration like other users in apache, but i send/get following headers:
Server Apache/2.2.21 (Debian)
X-Powered-By PHP/5.3.8-2
Content-Encoding gzip
Edge-Control cache-maxage=31536000s
Cache-Control public, max-age=31536000
Pragma public
Expires Tue, 30 Oct 2012 11:41:15 GMT
ProcessID 3918
Etag c2e6665981c6441ab860d12a6853a002
Vary Accept-Encoding
Content-Type text/html; charset=UTF-8
Content-Length 4193
Date Mon, 31 Oct 2011 10:40:54 GMT
X-Varnish 630755530
Age 0
Via 1.1 varnish
Request Headersview source
Host i.host.com
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1 Iceweasel/7.0.1
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection keep-alive
Cookie OAX=1I6M006JYzwABwNj; __utma=164510763.315921951.1317626740.1319722611.1319795071.39; __utmz=164510763.1317626740.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=229641156.678788339.1317628577.1320057939.1320060143.19; __utmz=229641156.1317628577.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); lang=es; __utmc=229641156; __utmb=229641156.4.10.1320060143
If-None-Match c2e6665981c6441ab860d12a6853a002
Cache-Control max-age=0
other user example that works:
Server Apache/2.2.4 (Win32) PHP/5.3.5
X-Powered-By PHP/5.3.5
Set-Cookie lang=es; expires=Mon, 31-Oct-2011 11:46:01 GMT; path=/
Edge-control cache-maxage=31536000s
Cache-Control public, max-age=31536000
Pragma public
Expires Tue, 30 Oct 2012 10:46:01 GMT
ProcessID 3824
Etag f9dcd4b1ec3f40e6763e80a3ba195de3
Content-Type text/html; charset=UTF-8
Transfer-Encoding chunked
Date Mon, 31 Oct 2011 10:46:01 GMT
X-Varnish 630755925
Age 0
Via 1.1 varnish
Connection keep-alive
Cabeceras de la peticiónver fuente
Host t.host.com
User-Agent Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language es-es,es;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection keep-alive
Cookie OAX=1I6M001BiwUAB0EE; __utma=164510763.1374034467.1296142874.1319189845.1319204625.107; __utmz=164510763.1319204625.107.79.utmcsr=tamara.eitb.com|utmccn=(referral)|utmcmd=referral|utmcct=/eu/eguraldia/eskia/; __utma=261580168.2132154740.1302611206.1319197586.1320057948.90; __utmz=261580168.1302611206.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); lang=es; __utmb=261580168.1.10.1320057948; __utmc=261580168
i don't know why i receive different headers from varnish...
i get:
Content-Encoding gzip
Vary Accept-Encoding
why i get this headers and the other users no???
both have mod_deflate on...
you´re getting both Content-Encoding and Vary in order to Varnish, an a user-agent, properly handle the request. While Content-Encoding tells Vanish, an a user-agent, that the page content is encoded in a zip format, gzip in your case, the header Vary tells, specially Varnish to cache both versions of the page with or without gzip compression.
If backend is compression encoding aware, it can deliver a version with compression (gzip or deflate) or it can deliver a version withou any compression (text/html for example).
But, to enable ESI processing in varnish, use the esi; command on your vcl file. See this page for more information: