The newest version of Safari (mobile & desktop) buffers videos 4x slower than other browsers because it sends many small sized range-bytes requests opposed to a few large ones. An example request and response is below (this request continues with a small size of 64kb until enough data is loaded for the video to play, in Chrome, Firefox and other browsers the range-bytes request is much larger and so the data is delivered much faster in one stream).
Is it possible to get around this issue by forcing my web server (apache) to ignore Safari's small range-byte request of 64kb, and instead send a larger amount of data (about 5MB)? The request is made directly to the video file.
Summary
URL: http://example.org/video.mp4?rand=942824
Status: 206 Partial Content
Source: Network
Request
GET /video.mp4 HTTP/1.1
Accept: */*
Connection: keep-alive
Range: bytes=0-65535
Accept-Encoding: identity
Response
HTTP/1.1 206 Partial Content
Content-Type: video/mp4
Content-Range: bytes 0-65535/467342440
Accept-Ranges: 0-467342440
Content-Length: 65536
Connection: keep-alive
Server: nginx/1.2.1
UPDATE: I managed to change the request range header using the below code, however even though the 5mb is downloaded quickly, safari continues sending these small 64kb range requests and ignores the 5mb that was downloaded so this is not a solution.
SetEnvIf Range bytes=0-65535 HAVE_MyRequestHeader
RequestHeader unset Range env=HAVE_MyRequestHeader
RequestHeader set Range bytes=0-5000000 env=HAVE_MyRequestHeader
No. You can not change it server side. The client makes a request the server fulfills the request. Sending data the client didn’t ask for will likely cause errors.
Related
I don't know if I am not correctly understanding how the caching aspect of ETags work if there is some other issue I am dealing with, but I'll walk you through my situation.
From my understanding ETags are a unique hash that is created based on the file information and they are sent as part of Response header to uniquely identify the file. If the file is updated then the info is changed and hence the ETag for the file is also changed.
In my project, I need a fresh JS file to be fetched everytime I make changes to the file. I can't use version tags or unique hashes as part of the file name. I thought an ETag would work where
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
myFile.js
Client <------------------ SERVER
// myFile.js has not been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 304
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
Client uses cached version of file
// myFile has been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "88-58c3a1cb8474f" // new etag generated
......
myFile.js
Client <------------------ SERVER
So, if you request the file again and no changes have been made..the etag will remain the same and you'll get a 304 will indicate that the cached version should be used.
If the file has been changed the etag will be different as well and a fresh copy of the file will be sent by the server.
This is how I expected it to work.
MY PROBLEM:
When I update myFile.js it seems like I never get the new ETag has back. It just defaults to the cahced version of the file. If I clear the cache then I get the latest file and the new ETag. This to me seems to defeat the purpose. Is this how it works or am I understanding something incorrectly here?
I have hundreds of hardware devices at customers which need to send HTTP data through a telnet interface.
The destination is an Apache 2 Webserver with a PHP script waiting for the data.
This is already working however we found that the hardware involved is not able to handle hw-flow-control, this means that once data is filled (around 250 bytes) the buffer can overflow resulting in data corruption.
Fixing the HW-flow is not an option, the "modem" firmware is closed and can not be modified by the vendor anymore as it's quite old hardware.
Normally we'd use this:
POST / HTTP/1.1
Host: api.server
User-Agent: P8
Content-Type: application/x-www-form-urlencoded
Accept: */*
Content-Length: 767
VARIABLE=URLENCODED_DATA(total length 767 bytes)
This would work perfectly fine with flow-control, but in my case the 767 bytes are too much.
After around 200 bytes buffers would be overwritten and some bytes are lost.
The only current way to get it working now was using a delay when sending to the "modem" so it can empty it's buffers in time. However in the field this will not work due to instable internet connections with unpredictable timings.
I am not an expert in HTTP, I just hope it is possible to fragment a package.
I thought about using "Connection: keep-alive" or something similar.
My main question:
Is there a way to send POST data ($VARIABLE) to a Apache 2 server in smaller chunks in a way that makes the HTTP server combine them to one stream internally ?
Pseudo code:
POST / HTTP/1.1
Host: api.server
User-Agent: P8
Content-Type: application/x-www-form-urlencoded
Accept: */*
Content-Length: 400
Connection: keep-alive
VARIABLE=URLENCODED_DATA(200 bytes)
END\n\n
Server responds in TCP stream once received with "OK".
Next chunk is sent:
VARIABLE=URLENCODED_DATA(200 bytes)
Connection is closed.
As 400 bytes have been reached the process is ready, Apache forwards VARIABLE to PHP scripts POST input.
So like a HTTP flow-control within an open TCP connection.
Maybe there is a HTTP feature which is built for that purpose, or something that can be "ab"used to act in that way. keep-alive was just a guess.
If current HTTP protocols do not have such a feature the only way I can think about solving my issue is to implement flow-control on PHP side.
I hope for a better way than that though.
Update:
Meanwhile I found two interesting parameters:
Expect: 100-continue
Transfer-Encoding: chunked
What I would need is a mix of both.
A chunked transfer encoding which is expecting a 100-continue after each chunk !
This is a very interesting question, and it really has nothing to do with HTTP but with TCP.
The way to solve this is to use an intermediary proxy that takes care of spoon-feeding your devices. Ideally, this device will be able to set the window size on the TCP packet ACKs to whatever the size of the buffer the device is. That window size will close to zero when the device cannot handle any more. If you do this, you will be utilizing TCP's built-in flow control and solve the problem in a simple way.
Another thing you can do is keep this entirely in the application layer and have this intermediary proxy buffer all of the data from the response. For most normal HTTP responses this will be okay.
My iOS app makes Rest calls to my WCF web service.
The responding speed is very slow, over 3 min.
However, when I set up Fiddler as a Proxy to monitor the iOS traffic. The call was finished in 1 sec.
What does make Fiddler magically accelerate the Rest call from iOS?
p.s. Fiddler is setup on a windows PC where uses the same network with iOS App.
The rest call example (from Fiddler)
Request
GET https://xxxx.xxxx.com/Deals HTTP/1.1
Host: xxx.xxxx.com
Proxy-Connection: keep-alive
Accept-Encoding: gzip
Content-Type: application/json
Cookie: ASPXAUTH=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Connection: keep-alive
User-Agent: Natural xxxx x.x.x (iPad; iPhone OS 7.0.2; en_US)
Response
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 891437
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/7.5
LastFetchDateTimeUTC: 2014-02-14T16:52:43.5465273Z
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 14 Feb 2014 16:52:45 GMT
Response body is a large json (2MB)
p.s.
Except for Fiddler, we also tried to install wireshark and use it to capture traffic on the mac while running the app from on the simulator.
We see a lot DUP ACK, I guess that's causing tcp re-transmission
p.s.
We pinged from iOS too, there is no delay to the WCF web service.
Help!
UPDATE:
We found out a problem, looks like the respond time decreases with the length of the body. Does it mean anything?
The WireShark logs should provide you plenty of information about what happens in each case. When Fiddler "magically" makes things faster, it's typically due to:
Better connection reuse (e.g. Fiddler may reuse connections better than client)
Better buffer sizes (e.g. not using tiny buffers for read/write)
Non-broken proxy determination behavior
I wrote a bit about these in this blog post.
We solved this problem by proving that server is a shitty one. We deployed the same service on another VM and it works. Must be the a broken network card
Please note: This is not a complain about a shoddy CMS.
Just toying with Apache Bench and got terrible results with our custom CMS, more exactly i got:
Requests per second: 0.37 [#/sec] (mean)
When i run another test with a plain php file i got:
Requests per second: 4786.07 [#/sec] (mean)
Another test with a previous version of the CMS:
Requests per second: 6068.66 [#/sec] (mean)
The website(s) are working fine, no problems detected, Google's Webmaster Tools reports our sites as faster than 80% of the pages which is fine, i think.
The test was:
ab -t 30 -c 10 http://example.com/
Maybe some kind of Apache problem? Bad .htaccess config, or similar?
Update:
Just ran a simple test with sockets and the results are similar. Page loads very, very slowly. If i ran my script with another website everything is fine.
Also, there's a small hint about a chunk length problem. (Bad Apache Headers, or line endings?)
The site is gzipped, and when verbose logging turned on, i see these lines in the response:
LOG: Response code = 200
LOG: header received:
HTTP/1.1 200 OK
Date: Tue, 04 Oct 2011 13:10:49 GMT
Server: Apache
Set-Cookie: PHPSESSID=ibnfoqir9fee2koirfl5mhm633; path=/
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Cache-Control: post-check=0, pre-check=0
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
2ef6
Always at the same place, in the middle of the HTML-source, then <!DOCTYPE HTML> again.
Please, help.
Update #2:
Just checked my HTTP headers with Rex Swain's HTTP Viewer and got these results:
HTTP/1.1·200·OK(CR)(LF)
Date:·Wed,·05·Oct·2011·08:33:51·GMT(CR)(LF)
Server:·Apache(CR)(LF)
Set-Cookie:·PHPSESSID=n88g3qcvv9p6irm1fo0qfse8m2;·path=/(CR)(LF)
Expires:·Sat,·26·Jul·1997·05:00:00·GMT(CR)(LF)
Cache-Control:·no-store,·no-cache,·must-revalidate(CR)(LF)
Pragma:·no-cache(CR)(LF)
Cache-Control:·post-check=0,·pre-check=0(CR)(LF)
Vary:·Accept-Encoding(CR)(LF)
Connection:·close(CR)(LF)
Transfer-Encoding:·chunked(CR)(LF)
Content-Type:·text/html;·charset=UTF-8(CR)(LF)
(CR)(LF)
Do you notice anything unusual?
If it works well with ordinary web browsers (as you mentioned in the comments) the CMS handle the requests from Apache Benchmark differently.
A quick checklist:
AFAIK Apache Benchmark just send simple requests without any cookie handling, so try to set -C with a valid cookie (copy the values from a web browser).
Try to send exactly the same headers to the CMS as the web browser sends. Save a dump of a valid request with netcat, HttpFox or a packet sniffer and set the missing headers with -H.
Profile the CMS on the server while you're sending to it a request with Apache Benchmark. Maybe you found the bottleneck. Two poor man's error_log calls with a timestamp in the first and the last line of the index.php (or the tested script's entry point) could show how fast is the PHP script and help to calculate the overhead of the Apache HTTP Server and network.
If you run socket tests and browser tests from different machines it's could be a DNS issue (turn off HostnameLookups in Apache). Try to run them from the same machine.
Try ab -k ... or ab -H "Connection: close" ....
I guess the CMS does some costly initialization when it initializes the session and it's happens when it processes the first request. Since Apache Benchmark does not send the cookies back the CMS it creates a new session for every request and it's the cause of the slow answers.
A second guess is that the CMS handle the incoming http headers differently and the headers which was sent (or the lack of them) by Apache Benchmark trigger some costly/slow processing. It looks more appropriate since the report of the Google's Webmaster Tools.
Apache Benchmark sends HTTP 1.0 request, for example:
GET / HTTP/1.0
Host: localhost:9100
User-Agent: ApacheBench/2.3
Accept: */*
It looks to me that your server does not send any http header about Keep-Alive settings but it assumes that the client uses keep-alive when the client uses HTTP 1.0. It's not an RFC compliant behaviour:
From RFC 2616, 19.6.2 Compatibility with HTTP/1.0 Persistent Connections:
Some clients and servers might wish to be compatible with some
previous implementations of persistent connections in HTTP/1.0
clients and servers. Persistent connections in HTTP/1.0 are
explicitly negotiated as they are not the default behavior.
By default Apache Benchmark doesn't use keep-alive so it waits when the response arrives for the closing of the socket. The server closes it after 15 seconds idle. Downloading the main page with wget also takes 15 seconds. Wget also uses HTTP 1.0 in the request.
I think it's a bug in the PHP code of the CMS since ab works well on the same server with a plain php file. Anyway, you can workaround it with using keep-alive connections (-k):
ab -k -t 30 -c 10 http://example.com/
or with explicitly disabling persistent connections:
ab -H "Connection: close" -t 30 -c 10 http://example.com/
but it's still a server side issue and your original ab commands is right.
Please note that this bug probably affects only HTTP 1.0 clients (like Apache Benchmark, wget) and clients with regular browsers will not notice it.
My website serves gzipped content. I verified with Firebug and YSlow. However, JMeter does not request the gzipped content. Therefore, it gets all uncompressed content. As a result, my test cases take much longer (6-10x longer) than they do in reality.
How can I make JMeter request gzipped content from a website?
FYI, I am using the latest stable build: JMeter 2.3.4 r785646.
Add an HTTP Header Manager to the Thread Group in your Test Plan.
Add the name-value pair:
Name: Accept-Encoding
Value: gzip,deflate,sdch
This will ensure that all JMeter requests use HTTP compression.
To verify:
Add this Listener to the Thread Group: View the Results Tree
Run your test plan
View the Sampler result tab for one of the webpages.
Do you see these name-value pairs?
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
If yes, then you've successfully setup gzip requests in JMeter. Congrats.
Another way to verify is in the Summary Report stats:
You'll see that the Avg Bytes values are the uncompressed sizes. That's OK. For whatever reason, that's how JMeter works. Pay attention to the KB/sec column. That will show an improvement of 6-10x with gzip enabled.