I'm trying to provide live parsing of a file upload in CGI and show the data on screen as it's being uploaded.
However, Apache2 seems to want to wait for the full POST to complete before sending the CGI application anything at all.
How can I force Apache2 to stop buffering the POST to my CGI application?
EDIT
It appears that it's actually the output of the CGI that's being buffered. I started streaming the data to a temp file to watch it's progress. That, and I have another problem.
1) The output is being buffered. I've tried SetEnvIf (and simply SetEnv) for "!nogzip", "nogzip", and "!gzip" without success (within the CGI Directory definition).
2) Apache2 appears to not be reading the CGI's output until the CGI process exits? I notice that my CGI app (flushing or not) is hanging up permanently on a "fwrite(..., stdout)" line at around 80K.
EDIT
Okay, Firefox is messing with me. If I send a 150K file, then there's no CGI lockup around 80K. If the file is 2G, then there's a lockup. So, Firefox is not reading the output from the server while it's trying to send the file... is there any header or alternate content type to change that behavior?
EDIT
Okay, I suppose the CGI output lockup on big files isn't important actually. I don't need to echo the file! I'm debugging a problem caused by debugging aids. :)
I guess this works well enough then. Thanks!
FINAL NOTE
Just as a note... the reason I thought Apache2 was buffering input was that I always got a "Content-Length" environment variable. I guess FireFox is smart enough to precalculate the content length of a multipart form upload and Apache2 was passing that on. I thought Apache2 was buffering the input and reporting the length itself.
Are you sure it's the input being buffered that's the problem? Output buffering problems are much more common, and might not be distinguishable from input buffering, if your method of debugging is something like just printing to the response.
(Output buffering is commonly caused either by unflushed stdout in the script or by filters. The usual culprit is the DEFLATE filter, which is often used to compress all text/ responses, whether they come from a static file or a script. In general it's a good idea to compress the output of scripts, but a side-effect it that it will cause the response to be fully buffered. If you need immediate response, you'll need to turn it off for that one script or all scripts, by limiting the application of AddOutputFilterByType to particular <Directory>s, or using mod_setenvif to set the !nogzip note.)
Similarly, an input filter (including, again DEFLATE) might cause CGI input to be buffered, if you're using any. But they're less widely-used.
Edit: for now, just comment out any httpd conf you have enabling the deflate filter. You can put it back selectively once you're happy that your IO is unbuffered without it.
I notice that my CGI app (flushing or not) is hanging up permanently on a "fwrite(..., stdout)" line at around 80K.
Yeah... if you haven't read all your input, you can deadlock when trying to write output, if you write too much. You can block on an output call, waiting for the network buffers to unclog so you can send the new data you've got, but they never will because the browser is trying to send all its data before it will start to read the output.
What are you working on here? In general it doesn't make sense to write progress-info output in response to a direct form POST, because browsers typically won't display it. If you want to provide upload-progress feedback on a plain HTML form submission, this is usually done with hacks like having an AJAX connection check back to see how the upload is going (meaning progress information has to be shared, eg. in a database), or using a Flash upload component.
From an (old version) of the Apache HTTP Server manuals:
Every time your script does a "flush"
to output data, that data gets relayed
on to the client. Some scripting
languages, for example Perl, have
their own buffering for output - this
can be disabled by setting the $|
special variable to 1. Of course this
does increase the overall number of
packets being transmitted, which can
result in a sense of slowness for the
end user.
Have you tried flushing STDOUT or checking if the language you are using has buffering you can disable?
here's a useful guide for controlling buffering when using perl on the server side:
http://perl.plover.com/FAQs/Buffering.html
many of the ideas and concepts apply to other languages too, such as using buffered and unbuffered output, raw system calls to read and write data vs I/O libraries which do their own buffering.
Related
I'm not sure if this makes sense so please comment if I need to provide more info:
My webserver is used to upload files (receives files as Multipart/form-data and uploads them to another service). Using WebFlux, the controller defines the argument as a #RequestPart(name = "payload") final Part payload which wraps the header and Flux.
Reactor / Netty uses DirectByteBuffers to accomodate the payload. If the request handler cannot get enough direct memory to handle the request, it's gonna fail on an OOM and return 500. So this is normal / expected.
However, what's supposed to happen after?
I'm running load tests by sending multiple requests at the same time (either lots of requests with small files or less requests with bigger files). Once I get the first 500 due to an OOM, the system becomes unstable. Some requests will go through, and other fails with OOM (even requests with very small payload can fail).
This behaviour leds me to believe the allocated Pooled buffers are not shared between IO Channels? However this seems weird, it makes the system very easy to DDOS?
From the tests I did, I get the same behaviour using unpooled databuffers, although for a different reason. I do see the memory being unallocated when doing jcmd <PID> VM.native_memory but they aren't released to the OS according to metrics & htop. For instance, the reserved memory shown by jcmd goes back down but htop still reports the previous-high amount and it eventually OOM.
So Question :
Is that totally expected or am I missing a config value somewhere?
Setup :
Spring-boot 2.5.5 on openjdk11:jdk-11.0.10_9
Netty config :
-Dio.netty.allocator.type=pooled -Dio.netty.leakDetectionLevel=paranoid -Djdk.nio.maxCachedBufferSize=262144 -XX:MaxDirectMemorySize=1g -Dio.netty.maxDirectMemory=0
I have rewritten web application from using mod_python to using mod_wsgi. Problem is that now it takes at least 15 seconds before any request is served (firebug hints that almost all of this time is spent by receiving data). Before the rewrite it took less than 1 second. I’m using werkzeug for app development and apache as a server. Server load seems to be minimal and same goes for memory usage. I’m using apache2-mpm-prefork.
I’m using the default setting for mod_wsgi - I think it’s called the ‘embedded mode’.
I have tested if switching to apache2-mpm-worker would help but it didn’t.
Judging from app log it seems that app is done with request quite fast - less than 1 second.
I have changed the apache logging to debug, but I can’t see anything suspicious.
I have moved the app to run on a different machine but it was all the same.
Thank in advance for any help.
Sounds a bit like your response content length doesn't match how much data you are actually sending back, with content length returned being longer. Thus browser waits for more data until possibly times out.
Use something like:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response
to verify what data is being sent back and that things like content length match.
Otherwise it is impossible to guess what issue is if you aren't showing small self contained example of code illustrating problem.
The Seaside book says: "saving [an image] while processing http requests is a risk you want to avoid“.
Why is this? Does it just temporarily slow down serving http requests or will requests get lost or will errors occur?
Before an image is saved registered shutdown actions are executed. This means source files are closed and web servers are shut down. After the image is saved it executes the startup actions, which typically bring up the web-server again. Depending on the server implementation open connections might be closed.
This means that you cannot accept new connections while you save an image and open connections might be temporarily suspended or closed. For both issues there are (at least) two easy workarounds:
Fork the image using OSProcess before you save it (DabbleDB, CmsBox).
Use multiple images and a load balancer so that you can remove images one at a time from the active servers before saving them.
It seems that it's just a question of slowing things down. There is this quite thorough thread on the Seaside list, the most relevant post of which is this case study of an eCommerce site:
Consequently, currently this is what happens:
image is saved from time to time (usually daily), and copied to a separate "backup" machine.
if anything bad happens, the last image is grabbed, and the orders and/or gift certificates that were issued since the last image save
are simply re-entered.
And, #2 has been very rarely done-- maybe a two or three times a
year, and then it turns out it is usually because I did something
stupid.
Also, one of the great things about Smalltalk is that it's so easy to run quick experiments. You can download Seaside and put a halt in a callback of one of the examples. For example:
WACounter>>renderContentOn: html
...
html anchor
callback: [
self halt.
self increase ];
with: '++'.
...
Open a browser on the Seaside server (port 8080 by default)
Click "Counter" to go to the example app
Click the "++" link
Switch back to Seaside. You'll see the pre-debug window for the halt
Save the image
Click "Proceed"
You'll see that the counter is correctly incremented, with no apparent ill effect from the save.
I need a solution to play a segment of an mp3. I have a few 1,000 audio files which are currently stored on Amazon S3, and would like to allow users to play them, however I would like to limit the play length to 30 seconds or so in the middle of the recording.
I'm not sure if I need to create an entirely new file (snippet) such as I would for a thumbnail if it were an image, or if it's possible using some player/steam to safely limit it that way so they cannot access the whole song.
I'm coming from a Rails environment and using Paperclip to handle the files and JPlayer to play them if it matters.
Any pointers or best practices?
This is possible by using the HTTP "Content-range" header. This header says 'please just give me the bytes from here to here and ignore the rest'. If the web server is set up to handle them (Apache is for instance), then you get a 206 response with a body of just those bytes.
You must create a small proxy application that effectively acts as a gateway between the listener and Amazon.
To see if your host will respond try this from the command line:
curl -v -I http://www.mfiles.co.uk/mp3-downloads/01-Tartaros%20of%20light.mp3
Where the url is one of yours. If you are lucky you will see:
Accept-Ranges: bytes
Content-Length: 5284483
This means that the server does accept the Content-range header and the full length of the file is 5284483 bytes long.
Let's request the first third of the file:
curl -H'Range: bytes=0-1761494' http://www.mfiles.co.uk/mp3-downloads/01-Tartaros%20of%20light.mp3 > /tmp/test1.mp3
You should now be able to play /tmp/test1.mp3 and hear the first third of the track.
The next step is to create a proxy application. A good approach would be to use https://github.com/aniero/rack-streaming-proxy but you would probably need to fork the project to send the 'Range: bytes=0-1761494' header. Alternatively have a look at Sinatra.
A bonus here is that because you are proxying the remote server, you could obfuscate the actual URL of the file by having a simple database table with an ID for each file. I would suggest writing a small script that also stores the byte length of each file, so that you don't have to calculate the range for each request.
Thus a GET to "/preview/12345" would proxy "http://amazon.com/my_long_url" and give you just the first third of the file.
On top of that, you could put Varnish in front of your own server, which would cache these partial MP3 files and mean that you are not having to constantly go back to Amazon to get the files.
Unfortunately, you'll need to make new snippets - there isn't really a way to tell a user's browser "download this entire mp3 file, but only play and allow access to the middle 30 seconds".
i think it is simplier to solve the problem in the client side.
Are you using flash to play the audio files?
If yes, I have done something similar (but with videos) using JWPlayer (it also supports audio files).
You can develop a custom plugin to control the snippet you want to play and then stop the audio file and show a message or something like that.
This solution combined with signed urls or/and rtmp streaming with CloudFront can be very safe.
Due to the mp3 format limitation, you cannot seek to the arbitrary frame in the middle of the song and start transmission from that point.
So, there are basically three options:
1. Create new files offline. Very easy, but space consuming.
2. Transcode files on the fly. CPU consuming, degrades quality.
3. Limit playback with first X seconds: just peek into the song' header, get its bitrate and calculate size of the byte chunk to serve
And don't ever transmit more than you need: people will manage to intercept the stream and save it to disk (business side); save your users' traffic (good karma).
While I can get microsecond resolution on the time taken to process a request (%D) to help reconstruct the sequence of requests I would like to look at this in relation to the times of multiple requests generated by a particular page. However as far as I can tell, the %t specifier only provides accuracy to the nearest second. Which makes it impossible to reconstruct the original sequence of events.
Is there another way to get this information in my access_log files?
TIA
This is now possible with Apache 2.4. Use for example the following log-format instead of %t:
[%{%d/%b/%Y:%H:%M:%S}t.%{msec_frac}t %{%z}t]
This will give times like [10/Apr/2012:10:47:22.027 +0000]
Unfortunately, no. This got covered a while back (How to timestamp request logs with millisecond accuracy in Apache 2.0), and it's still true for the most recent stable (2.2.x) Apache branch.
I know of at least one workaround, though, if you're interested: You can pipe Apache's logs to an external process (see the docs page at http://httpd.apache.org/docs/current/mod/mod_log_config.html, under the "CustomLog" directive) which would add timestamps and actually write to the log file.
Note that this method does NOT capture the true request RX time. Apache doesn't output an access log entry until AFTER it's completed sending its response. Plus, there's an additional variable delay while Apache writes into the pipe and your timestamper reads from it (possibly including some buffering, there). If you turned on Apache's "BufferLogs" directive, there's going to be more variable buffering delay. When the system is under load, and perhaps in other edge cases, the average delay could easily grow to a second or more.
If the delays aren't too bad (i.e., "BufferedLogs off", low system load), you can probably get a pretty tight estimate by subtracting the "%D" value from your external timestamp.
Some people (including me) pipe Apache's access logs to the local Syslog daemon (via the 'logger' command or whatever). The syslog daemon takes care of timestamping, among other things.