Environment: Debian, lighttpd, fam. It is unclear whether virtual private server's kernel Dnotify is enabled or not.
After my CGI creates a file in file system, it responds a short message to client, java script running in browser, notifying the client that the file is ready for download. There is one issue here - most of the time the client gets http 404 error from lighttpd especially when all these software run in localhost. I think this problem is caused by the latency of lighttpd being notified by fam of the file creation. I imagine the all events happen in the following sequence:
CGI creates a file.
CGI responds to client.
client fetch the file from lighttpd.
lighttpd does not find the file in its cache.
I can imagine that such problem will be worse once the file is created in remote host and is notified to lighttpd via NFS.
My question is: how to code CGI to tell lighttpd about file creation before respond to client?
That is, how do I "insert" a notification step in between above steps 1. and 2.? If I can not do this with lighttpd but I can do it with nginx, I can consider switching to the latter, too. Similarly, I will be fine to switch from fam to other file alternation monitors.
Best Regards,
lighttpd is not caching the non-existence of the file, so the problem is more likely in your CGI script. The CGI script should create the file and should close it so that the kernel knows that the contents should be flushed to disk, as they might still be buffered in stdio in the CGI process. Only after that should the script send a message to the client.
As you noted, extra steps are required if the CGI is run on a different server from the one that receives the second request for the file, and the file is stored on NFS or other remote filesystem. The easiest solution is to have the client wait a second or two and retry to download the file a few times, as typical NFS caching settings are about 5 seconds.
Related
I am running a Perl CGI tool that executes a system command (Unix) which may run for a few seconds up to an hour.
After the script is finished, the tool should display the results log on the screen (in a browser).
The problem is that after about 5 minutes I get a timeout message "Gateway Time-out" - the system command continue to run but I'm unable to display to the user the results of the run.
In the Apache config file (httpd.conf): Timeout 300.
Is there a simple way ordering the Apache to increase the timeout only for a specific run?
I don't really want to change the Apache timeout permanently (or should I?) and not dramatically update the code (a lot of regression tests).
Thanks in advance.
Mike
Make the script generate some output every once in a while. The timeout is not for running the program to completion, but is a timeout while Apache is waiting for data. So if you manage to get your program to output something regularly while running, you will be fine.
Note that HTTP clients, i.e. browsers, also have their own timeout. If your browser does not get any new data from the web server five minutes (typically), the browser will declare a timeout and give up even if the server is still processing. If your long running processing gives some output every now and then, it will help against browser timeouts too!
For completeness:
Though the accepted answer is the best (it's variously known as KeepAlive packets in TCP/IP, or Tickle packets way back in appletalk days) you did ask if you can do dynamic Apache config.
An apache module could do this. Oh, but that's a pain to write in C.
Remember that mod_perl (and to some extent mod_python, though it's deprecated) do not only handlers but wrap the internal config in perl as well. You could write something complicated to increase the timeout in certain situations. But, this would be a bear to write and test, and you're better off doing what Krisku says.
There doesn't seem to be any way to specify a timeout on the <!--#include virtual=... --> directive, but if you use mod_cgid instead of mod_cgi then starting with Apache 2.4.10 there's a configurable timeout parameter available which you can specify in httpd.conf or .htaccess:
CGIDScriptTimeout nnns
...where nnn is the number of seconds that Apache will allow a cogitating CGI script to continue to run.
Caveat: If you use PHP with Apache, then your Apache is presumably configured in /etc/httpd/conf.modules.d/00-mpm.conf to use "prefork" MPM (because PHP requires it unless built with thread-safe flags), and the default Apache installation used mod_cgi with the prefork MPM, so you'll probably need to edit /etc/httpd/conf.modules.d/01-cgi.conf to tell Apache to use mod_cgid instead of mod_cgi.
Although the comment in 01-cgi.conf says, "mod_cgid should be used with a threaded MPM; mod_cgi with the prefork MPM," that doesn't seem to be correct, because mod_cgid seems to work fine with prefork MPM and PHP, for me, with Apache 2.4.46.
Although that doesn't give you complete control over server timeouts, you could specify a different CGIDScriptTimeout setting for a particular directory (e.g., put your slow .cgi files in the ./slowstuff/ folder).
(Of course, as krisku mentioned in the accepted answer, changing CGIDScriptTimeout won't solve the problem of the user's web browser timing out.)
i'm trying to write a webserver. i didn't want to write a module for php, so i figured i'd pass information to php-fpm like nginx and apache does. i did some research, and setup to prototypes, and just can't get it to work.
i've set up a php service listening on port 9999 that will print_r($GLOBALS) upon each connection. i've set up nginx to pass php requests to 127.0.0.1:9999. the requests ARE being passed, but only argc (1) and argv (the path to the php service), and $_SERVER vars are populated. the $_SERVER vars has a lot of information about the current environment that the php process is acting in, but i don't see ANY information about the connected user or their request -no REMOTE_ADDR, no QUERY_STRING, no nothing...
i'm having trouble finding documentation on HOW to pass this information from the cli or from a prototype server to a fastcgi php process. i've found a list of what some of the older CGI vars are, but no information on HOW to pass them, or if any of them are outdated with fastcgi.
so, again, i'm asking HOW you pass info from your server prototype or cli to a php-fpm or fastcgi process -or, WHERE can i find proper and clear and definitive documentation on this subject? (and no, the RFC is not the answer). i've beed reading over fastcgi.com and wikipedia as well as numerous other search results...
=== update ===
i've managed to get a working fastcgi "service" up and running via a prototype in php. it listens on 9999, parses a binary fcgi request from the cli and even from nginx, it formats a binary fcgi response, sends it back over the network, and the cli displays it fine, and nginx even returns the decoded fcgi response back to the browser just like nature intended.
now, when i try to do things the other way around --write my prototype server that forms a binary fcgi packet and sends it to PHP-FPM, i get NOTHING -no error output on the cli or from the error logs (i can't ever get php-fpm to write to the error logs any way [-_-]). so, WHY wouldn't php-fpm be giving me SOME kind of response, either in error text, or in binary network packet, or ANYTHING???
so, i can SEND data from cli to to fastcgi, but i can't get any thing back, unless it's MY OWN fastcgi process (and no, i didn't take over php-fpm's port -i'm on 9999 and it's on 9000).
=============
TIA \m/(>_<)\m/
got it.
the server passes information to the fastcgi process in the form of a network data packet. much like a dns packet, you have to use the respective binary character manipulation functions for your language to formulate the packate, including it's header and payload information, then send it across the pipe / network to the fastcgi server who will then parse the binary packet into a response. fun stuff -could have been more well documented, ahem :-\
oh, and if you prototype a listener in php, you can't access this packet via any php vars, you actually have to read the connection you're listening on (because it's a binary network packet and not plain text 'post' data sent).
i've managed to get a working fastcgi "service" up and running via a prototype in php. it listens on 9999, parses a binary fcgi request from the cli and even from nginx, it formats a binary fcgi response, sends it back over the network, and the cli displays it fine, and nginx even returns the decoded fcgi response back to the browser just like nature intended.
How does Apache (most popular version nowadays, i guess) handle a connection to a script when this script is already being executed for another connection?
My guess has always been - upon receipt of a request to a script, script's contents are copied-to-memory/compiled/executed, and IF during this process there's another request to this script - same things happen (assuming Apache does not lock the script file, and simply gives another share of memory/cpu for another compilation/memory-storage/execution)
Or is there a queuing/waiting mechanism involved?
Assuming this additional connection is afforded enough memory, cpu, and does not pass maximum connections setting.
The quickly (and easy) answer is every request is processes by a new process.
Apache listens in some port and for each request create a new process that handles that request. That means no shared memory.
Also take a look to processes with "ps" command, you will see one "http" process for each request.
Take a look here for more complex working: http://httpd.apache.org/docs/2.0/mod/worker.html
and look at google too :) http://docstore.mik.ua/orelly/weblinux2/apache/ch01_02.htm
I'm building a web tool which allows users to upload PDFs to a server using their web browsers. The server is based on Python (Paste + Pyramid).
The problem I have right now is the following: If a user uploads a rather large file (let's say 100 MB) and they cancel the upload before it is completed, my handler code on the server is still called (instead of the request being aborted).
The problem is that the request.POST['myfile'].file is incomplete when that happens. This effectively means that the PDF file is corrupted if I simply write it to some place on the server.
When I watch the server's log, it shows a "broken pipe" exception within the Paste server; however I have no idea how to catch that exception and have it prevent my view/handler code from executing and storing the incomplete file.
Seems like the paster HTTP server does not correctly validate the uploaded form data and simply passes the request down the WSGI pipeline even if the connection (HTTP POST) was closed by the user.
I worked around this issue by simply setting up NGINX to act as a reverse proxy. This also adds some security benefits as it might be better tested than paster.
Update:
My main problem was that I was using runserver (the built in web server of manage.py). After some trial and error we ended up using WSGI.
More specifically, uWSGI and Nginx as web server. Static content is served directly by Nginx while dynamic pages are piped through uWSGI and are handled by the Python web app.
Unless you are doing something fancy (like tracking the upload progress, etc), your pylons controller should not be invoked until the entire file has been uploaded.
I am building a cgi application, and now I would like it to be like an application that stands and parses each connection, with this, I can have all session variables saved in memory instead of saving them to file(or anyother place) and loading them again on a new connection
I am using lamp within a linux vmware but I can't seem to find how to install the module for it to work and what to change in the httpd.conf. I tried to compile the module, but I couldn't because my apache isn't a regular instalation, its a lamp already built one, and it seems that the mod needs the apache directory to be compiled. I saw some coding examples out there, so I guess is not that hard once its runing ok with Apache
Can you help me with this please?
Thanks,
Joe
Using FastCGI just means that you spawn a number of processes which will handle requests they get from the actual webserver instead of the webserver spawning a new process whenever a request arrives.
Use something like memcached if you want to keep stuff like sessions in memory.