Using PhantomJS and bash, I'm working on a little piece of anti-malware that reads a web page, grabs all the domains that are delivering assets to the browser, then prints each server's country of origin. It works fine except for one site that has a... uh... 'suboptimal' piece of javascript that calls to an external server every 5 seconds. PhantomJS just loads the resource over and over and over, page.open() never finishes, and page.onLoadFinished() is never called.
Is there a way around this? Can I set a time limit on page.load()? I guess, as a workaround, can I set a time limit on the Linux process?
Thanks in advance, and if anyone is interested in a copy of this script let me know and I'll post it somewhere public.
I solved this problem using the solutions given here to set a execution time limit on the phantomjs command and kill it if needed.
Command line command to auto-kill a command after a certain amount of time
Related
I'm running PHP-FPM with Redis on AWS.
Currently I'm having a really strange issue that I can't seem to figure out.
When I INCR or HINCRBY and increment by 1 it always increments by around 20 to 30 instead.
I have tried the following:
Commented out all other redis code (no change).
Setup a single PHP page using the same code outside of the site (this works fine - increments by 1).
In the main site (that is having the issue) I put the code in the header, after the last HTML tag and other places and it behaves the same.
I have an AJAX page within the site which is invoked separately if requested and this works fine. Therefore the issue only occurs during the main site load.
I've tested redis-cli using the commands and this works fine.
I can't seem to find any loads on the AWS Redis System to read so I'm not sure exactly what is occurring here but it appears the command is running multiple times.
I also read the value back after its written and the value reports correctly. So the increment seems to work - however when I re-check redis using a GUI tool I can see it’s increased by a much larger number.
I'm really at a loss for what to try next and was hoping someone might have some advice.
Thank you.
I'm running CGI-LUA scripts with lighttpd on embedded device. The web client attempts to run via POST three scripts every 3 seconds.
Most of the time it works, but the issue is that from time to time I get 500 internal server error, like the server fails to run the script, though nothing changed and in the 'top' I see that the CPU is idle most of the time.
I'm new to web, any ideas?
If I were trying to solve this problem I would start with:
1) Look in /var/log/lighttpd/error.log to see what lighttpd is reporting when the failure occurs.
2) Write a very simple CGI-LUA script that does something traceable, like touch a file with the current unixtime as its name, and hit it every 3 seconds instead of your script. This will help you figure out if the problem is in CGI-LUA or in your script.
3) Run your script outside CGI-LUA repeatedly in a loop to see if it ever fails.
I have a CGI script run by Hiawatha web server that needs to
return some data to the client,
do some system work (may take 20-30 seconds)
and then return yet more data.
So far I haven't been able to achieve this result: the script doesn't write data as the commands get executed, rather, it writes everything in a single shot when its execution ends. Is it even possible to achieve with Hiawatha what I described above? Thank you.
I figured it out: you need to set the WaitForCGI = yes option in the config file.
I have a CGI script that takes a really long time to execute. Long story short, it needs to process a lot of data, run a bunch of slow commands, and make some slow web queries, during which time it doesn't output anything, and when it's done, it finally prints its results out in JSON format. It takes several minutes to run, which is longer than the Timeout directive set in my Apache web server's httpd.conf.
I am not at liberty to change that Timeout value globally for everyone on the entire server. I thought of maybe overriding that in a per-directory basis using a .htaccess file, but it looks like the Timeout directive is not in .htaccess context, so that cannot be done. From what I understand, my script must continually output data, and if it doesn't output data for the Timeout number of seconds, Apache gives up.
I am getting the following error in Apache: (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed
What can I do?
Well, to offer the stupidly simple solution, why not just make the script occasionally produce some output while it's working? You could just print "Processing..." every few steps, or if you want to be more creative, have it print some status updates to indicate what it's doing. Or if you're worried about getting bored, print out a funny poem a line at a time. (Kind of reminds me of http://pages.cs.wisc.edu/~veeve/404.html)
If you don't want to do that, the next thing that comes to my mind is to use asynchronous processing. Basically, you'll have to spawn a separate process from the CGI script, and do the lengthy processing in that separate process. The main CGI script itself just outputs a simple HTML page that says the process is working and then exits. That HTML page would also have to contain some logic for periodically checking to see whether the background process on the server has finished. It could be a <meta http-equiv="refresh" ...> HTML element, or you could use AJAX.
I came up with a solution.
I would start outputting a dummy HTTP header, like Dummy: ..., and I can put whatever data I want as the value of that header, and it wouldn't affect the rest of the output. So I would output a character to that dummy value every minute or so, preventing it from timing out. And when I am ready, I can print a line return and continue printing the rest of my (real) HTTP headers and the content of the document.
A very pragmatic approach could be to start a background job and email the response to the client. 1O-1 they'd prefer that rather than having a browser window open all afternoon.
I have a PHP script that seemed to stop running after about 20 minutes.
To try to figure out why, I made a very simple script to see how long it would run without any complex code to confuse me.
I found that the same thing was happening with this simple infinite loop. At some point between 15 and 25 minutes of running, it stops without any message or error. The browser says "Done".
I've been over every single possible thing I could think of:
set_time_limit ( session.gc_maxlifetime in the php.ini)
memory_limit
max_execution_time
The point that the script is stopped is not consistent. Sometimes it will stop at 15 minutes, sometimes 22 minutes.
Please, any help would be greatly appreciated.
It is hosted on a 1and1 server. I contacted them and they don't provide support for bugs caused by developers.
At some point your browser times out and stops loading the page. If you want to test, open up the command line and run the code in there. The script should run indefinitely.
Have you considered just running the script from the command line, eg:
php script.php
and have the script flush out a message every so often that its still running:
<?php
while (true) {
doWork();
echo "still alive...";
flush();
}
in such cases, i turn on all the development settings in php.ini, of course on a development server. This display many more messages, including deprecation warnings.
In my experience of debugging long running php scripts, the most common cause was memory allocation failure (Fatal error: Allowed memory size of xxxx bytes exhausted...)
I think what you need to find out is the exact time that it stops (you can set an initial time and keep dumping out the current time minus initial). There is something on the server side that is stopping the file. Also, consider doing an ini_get to check to make sure the execution time is actually 0. If you want, set the time limit to 30 and then EVERY loop you make, continue setting at 30. Every time you call set_time_limit, the counter resets and this might allow you to bypass the actual limits. If this still isn't working, there is something on 1and1's servers that might kill the script.
Also, did you try the ignore_user_abort?
I appreciate everyone's comments. Especially James Hartig's, you were very helpful and sent me on the right path.
I still don't know what the problem was. I got it to run on the server with using SSH, just by using the exec() command as well as the ignore_user_abort(). But it would still time out.
So, I just had to break it into small pieces that will run for only about 2 minutes each, and use session variables/arrays to store where I left off.
I'm glad to be done with this fairly simple project now, and am supremely pissed at 1and1. Oh well...
I think this is caused by some process monitor killing off "zombie processes" in order to allow resources for other users.
Run the exec using "2>&1" to log anything including stderr.
In my output I managed to catch this:
...
script.sh: line 4: 15932 Killed php5-cli -d max_execution_time=0 -d memory_limit=128M myscript.php
So something (an external force, not PHP itself) is killing my process!
I use IdWebSpace which is excellent BTW but I think most shared hosting providers impose this resource/process control mechanism just to be sane.