As part of the performance testing of a Drupal web application and its infrastructure, I installed sensors on all hardwares and middleware bricks, but I dont know how to collect php metrics from mod_php (response time between apache and mod_php, ...).
Thank you for your help
You've not said what you are rying to acheive with this monitoring / testing. Mod_php (like all apache modules) is a shared library, hence there is no measurable response time between apache and mod_php.
Measuring request response time on the webserver is a good idea- it gives some insight into into infrastrcture problems but tells you very little about what performance the client is experiencing (at least it shouldn't). Hint: you should be logging and analysing %D. This is where RUM comes in - although this is hard to implement on a test environment.
It is possible to project data from PHP into Apache using apache_note() (e.g. for logging memory_get_peak_usage()).
Other things you should plumb in:
an opcode cache manager
SAR
mod_status
...and capture the data.
Related
I am setting up a very simple HTTP server for the first time, am considering my options, and would appreciate any feedback on the best way to proceed. My goal is pretty simple: I'm not serving any files, I only need to respond to a very specific HTTP POST request that will contain geolocation data, run some Python code, and return the results as JSON. I do need to be able to respond to multiple simultaneous requests. I would like to use HTTPS.
In looking on stackoverflow it seems I can potentially go with BaseHTTPServer and ThreadingMixIn, or Apache and mod_wsgi. I already have Apache installed, but have never configured it. Are there compelling reasons to go the more complicated Apache route (more complicated to me, because I will need to do research on configuring Apache and getting mod_wsgi going but already have a test instance of BaseHTTPServer up and running), or is it equally safe, secure (very important), and performance-oriented to use BaseHTTPServer for something so simple?
BaseHTTPServer is not a production grade server.
If you don't understand how to set up Apache, but want to get something with mod_wsgi running quickly and easily, then you probably want to look at mod_wsgi express.
This gives you a way of installing mod_wsgi using Python 'pip' and also provides you a way of starting up Apache/mod_wsgi with a auto generated Apache and mod_wsgiconfiguration such that you don't even need to know how to configure Apache.
The next version of mod_wsgi express to be released (version 4.3.0, likely released this week), can even set up a HTTPS site for you, with you just needing to have obtained a valid certificate or generated a self signed certificate.
I would suggest if interested you use the mod_wsgi mailing list to ask for more details about using mod_wsgi express for running a HTTPS site.
http://code.google.com/p/modwsgi/wiki/WhereToGetHelp?tm=6#Asking_Your_Questions
You can start playing around though with it for a normal HTTP site by following instructions at:
https://pypi.python.org/pypi/mod_wsgi
I am running a Perl CGI tool that executes a system command (Unix) which may run for a few seconds up to an hour.
After the script is finished, the tool should display the results log on the screen (in a browser).
The problem is that after about 5 minutes I get a timeout message "Gateway Time-out" - the system command continue to run but I'm unable to display to the user the results of the run.
In the Apache config file (httpd.conf): Timeout 300.
Is there a simple way ordering the Apache to increase the timeout only for a specific run?
I don't really want to change the Apache timeout permanently (or should I?) and not dramatically update the code (a lot of regression tests).
Thanks in advance.
Mike
Make the script generate some output every once in a while. The timeout is not for running the program to completion, but is a timeout while Apache is waiting for data. So if you manage to get your program to output something regularly while running, you will be fine.
Note that HTTP clients, i.e. browsers, also have their own timeout. If your browser does not get any new data from the web server five minutes (typically), the browser will declare a timeout and give up even if the server is still processing. If your long running processing gives some output every now and then, it will help against browser timeouts too!
For completeness:
Though the accepted answer is the best (it's variously known as KeepAlive packets in TCP/IP, or Tickle packets way back in appletalk days) you did ask if you can do dynamic Apache config.
An apache module could do this. Oh, but that's a pain to write in C.
Remember that mod_perl (and to some extent mod_python, though it's deprecated) do not only handlers but wrap the internal config in perl as well. You could write something complicated to increase the timeout in certain situations. But, this would be a bear to write and test, and you're better off doing what Krisku says.
There doesn't seem to be any way to specify a timeout on the <!--#include virtual=... --> directive, but if you use mod_cgid instead of mod_cgi then starting with Apache 2.4.10 there's a configurable timeout parameter available which you can specify in httpd.conf or .htaccess:
CGIDScriptTimeout nnns
...where nnn is the number of seconds that Apache will allow a cogitating CGI script to continue to run.
Caveat: If you use PHP with Apache, then your Apache is presumably configured in /etc/httpd/conf.modules.d/00-mpm.conf to use "prefork" MPM (because PHP requires it unless built with thread-safe flags), and the default Apache installation used mod_cgi with the prefork MPM, so you'll probably need to edit /etc/httpd/conf.modules.d/01-cgi.conf to tell Apache to use mod_cgid instead of mod_cgi.
Although the comment in 01-cgi.conf says, "mod_cgid should be used with a threaded MPM; mod_cgi with the prefork MPM," that doesn't seem to be correct, because mod_cgid seems to work fine with prefork MPM and PHP, for me, with Apache 2.4.46.
Although that doesn't give you complete control over server timeouts, you could specify a different CGIDScriptTimeout setting for a particular directory (e.g., put your slow .cgi files in the ./slowstuff/ folder).
(Of course, as krisku mentioned in the accepted answer, changing CGIDScriptTimeout won't solve the problem of the user's web browser timing out.)
I am designing a AWS deployment solution for a new dynamic website project. I have acquired an EC2 instance for testing the environment. Need some help on how do I do a load testing on an Ec2 instance to determine how many HTTP requests it can safely handle... P.S. I am new to the AWS platform.
Thanks...
RedLine offers an EC2 Load Testing solution that will automate the distribution of load tests on your own EC2 instances.
Late to the party but could help someone in the future:
A possible tool for load tests, stress tests, whatever you may call them, is Apache JMeter, but there are plenty of alternatives.
A simple starting setup, further explained in this excellent tutorial on DigitalOcean, can exist of a Thread Group containing an HTTP Request Sampler and a View Results in Table Listener. The Thread Group can be used to configure the amount of "clients" you want to simulate. The Request Sampler will be used to configure the server's properties (hostname, path, etc). The Table View Listener outputs a handy CSV file that can be used to calculate means, compare different types of EC2 instances,...
JMeter is a beautiful program with a GUI that can be run on your local workstation, producing an XML file that can be executed on another EC2 instance, for instance. You can even do simple manual edits to the XML file on your server afterward, if necessary.
Take a look at Amazon's testing policy to make sure you're not doing anything illegal.
A couple of quick points;
Set the environment up exactly like it's supposed to run. If there's a database involved, you'll want to involve that in the testing too. Synthetic <?php echo "ok"; CPU based benchmarks won't help you much since normally very little of the time spent replying to HTTP requests is actual CPU time.
A recommendation is to use a service for the benchmarking. Setting load testing up is not without its complexities, and unless you consider benchmarking your core business, you're probably better off using something like Neustar to load and measure your site (there are many services, they're not necessarily what fits you best, just pulled one out of memory)
Of course you can set a load test up yourself, but getting that done right is not anything that can be described in a few sentences. There are very well paid people that only do that for a living :)
There is good experience in using curl-loader aka Davilka tool, also on Amazon EC2 env
http://curl-loader.sourceforge.net
When using XAMPP (1.7.5 Beta) under Windows 7 (Ultimate, version 6.1, build 7600), it takes several seconds before pages actually show up. During these seconds, the browser shows "Waiting for site.localhost.com..." and Apache (httpd.exe, version 2.2.17) has 99% CPU load.
I have already tried to speed things up in several ways:
Uncommented "Win32DisableAcceptEx" in xampp\apache\conf\extra\httpd-mpm.conf
Uncommented "EnableMMAP Off" and "EnableSendfile Off" in xampp\apache\conf\httpd.conf
Disabled all firewall and antivirus software (Windows Defender/Windows Firewall, Norton AntiVirus).
In the hosts file, commented out "::1 localhost" and uncommented "127.0.0.1 localhost".
Executed (via cmd): netsh; interface; portproxy; add v6tov4 listenport=80 connectport=80.
Even disabled IPv6 completely, by following these instructions.
The only place where "HostnameLookups" is set, is in xampp\apache\conf\httpd-default.conf, to: Off.
Tried PHP in CGI mode by commenting out (in httpd-xampp.conf): LoadFile "C:/xampp/php/php5ts.dll" and LoadModule php5_module modules/php5apache2_2.dll.
None of these possible solutions had any noticeable effect on the speed. Does Apache have difficulty trying to find the destination host ('gethostbyname')? What else could I try to speed things up?
Read over Magento's Optimization White Paper, although it mentions enterprise the same methodologies will and should be applied. Magento is by no means simplistic and can be very resource intensive. Like some others mentioned I normally run within a Virtual Machine on a LAMP stack and have all my optimization's (both at server application levels and on a Magento level) preset on a base install of Magento. Running an Opcode cache like eAccelerator or APC can help improve load times. Keeping Magento's caching layers enabled can help as well but can cripple development if you forget its enabled during development, however there are lots of tools available that can clear this for you from a single command line or a tool like Alan Storms eCommerce Bug.
EDIT
Optimization Whitepaper link:
https://info2.magento.com/Optimizing_Magento_for_Peak_Performance.html
Also, with PHP7 now including OpCache, enabling it with default settings with date/time checks along with AOE_ClassPathCache can help disk I/O Performance.
If you are using an IDE with Class lookups, keeping a local copy of the code base you are working on can greatly speed up indexing in such IDEs like PHPStorm/NetBeans/etc. Atwix has a good article on Docker with Magento:
https://www.atwix.com/magento/docker-development-environment/
Some good tools for local Magento 1.x development:
https://github.com/magespecialist/mage-chrome-toolbar
https://github.com/EcomDev/EcomDev_LayoutCompiler.git
https://github.com/SchumacherFM/Magento-OpCache.git
https://github.com/netz98/n98-magerun
Use a connection profiler like Chrome's to see whether this is actually a lookup issue, or whether you are waiting for the site to return content. Since you tagged this question Magento, which is known for slowness before you optimize it, I'm guessing the latter.
Apache runs some very major sites on the internets, and they don't have several second delays, so the answer to your question about Apache is most likely no. Furthermore, DNS lookup happens between your browser and a DNS server, not the target host. Once the request is sent to the target host, you wait for a rendered response from it.
Take a look at the several questions about optimizing Magento sites on SO and you should get some ideas on how to speed your site up.
In my opinion, web server is responsible to deliver content to client. If it is static content like pictures and static html document, web server just deliver them as bitstream directly. If it is some dynamic content that is generated during processing client's request, the web server will not generate the conetnt itself but call some external proram to genearte the content.
AFAIK, this kind of dynamice content generation technologies include the following:
CGI
ISAPI
...
And from here, I noticed that:
...In IIS 7, modules replace ISAPI
filters...
Is there any others? Could anyone help me complete the above list and elabrate on or show some links to their evolution? I think it would be very helpful to understand application such as IIS, TomCat, and Apache.
I once wrote a small CGI program, and though it serves as a content generator, it is still nothing but a normal standalone program. I call it normal because the CGI program has a main() entry point. But with the recenetly technology like ASP.NET, I am not writing complete program, but only some class library. Why does such radical change happens?
Many thanks.
well, the biggest missing piece in your question is that you can have the webserver generating the content dynamically as well. This is common with most platforms outside of PHP and Perl. You often set that website behind apache or nginx used as a proxy, but it doesn't "call an external progam" in any reasonable sense, it forwards the http request to the proxied server. This is mostly done so you can have multiple sites on the same server, and also so you can have apache/nginx protect you against incorrect requests.
But sure, we can, for the sake of the question, say that "proxying" is a way to call an external program. :-)
Another way to "call the external program" is Pythons WSGI, where you do call a permanently running server. So again you don't start an external program, it's more like calling the module in ASP (although it's a separate program, not a module, but you don't start it with every request, you use an API).
The change from calling external programs as in CGI to calling modules like in ASP.NET, process with WGI or proxying to another webserver happened because with CGI you have to start a new prpogram for each request. The PERL/PHP interpreter needs to be laoded into memory, and all modules they use as well. This quickly becomes very heavy and process/memory intensive.
Therefore, to be able to use bigger systems that are permanently running, other techniques have been developed. Most of them are platform/language dependent, and the only one that is platform independent is really to make a complete webserver and then use apache/nginx as a proxy in front (in which case the apache/nginx strictly isn't necessary any more).
I hope this cleared things up a bit.
fastcgi and wsgi are two more interfaces content generators can use to talk to a webserver -- the reason more recent interfaces aren't complete programs is that forking and executing things that expect to be executables is costly.
OTOH, writing your little generator in such a way that it doesn't leak anything between invocations is harder than having the liberty to just exit at the end (and rely on environment variables and command line arguments like a normal executable).
This is all for performance reasons, but then you have more complicated content generators and process management in the webservers.