A similar question was already asked (
Performing a Stress Test on Web Application?), but I'd like to test a web application that prevents double-submits and takes some counter-XSRF actions and therefore REQUIRES JavaScripts to be evaluated.
Has anybody done stress tests with web apps that require (and use) JS and any experience to share?
jMeter wouldn't work I guess...
Thanks!
Watir?
Watir is a simple open-source library for automating web browsers.
Watir drives browsers the same way people do. It clicks links, fills in forms, presses buttons. Watir also checks results, such as whether expected text appears on the page.
It drives Internet Explorer, but is also functional with Firefox (and Safari to some extent).
The problem with Watir and Selenium RC or any other full browser solution is that they need a full browser :P
Browsers are very expensive to run, often requiring 300MB or more of RAM. Multiply those requirements by even 100 and you need massive hardware. Fortunately, there is a solution: I recently started a company that does exactly what you're looking for.
Check out http://browsermob.com and you can run a limited test (up to 25 users) to get a feel for the app. Feel free to contact us if you have any questions at all!
One solution that may be worth pursuing is to run Selenium on Amazon EC2 to provide the scalability you need. There is a tutorial over at Selenium using a sample that ships with Selenium grid. Windows machines are 12.5 cents an hour for a small machine meaning that a 500 machine test is going to cost $62.50 an hour.
PROS:
Selenium runs in a real browser meaning that your Javascript is executing as it would on a client
Low cost - trying to do this on your own hardware would cost significantly more
CONS:
You would have to establish network connectivity from Amazon to your application
The testers I work with use Bad Boy for load testing. I'm fairly certain you can test interactions that use javascript, so you should be able to test stuff like double-submits.
As far as your backend is concerned, it doesn't matter what triggers a request whether it's from JavaScript or a load testing tool as long as the request is valid.
You can create a bunch of fake requests that do lots of different things (hopefully representative of actual usage patterns) and slam your webserver with a load testing tool.
There's a bunch out there:
jMeter
http_load
Grinder
httpperf
Because JMeter is not a browser, it won't interpret the JavaScript code on the page you GETting:
JMeter does not process Javascript or applets embedded in HTML pages.
[JMeter Wiki]
So, what can you do? You can add WebDriver to JMeter test and by this, evaluate the web pages.
Web Driver Sampler automates the execution and collection of
Performance metrics on the Browser (client-side). A large part of
performance testing, up to this point, has been on the server side of
things. However, with the advancement of technology, HTML5, JS and CSS
improvements, more and more logic and behaviour have been pushed down
to the client. This adds to the overall perceived performance of
website/webapp, but this metric is not available in JMeter. Things
that add to the overall browser execution time may include:
Client-side Javascript execution - eg. AJAX, JS templates
CSS transforms - eg. 3D matrix transforms, animations
3rd party plugins - eg. Facebook like, Double click ads, site analytics, etc
All these things add to the overall browser execution time, and this
project aims to measure the time it takes to complete rendering all
this content.
Official guide: https://jmeter-plugins.org/wiki/WebDriverTutorial/
I've tried Badboy which is OK. The big, fat, heavy tool is SilkTest. It requires a lot of programming to get up and running, but you can get something very solid done!
If you only need to stress test request from e.g. IIS log files, I have a custom build tool. I will publish it at CodePlex very soon.
Selenium RC is another alternative.
Also related, check out my recent article on Ajaxian. I think it does a good job of explaining why real browsers do matter and why executing JavaScript is becoming more important for load testing.
http://ajaxian.com/archives/why-load-testing-ajax-is-hard
There's a new tool in this area called k6
it has a way to access the DOM, and I'm planning to try it in a project.
background story:
You can visit this and this blog.
maybe it will help.
Related
I am interested in automating repetitive data entry in some forms for a website I frequent. So far the tools I've looked up that would provide support for this in a headless fashion could be Selenium WebDriver and Mechanize.
My question is, is there a fundamental technical difference in using once versus the other? Selenium is mostly used for testing. I've also noticed some folks use it for doing exactly what I'm looking for, and that's automating data entry. Testing becomes a second benefit in that case.
Is there reasons to not use Selenium for what I want to do over Mechanize? Does it not matter and both of these tools will work?
I'm not asking which is better, I'm asking which is the right tool for the job. Perhaps I'm not understanding the premise behind the purpose of each tool.
These are completely different tools that somewhat "cross" in the web-scraping, web automation, automated data extraction scope.
mechanize is a mature and widely-used tool for programmatic web-browsing with a lot of built-in features, like cookie handing, browser history, form submissions. The key thing to understand here is that mechanize.Browser is not a real browser, it cannot execute and understand javascript, it cannot send asynchronous requests often needed to form a web page.
This is where selenium comes into play - it is a browser automation tool which is also widely used in web-scraping. selenium usually becomes a "fall-back" tool - when someone cannot web-scrape a site with mechanize or RoboBrowser or MechanicalSoup (note - another alternatives) because of, for instance, it's javascript "heaviness", the choice is usually selenium. With selenium you can also go headless, automating PhantomJS browser, or having a virtual display. As a commonly mentioned drawback, performance is often mentioned - with selenium you are working with a target site as a real user in a web browser, which is loading additional files needed to form a page, making XHR requests, rendering etc.
And this itself does not mean you should use selenium everywhere - choose the tool wisely, choose it because it fits the problem better, not because you are more familiar with an instrument.
Also note that you should, first, consider using an API (if provided by the target website) instead of going down to web-scraping. And, if it comes to it, be a good web-scraping citizen:
How to be a good citizen when crawling web sites?
Web scraping etiquette
I have a personal need to scrape/automate-access-to my linkedin account (copy my contacts, etc), and obviously the site is too ajaxy to just use wget, urllib, etc.
I cannot use the LinkedIn API, as it happens to restrict some use cases I'm interested in.
I am proficient in Python and Javascript. I've used webdriver in the past for small scraping projects, but it was long ago enough that there's probably a similar overhead in re-learning it vs learning phantomjs.
I am not planning to run any kind of high-volume cluster-based scraping operation, this is all going to be running on my local machine at some appropriate rate limit so as not to piss off linkedin. It's mostly just for personal convenience, automation, etc.
I've heard good things about phantomjs, but I'd like to understand what if any advantage it has over webdriver (or vice versa). I guess phantomjs is "headless", meaning it doesn't actually have to run a browser, which I guess makes it easier to write command line scripts or consume fewer resources or some other property that I would love to have explained to me!
I can appreciate the argument that webscraping programs should be javascript, since that's more of a browser-native language, but would love to hear if that's a primary reason why people are using phantomjs (or one of its cousins)
I've used both Selenium and Phantom/Casper in scraping jobs, and also used both in functional testing jobs. If I was going to do what you describe I would choose CasperJS. I would choose CasperJS over PhantomJS because:
Easier to describe the flow of steps. (You have to deal with all the async callbacks when using PhantomJS directly.)
SlimerJS can be swapped in to have it use Gecko (i.e. Firefox), with no additional effort. (I don't think this will matter with LinkedIn, but PhantomJS 1.9.x is based on a fairly old WebKit, so when sites use newer HTML5 features it can sometimes fail.)
Reasons to choose CasperJS over Selenium:
The flow of steps is quite easy to describe in CasperJS.
Selenium feels more like hard work. This might be because PHP is my preferred glue language, and since Selenium 2.0, PHP has been treated as an outsider. But also it has the philosophy of only allowing actions that a user could do in the browser with keyboard and mouse. This is sometimes not flexible enough.
Selenium breaks each time Firefox gets updated, and I have to install the latest version. Irritating. (PhantomJS and SlimerJS have their browser internally, so are cleanly independent of system updates to your desktop browser.)
As you are proficient in both Python and JavaScript, I would say none of the above are killer reasons. It doesn't really matter which you choose, the effort is going to be roughly the same.
I am a novice in testing.
I am working on Linux.
I was reading about testing in headless mode and came across two things. One was X virtual frame buffer which does graphical operations in memory. So, no output is displayed. The implementation details I found in this link http://www.seleniumtests.com/2012/04/headless-tests-with-firefox-webdriver.html.
The other one that I came across was HtmlUnitDriver. This also does not open any browser while running the test. I wrote a basic sample code using HtmlUnitDriver and the assertions seem to work fine.
I understand that HtmlUnitDriver doesn't work too well with javascript. But apart from this, are there any major differences to choose one over the other?
I am going to be testing a web application that does have some abount of javascript in it.
I am a novice in this field. So, any answers, suggestions, etc. will be appreciated.
Thank you in advance
From my experience with both approaches:
HtmlUnit will in most practical cases be faster than a real browser with xvfb -- simply because it doesn't spend time rendering the pages. (A data point: 17 secs. HtmlDriver vs. 62 secs. FirefoxDriver for a specific test suite I'm using now).
It is easier to run several tests concurrently -- and it consumes a lot less resources -- using HtmlUnit. This can be very important if you have a large number of tests and you need them to finish fast (e.g. you want to follow the 10-minute-build rule).
As you said, HtmlUnit has its own quirks with JavaScript and the DOM. Not better or worse than any other browser (Firefox, Safari, IE, Chrome, ... -- they all have their own quirks), but one on which it is very questionable to spend time fixing bugs. I also find such bugs very difficult to diagnose, but that may be only my ignorance.
One advantage of real browsers + xvfb is that you can always use the exact same tests without xvfb and see what's going on -- possibly even use a console to run some JavaScript to diagnose issues. I sometimes feel quite blind when working with HtmlUnit, and because of above-mentioned quirks you can't always use the same exact tests code in both environments.
So, in summary, unless total test duration is important and you're ready to spend some time fighting HtmlUnit, it's just easier to go with a regular browser + xvfb.
I also like using xvnc, which has the added benefit of allowing you to connect to the screen of a running test and see what's going on (not sure whether you can do that with xvfb).
My duty is to pick up and continue developing PHP website for a small sized business client. Project has no testing code. I want to quickly establish at least very basic regression testing for the backend of the site.
I need to test the full contents of the web page char to char. Must see the diff of failed tests.
I need to be able to set up cookies and GET/POST data.
Once every few days I am updating the local database from production database. I would like to then have an overview of failed tests and very quickly update my test suits so that everything is passing again.
Is using WatiN or Selenium a good idea? My local environment is Linux.
About Selenium (and only Selenium as I don't know WatiN) - it can only do what you can do in your broswer. It can click, type in fields, submit forms, take screenshots (that's a very good one), set up cookies (so yes to this one). You can always set up GET data through the URL. But I am not aware of any technique in Selenium that would allow you to set up POST data in any other way than navigating in a browser. Also, because the tests are in your browser, they are not particularly fast. E.g, on our product, a single thorough test with ~250 steps takes about 10 minutes on my computer to complete. Of course, you can always divide that between many computers using Selenium Grid. It's just more work.
To conclude it - I'd say yes, Selenium is good for your needs as there are so many ways to write a good test in it that everyone finds his style. It is good for quick checks, functionality affirmations, but also for full-scale tests etc. But if you want to do some really advanced stuff, then that's a job for a long time. Selenium offers so much functionality in so many different ways that it is definitely a full-time job to understand them and know how to use them.
Try Selenium-IDE for 20 minutes. It is just an addon for Firefox that can record your actions and then replay them. If you like what you see, go for it. If no, hire someone who will.
I'm not sure if I am too late here, but in regards to WatiN it is only IE based, so if you plan to use any other browser you are better off with Selenium WebDriver (though WatiN has some Firefox support). From what I have found (I have used both WatiN and Selenium) Selenium can achieve more low level interactions (also see Selenium Grid), but really I think it is dependant on what you are looking to achieve and personal preference. If you have time to write your own wrapper to interact with WatiN/Selenium you will find the tests themselves are rather quick to run. Also, the beauty of Automation is that once these tests are written you can run them and walk away while they complete.
In the bad old days of interactive console applications, Don Libes created a tool called Expect, which enabled you to write Tcl scripts that interacted with these applications, much as a user would. Expect had two tremendous benefits:
It was possible to script interactions that otherwise would have had to be repeated by hand, tediously. A classic example was dialup Internet access hell (from the days before PPP).
It was possible to write scripts to test one's own interactive applications, programmatically, as part of a regression suite.
Today most interactive applications are on the web, not on the console. Hence my question: is there any tool that provides the ability to interact with web pages and web forms programmatically, much as Expect provides the ability to interact with console applications programmatically?
(The closest thing I am aware of is Chickenfoot.)
You might be looking for Selenium
I've used Selenium RC in conjunction with Python to drive web page interactions programmatically. This has allowed me to write pretty extensive user tests in which forms and inputs are driven and their results are measured.
Check out the Selenium IDE on Firefox (as mentioned above). It allows you to record tests in the browser and play them back, either using the IDE itself, or the Remote Control app.
Perl Mechanize works pretty well for this exact issue.
HTTPS and some authentication issues are tricky at times. I will be posting couple questions about those in the future.
I did a ton of Expect work in a former life and always thought Don Libes' Expect book was one of the best-written and most enlightening technical books I'd ever seen.
Hands down I would say that Perl's WWW::Mechanize library is what you want. I note above that you were having trouble finding documentation. There is good documentation for it! Look up the module's distribution on search.cpan.org and see what all is packaged with it. There's a FAQ, Cookbook with examples, etc. Plus I've always been able to get help on the web. If you can't get it here, try at use.perl.org or perlmonks.org. WWW::Mechanize's author, Andy Lester, is present on Stack Overflow. (He's also an all around friendly and helpful guy.)
I believe WWW::Mechanize also has a program that is analogous to Expect's autoexpect program: you set up a proxy process running this program as a server, point your browser to it as a proxy, perform the actions you want to automate, and then the proxy program gives you a WWW::Mechanize program for you to use as a base for your project. (If it works like autoexpect, you will certainly want to make modifications from there.)
As mentioned above, WWW::Mechanize is a browser (to be more exact, it is a web client or http client) that happens to be programmable. The last time I looked, there was even work in progress to make it support JavaScript.
In addition to Selenium, if you're doing the Ruby/Rails thing, there's Webrat.