I have been using selenium webdriver as my main method to do functional tests. So far its been working greate with our product.
I need to do some performance and/or load tests on the website, I was wondering if there is a tool that would incorporate my selenium tests or a tool which i can use with the recorded tests as the base.
Currently i am using selenium webdriver with C#
Any help is appreciated.
Given you have tests written in C# the most obvious way would be using Visual Studio Load Testing capabilities.
If you are looking for a free and open-source solution I would recommend going for Apache JMeter. JMeter has integration with Selenium via WebDriver Sampler plugin so you should be able to run your Selenium tests in multi-threaded manner. However you will need to convert your C# code into one of the WebDriver Sampler supported languages (default is JavaScript)
Remember that Selenium tests are very resource-intensive as real browsers consume a lot of CPU/RAM so the number of virtual users you will be able to mimic this way will be very limited. So recommended approach is creating main load on a HTTP protocol level and use one Selenium instance to check rendering speed while your application is under the load.
You can install WebDriver Sampler plugin using JMeter Plugins Manager.
Why do you need to run full browsers. HTTP layer tests are far simpler? You also should be looking at only a subset of business processes in performance which generate a preponderance of load.
Related
I would like to record the network requests of a Selenium test. Rather than using jmeter GUI, I would like to automate this process in a script. The idea would be to run a selenium test and record it simultaneously. Is there a built in way to do this?
GUI Jmeter is ideally for development of scripts, which are then run headless to get true performance figures.
You can develop selenium webdriver tests as junit tests, which integrate to JMeter very easily.
Running a recorded Selenium IDE test would not be so straightforward, better to export it as Java Webdriver.
Make sure you follow junit naming conventions and annotations (depending which version you use). Write the test class, including junit 'test' methods, then create a jmeter test plan with a junit sampler, and configure the junit sampler to run your test method[s].
maven and plugins can be used to make it all run seamlessly in headless mode on any host.
Once you have developed one of these, it will be easy to review the jmx test script and automate the process of creating more jmeter tests, if that is what you meant by automating the process.
Start JMeter's Proxy Server
Configure your Selenium script to use JMeter as a proxy. See Using a Proxy guide for configuration details for each driver
Run Selenium test - all requests should be recorded by JMeter.
Add the next test elements to JMeter test plan:
HTTP Cookie Manager - to represent browser cookies
HTTP Cache Manager - to represent browser cache
follow recommendations from How to make JMeter behave more like a real browser to properly configure embedded resources retrieval, user agent, request defaults, etc.
Configure Thread Group parameters according to your load scenario.
Replay the test.
I expect that you will need to apply some correlation, but it may be not required.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I am trying to understand testing framework better and been looking into Selenium. I've used HTMLUnit before, mainly when I needed to scrape some information off website or the likes.
In the context of writing test automation, what's the advantage / disadvantages of Selenium vs HTMLUnit? Looks to me Selenium is more complicated to set up than HTMLUnit, although at the same time there's a HTMLUnitDriver for Selenium which I think behave the exact same way as in HTMLUnit itself?
Selenium obviously provides more robust framework, it has the Selenium RC for pararel testing, it also has different browser drivers that can be used - although when you used the browser drivers, the test will actually open/close a browser application rather than headless.
May be I am not understanding Selenium correctly. Some directions and pointers would be great!
On another note - a separate question - I am also looking at doing automated testing on mobile browser, I see that Selenium has an IPhoneDriver for it, but then this is not a headless testing either as it requires actual iOS simulator.
Is there anyway to do headless testing on mobile sites? Would changing user-agent be sufficient? I've seen a couple posts around changing user-agent that seem to have their own challenges, eg. Set user-agent in Selenium RC
Thanks a lot!
well, would try to explain differences in detail.
Speaking about parallel testing, it better to use selenium grid.
Basic concept of selenium RC and selenium grid.
You can get into more details here
Some words about selenium webDriver:
The primary new feature in Selenium 2.0 is the integration of the WebDriver API. WebDriver is designed to providing an simpler, more concise programming interface along with addressing some limitations in the Selenium-RC API. Selenium-WebDriver was developed to better support dynamic web pages where elements of a page may change without the page itself being reloaded. WebDriver’s goal is to supply a well-designed object-oriented API that provides improved support for modern advanced web-app testing problems.
How Does WebDriver ‘Drive’ the Browser Compared to Selenium-RC?
Selenium-WebDriver makes direct calls to the browser using each browser’s native support for automation. How these direct calls are made, and the features they support depends on the browser you are using. Information on each ‘browser driver’ is provided later in this chapter.
For those familiar with Selenium-RC, this is quite different from what you are used to. Selenium-RC worked the same way for each supported browser. It ‘injected’ javascript functions into the browser when the browser was loaded and then used its javascript to drive the AUT within the browser. WebDriver does not use this technique. Again, it drives the browser directly using the browser’s built in support for automation.
WebDriver and the Selenium-Server
You may, or may not, need the Selenium Server, depending on how you intend to use Selenium-WebDriver. If you will be only using the WebDriver API you do not need the Selenium-Server. If your browser and tests will all run on the same machine, and your tests only use the WebDriver API, then you do not need to run the Selenium-Server; WebDriver will run the browser directly.
There are some reasons though to use the Selenium-Server with Selenium-WebDriver.
You are using Selenium-Grid to distribute your tests over multiple
machines or virtual machines (VMs).
You want to connect to a remote machine that has a particular browser
version that is not on your current machine.
You are not using the Java bindings (i.e. Python, C#, or Ruby) and
would like to use HtmlUnit Driver
Selenium-WebDriver’s Drivers
WebDriver is the name of the key interface against which tests should be written, but there are several implementations. These include:
HtmlUnit Driver
This is currently the fastest and most lightweight implementation of WebDriver. As the name suggests, this is based on HtmlUnit. HtmlUnit is a java based implementation of a WebBrowser without a GUI. For any language binding (other than java) the Selenium Server is required to use this driver.
Pros
Fastest implementation of WebDriver
A pure Java solution and so it is platform independent.
Supports JavaScript
Cons
Emulates other browsers’ JavaScript behaviour (see below)
JavaScript in the HtmlUnit Driver
None of the popular browsers uses the JavaScript engine used by HtmlUnit (Rhino). If you test JavaScript using HtmlUnit the results may differ significantly from those browsers.
When we say “JavaScript” we actually mean “JavaScript and the DOM”. Although the DOM is defined by the W3C each browser has its own quirks and differences in their implementation of the DOM and in how JavaScript interacts with it. HtmlUnit has an impressively complete implementation of the DOM and has good support for using JavaScript, but it is no different from any other browser: it has its own quirks and differences from both the W3C standard and the DOM implementations of the major browsers, despite its ability to mimic other browsers.
With WebDriver, we had to make a choice; do we enable HtmlUnit’s JavaScript capabilities and run the risk of teams running into problems that only manifest themselves there, or do we leave JavaScript disabled, knowing that there are more and more sites that rely on JavaScript? We took the conservative approach, and by default have disabled support when we use HtmlUnit. With each release of both WebDriver and HtmlUnit, we reassess this decision: we hope to enable JavaScript by default on the HtmlUnit at some point.
To investigate deeper into webDriver's setUp see this
From HtmlUnit documentation:
HtmlUnit is not a generic unit testing framework. It is specifically a way to simulate a browser for testing purposes and is intended to be used within another testing framework such as JUnit or TestNG.
So to conclude Selenium and HtmlUnit difference:
HtmlUnit is a java based implementation of a WebBrowser without a GUI and a way to simulate a browser for testing purposes and Selenium-WebDriver makes direct calls to the browser using each browser’s native support for automation. we can see that HtmlUnit provides API without GUI possibility for automation whereas WebDriver provides internal browsers' possibilities for automation.
Speaking about mobile automation,
Selenium also has an iPhone Driver
iPhone Driver wiki article
and Android Driver
Android Driver wiki article
See also this presentation
Unfortunately I can not give you my working experience evaluation of mobile drivers as I deal with web automation (no mobile). Also know that Cucumber (automation tool) is popular among mobile automators.
see this and this.
Hope it come a lil bit more clear for you now =)
Selenium and HTMLUnit are somewhat similar in concept, but Selenium is more mature/robust and has a lot more features.
Note that Selenium encompasses the recording (IDE) plugin for Firefox, which allows you to record tests and the RC/WebDriver automation framework which essentially drives a browser. The two can be used together to make test creation very easy.
The only advantage I could see to using HTMLUnit is that it is less resource intensive, so you could potentially run tests on less hardware, but with Selenium's parallel support even that isn't really true anymore.
When running tests from Jenkins overnight, you typically have no access to a windowing system such as X11 or Windows in which to run the web browser. I therefore see a benefit of using the HTMLUnit web driver in that case since it doesn't require access to a windowing system.
In my experience, HtmlUnit does a fine job with browsing automation, but might get a bit buggy when dealing with Javascript. I actually came to a case in which I wasn't able to automate an image download with HtmlUnit, and had to turn to Selenium, which performed beyond my expectations. The case is actually registered in an SO thread.
I have used Selenium WebDriver for automation.
There is a very easy method to causing the browser to be headless.
Simply apply ChromeOptions (In my case, other DriverOptions are available)
ChromeOptions("Headless")
There are many proficient methods that can come from using Options or Services, as another example
This will stop the Driver/CommandPrompt window from 'appearing' so it will remain 'silent' and unexposed.
ChromeDriverServices ("Silent")
At least at UX systems you can use for example Xvfb and point the browsers to that display to make them "headless"
See also http://infiniteundo.com/post/54014422873/headless-selenium-testing-with-firefox-and-xvfb
or How do I run Selenium in Xvfb?
Hi, I want to get into test automation, and the two tools I found during my first web search are Selenium and Jmeter.
Which one do you think is the first to have a look at? Or do I need both tools as they're totally different?
What I would need is the possibility to do Clientside-Certificate-Authentication, filling forms with different Information, and checking result pages.
Apache JMeter is definitely tool for performance testing and load/stress tests. But you can use it also for functional tests as well (in your example: fill form ->check results but with checking if results are as expected - but better don't do functional testing with him)
For functional testing on the other hand there are Selenium and also Canoo web test.
So final answer will be to combine those two. (I was using JMeter for performance tests and canoo web test for functional testing, but I guess that Selenium is much better choice now)
Use Selenium for your functional tests
Use JMeter for stress tests, and measure performance
In both cases you can record a session, so you can start your Selenium or JMeter engine, do something in your browser and then stop recording. After that you can use Selenium or JMeter to run the session recorded again.
Selenium tests browser fields and buttons. In Selenium you can fill a input field and click a button, wait for the page load and then inspect the page.
Jmeter could be used for testing user-browser GET and POST communications. In Jmeter you can request an URL and post some parameters like the browser do and then inspect the page response.
PROS and CONS:
Selenium is good if you want to test javascript page functionalities.
Selenium is good if you want to have your test cases written in Java, Javascript, Python or simple html text files. Selenium can format your test cases in many programming languages. JMeter always uses an XML format for store test cases.
JMeter is good if you don't want to deal with browser versions. JMeter works in all browsers. Selenium has a wide list of supported browsers, but will always have browser requirements.
JMeter is good if you want to also record HTTP, SOAP and RESTFul protocols. JMeter can be used for record and test communications between servers. JMeter doesn't need a browser to run, Selenium does.
JMeter can run SQL queries, bash scripts, Java classes, ... from JMeter test. By the other hand, Selenium tests can be embedded in Java, Python, Javascript, ... programs.
Both supports xpath, html inspection, css inspection, ...
as mentioned in above replies, Selenium is a tool for testing functionality. Usually its described as a tool for automated testing, while on other hand JMeter is a tool used for performance testing.
I would suggest to start off with Selenium, since the most important aspect of any web application is that it's working correctly. Try to create the basic suite of couple of tests, with the most important automated tests that verify some functionality. Once you have the base automated testing knowledge at least, I would move on to JMeter and performance testing.
In my personal experience, performance testing requires much more knowledge about the system being tested than automated testing. Both JMeter and Selenium should not be complex to learn, but for performance testing you need to more about web application tested.
I can see that since selenium 2.0, WebDriver and Selenium RC are packaged together for download. Now I primarily use WebDriver, but can I bring in Selenium RC in my testing scripts from now and then? Is there anything that Selenium RC is capable of but WebDriver is not, or vice versa?
You should probably start your research here (though you may have already gone over this): http://seleniumhq.org/docs/03_webdriver.html
I'll assume you're contrasting Selenium-RC to WebDriver, Selenium-IDE really isn't in the same ballpark.
Selenium uses JavaScript to automate web pages. This lets it interact very tightly with web content, and was one of the first automation tools to support Ajax and other heavily dynamic pages. However, this also means Selenium runs inside the JavaScript sandbox. This means you need to run the Selenium-RC server to get around the same-origin policy, which can sometimes cause issues with browser setup.
WebDriver on the other hand uses native automation from each language. While this means it takes longer to support new browsers/languages, it does offer a much closer 'feel' to the browser. If you're happy with WebDriver, stick with it, it's the future. There are limitations and bugs right now, but if they're not stopping you, go for it.
Selenium Benefits over WebDriver
Supports many browsers and many languages, WebDriver needs native implementations for each new language/browser combo.
Very mature and complete API
Currently (Sept 2010) supports JavaScript alerts and confirms better
Benefits of WebDriver Compared to Selenium
Native automation faster and a little less prone to error and browser configuration
Does not require Selenium-RC Server to be running
Access to headless HTMLUnit can allow tests to run very fast
Great API
I see this is an old question but found this is on the Selenium HQ home page:
Selenium WebDriver is the successor of Selenium Remote Control which
has been officially deprecated. The Selenium Server (used by both
WebDriver and Remote Control) now also includes built-in grid
capabilities.
So it's settled :-)
The biggest difference is RC runs from a vs, 2.0 uses Webdriver and launches the browser, instead of using a vs. In order to you RC in 2.0, check here: http://seleniumhq.org/docs/09_webdriver.html#emulating-selenium-rc
I dont know how to take 2.0 into RC though, but were do you see they are packaged together? They are two different products. Selenium 2 is webdriver, and Selenium RC is Selenium 1.
Personally, I found 2.0 a lot easier to program with. Plus by the end of the year Javascript alert support should be implemented, which is a huge plus!
In my current project we are testing our ASP.NET GUI using WatiN and Mbunit.
When I was writing the tests I realized that it would be great if we also could use all of these for stresstesting. Currently we are using Grinder to stresstest but then we have to script our cases all over again which for many reasons isent that good.
I have been trying to find a tool that can use my existing tests to create load on the site and record stats, but so far i have found noting. Is there such a tool or is there an easy way to create one?
We have issues on our build server when running WatiN tests as it often throws timeouts trying to access the Internet Explorer COM component. It seems to hang randomly while waiting for the total page to load.
Given this, I would not recommend it for stress testing as the results will be inaccurate and the tests are likely to be slow.
I would recommend JMeter for making threaded calls to the HTTP requests that your GUI is making
For load testing there is a tool which looks promising - LoadStorm. Free for 25 users. It has zero deployment needs as this is a cloud based service.
You could build a load controller for your stress testing. It could take your watin tests and run them in a multithreaded/multiprocessed way.
If you are comfortable using Selenium instead of WatiN, check out BrowserMob for browser-based load testing. I'm one of the Selenium RC authors and started BrowserMob to provide a new way to load test. By using real browsers, rather than simulated traffic, tests end up being much easier to script and maintain.