In my current project we are testing our ASP.NET GUI using WatiN and Mbunit.
When I was writing the tests I realized that it would be great if we also could use all of these for stresstesting. Currently we are using Grinder to stresstest but then we have to script our cases all over again which for many reasons isent that good.
I have been trying to find a tool that can use my existing tests to create load on the site and record stats, but so far i have found noting. Is there such a tool or is there an easy way to create one?
We have issues on our build server when running WatiN tests as it often throws timeouts trying to access the Internet Explorer COM component. It seems to hang randomly while waiting for the total page to load.
Given this, I would not recommend it for stress testing as the results will be inaccurate and the tests are likely to be slow.
I would recommend JMeter for making threaded calls to the HTTP requests that your GUI is making
For load testing there is a tool which looks promising - LoadStorm. Free for 25 users. It has zero deployment needs as this is a cloud based service.
You could build a load controller for your stress testing. It could take your watin tests and run them in a multithreaded/multiprocessed way.
If you are comfortable using Selenium instead of WatiN, check out BrowserMob for browser-based load testing. I'm one of the Selenium RC authors and started BrowserMob to provide a new way to load test. By using real browsers, rather than simulated traffic, tests end up being much easier to script and maintain.
Related
Currently I'm trying to run a load test which walks through a uniquely created URL. I know JMeter is often used for load testing, but I was specifically asked to do it through something like Selenium that uses real browsers to create the URL then open that URL and complete the steps within the URL. I have created a Selenium script that can easily do this, but I need to do this 100 times concurrently and can't find a good way to do.
Is there a way to do this? I've looked into Selenium Grid but I'm not sure if I even have enough nodes to run 100 browsers concurrently. Please if you have recommendations for software or a different method of doing this I would love to hear it. Thank you!
JMeter can be integrated with Selenium using WebDriver Sampler so you can re-use your code and rely on JMeter's multithreading capabilities.
If one machine won't be powerful enough to kick off 100 browsers - you can consider going for Distributed Testing
In general be aware that browsers don't do any magic, they just send HTTP requests, wait for responses and render them. JMeter is not capable only of rendering the page, but if you need to load test the backend - it can mimic browser's network footprint with 100% accuracy, just make sure to configure JMeter accordingly in order to behave like a real browser
JavaScript execution time and page rendering speed can be checked either using single WebDriver Sampler or a separate solution like Lighthouse
I am getting ready to start a new automation project and have done some reading on Cypress as a Selenium alternative. Given that Cypress apparently runs directly in the browser as opposed to Selenium's approach, is it difficult to perform test steps with Cypress tests that fall outside the browser such as communicating with a data store, interacting with services and interacting with product infrastructure such as remote file systems? With my limited exposure to Cypress, I have only seen browser tests so I was hoping someone could shed some light on this.
When it comes to automated testing for web applications, there are two main contenders: Selenium and Cypress. Both have their pros and cons, but which one is the best?
Selenium has been around for much longer than Cypress and is therefore more widely used. It is also more flexible, allowing for tests to be written in a variety of programming languages. However, Selenium can be slow and unreliable, and it is not as easy to use as Cypress.
Cypress, on the other hand, is a newer tool that is gaining popularity due to its simplicity and reliability. Cypress tests are written in JavaScript, making it easier for front-end developers to get started with automated testing. Cypress is also faster than Selenium and can run tests in parallel, making it more efficient.
So, which one should you use? It depends on your needs. If you need a more flexible tool that can be used with different programming languages, Selenium is a good choice. However, if you want a tool that is easier to use and more reliable, Cypress is the better option.
If you need access to things outside the browser, I would go with selenium. This is what I currently do, I have a webdriver wrapper which has "plugins" loaded so that I can make db statements, query the webserver and additionally issue selenium commands to the browser.
If you're looking for just test 100% within the browser, then cypress may be the way to go.
Alternatively, you could use selenium for workflow tests and cypress or even qunitjs for intra-browser unit tests.
In the app I work on, I actually ship a page which contains a qunit page with all of the in-browser tests. Then in a selenium test, in addition to the rest of the workflow, I browser to the qunit page and report on their status as well.
I am developing WebUI automation tests using Karate 0.9.5.RC5 and it is working wonderfully. Does anyone know how to execute performance testing in Karate for WebUI Automation tests?
That's great to hear and thanks for the feedback. To be honest, we have focused on API perf testing and UI functional test automation so far. Maybe you can help us by experimenting and sharing what you find.
You must be aware of the Gatling integration for API performance testing. So we have some pieces of the puzzle in place.
So maybe a hybrid strategy is best:
identify the API calls being made by the UI, in the future we would like to derive them automatically from the Chrome network / devtools
manually convert the API calls to Karate tests, note that the VS Code plugin has an option to import from cURL
now you can convert the Karate tests to a performance test, and for most teams, this is sufficient
if needed, you can add some Karate calls to load HTML and static resources to make the load profile more realistic
finally, it may be possible to run a Karate UI test in parallel, just to measure "real user" experience and measure the HTML / JS load times etc on the browser side. we don't have this in place yet, but it should be possible to get all the page timings and perf-stats from Chrome
potentially you could look at spinning up multiple Chrome instances in parallel using Docker - but again, this is something yet to be explored
I am developing WebUI automation tests using Karate 0.9.5.RC5 and it is working wonderfully. Does anyone know how to execute performance testing in Karate for WebUI Automation tests?
That's great to hear and thanks for the feedback. To be honest, we have focused on API perf testing and UI functional test automation so far. Maybe you can help us by experimenting and sharing what you find.
You must be aware of the Gatling integration for API performance testing. So we have some pieces of the puzzle in place.
So maybe a hybrid strategy is best:
identify the API calls being made by the UI, in the future we would like to derive them automatically from the Chrome network / devtools
manually convert the API calls to Karate tests, note that the VS Code plugin has an option to import from cURL
now you can convert the Karate tests to a performance test, and for most teams, this is sufficient
if needed, you can add some Karate calls to load HTML and static resources to make the load profile more realistic
finally, it may be possible to run a Karate UI test in parallel, just to measure "real user" experience and measure the HTML / JS load times etc on the browser side. we don't have this in place yet, but it should be possible to get all the page timings and perf-stats from Chrome
potentially you could look at spinning up multiple Chrome instances in parallel using Docker - but again, this is something yet to be explored
Hi, I want to get into test automation, and the two tools I found during my first web search are Selenium and Jmeter.
Which one do you think is the first to have a look at? Or do I need both tools as they're totally different?
What I would need is the possibility to do Clientside-Certificate-Authentication, filling forms with different Information, and checking result pages.
Apache JMeter is definitely tool for performance testing and load/stress tests. But you can use it also for functional tests as well (in your example: fill form ->check results but with checking if results are as expected - but better don't do functional testing with him)
For functional testing on the other hand there are Selenium and also Canoo web test.
So final answer will be to combine those two. (I was using JMeter for performance tests and canoo web test for functional testing, but I guess that Selenium is much better choice now)
Use Selenium for your functional tests
Use JMeter for stress tests, and measure performance
In both cases you can record a session, so you can start your Selenium or JMeter engine, do something in your browser and then stop recording. After that you can use Selenium or JMeter to run the session recorded again.
Selenium tests browser fields and buttons. In Selenium you can fill a input field and click a button, wait for the page load and then inspect the page.
Jmeter could be used for testing user-browser GET and POST communications. In Jmeter you can request an URL and post some parameters like the browser do and then inspect the page response.
PROS and CONS:
Selenium is good if you want to test javascript page functionalities.
Selenium is good if you want to have your test cases written in Java, Javascript, Python or simple html text files. Selenium can format your test cases in many programming languages. JMeter always uses an XML format for store test cases.
JMeter is good if you don't want to deal with browser versions. JMeter works in all browsers. Selenium has a wide list of supported browsers, but will always have browser requirements.
JMeter is good if you want to also record HTTP, SOAP and RESTFul protocols. JMeter can be used for record and test communications between servers. JMeter doesn't need a browser to run, Selenium does.
JMeter can run SQL queries, bash scripts, Java classes, ... from JMeter test. By the other hand, Selenium tests can be embedded in Java, Python, Javascript, ... programs.
Both supports xpath, html inspection, css inspection, ...
as mentioned in above replies, Selenium is a tool for testing functionality. Usually its described as a tool for automated testing, while on other hand JMeter is a tool used for performance testing.
I would suggest to start off with Selenium, since the most important aspect of any web application is that it's working correctly. Try to create the basic suite of couple of tests, with the most important automated tests that verify some functionality. Once you have the base automated testing knowledge at least, I would move on to JMeter and performance testing.
In my personal experience, performance testing requires much more knowledge about the system being tested than automated testing. Both JMeter and Selenium should not be complex to learn, but for performance testing you need to more about web application tested.