I've done load tests for users building scripts in LoadRunner or The Grinder, and now I'm trying out JMeter and it all feels incredibly clunky. Where are the scripts? Does everything have to be done through the UI? Is JMeter able to do complex scripting?
JMeter has good user friendly GUI. We create scripts in JMeter using the UI. JMeter saves the script in XML format with .JMX extension. Script creation is NOT very difficult as you say.
Check this site to get an idea.
Complex scripting can be done in JMeter using logic controllers.
Also JMeter
Is Free & Open source
Is Light weight & easy to install
supports any platform
Supports many protocols - HTTP/HTTPS, FTP, SOAP, LDAP, JDBC, JMS, SMTP, POP etc
Supports external plugins
Can be extended with Beanshell scripts, Groovy, Javascript, Java.
I have tried out Jmeter briefly but I think to it's for you to try it out yourself, there are a number of Jmeter tutorials on the Jmeter site itself as well as on youtube which should prove useful.
Below are some links:
http://jmeter.apache.org/usermanual/intro.html
https://www.youtube.com/watch?v=cv7KqxaLZd8
Hope these links help you in getting a better understanding of Jmeter.
What do you call "scripts"?
JMeter is designed so anyone could create tests using UI only. JMeter's Logic Controllers, Pre/Post-Processors, Assertions, etc. are quite enough to build a load test of any complexity.
If you feel yourself too creative and you're limited by JMeter Test Elements you're welcome to extend it using
Code-enabled test elements like BSF, Beanshell, JSR-223 Pre/PostProcessors, Samplers, Timers and Assertions
Developing custom functions
Developing custom implementation of Java Request Sampler
Developing custom implementation of your own Sampler or Function
Finally it is possible to either run existing or create new tests using JMeter API
Related
Does anybody know if there's a way to have test reports for Behat similar to what we could have for Codeception?
I mean, Behat with pretty format output just gives us steps from feature files. Whereas Codeception reports exact request body with headers and also reports responses.
Is there any way to have similar reports for Behat? Maybe some extension or plugin? I did a search but didn't find anything similar.
Behat is a generic TDD framework and does not know anything about web requests by default. You could easily use it for unit testing in your application or for testing some CLI application.
So, if you need extended information on requests during the step failures, you have to:
Provide Behat with those Request/Response objects
Customize Behat behavior in AfterStep, AfterScenario and/or AfterFeature hooks to make use of those Request/Response objects
Find or write custom output formatter to output that info
You can see examples of custom formatters (initially written by yours truly), which are used in Allure reporting tool (worth giving it a try for rich reporting exprience).
I am developing WebUI automation tests using Karate 0.9.5.RC5 and it is working wonderfully. Does anyone know how to execute performance testing in Karate for WebUI Automation tests?
That's great to hear and thanks for the feedback. To be honest, we have focused on API perf testing and UI functional test automation so far. Maybe you can help us by experimenting and sharing what you find.
You must be aware of the Gatling integration for API performance testing. So we have some pieces of the puzzle in place.
So maybe a hybrid strategy is best:
identify the API calls being made by the UI, in the future we would like to derive them automatically from the Chrome network / devtools
manually convert the API calls to Karate tests, note that the VS Code plugin has an option to import from cURL
now you can convert the Karate tests to a performance test, and for most teams, this is sufficient
if needed, you can add some Karate calls to load HTML and static resources to make the load profile more realistic
finally, it may be possible to run a Karate UI test in parallel, just to measure "real user" experience and measure the HTML / JS load times etc on the browser side. we don't have this in place yet, but it should be possible to get all the page timings and perf-stats from Chrome
potentially you could look at spinning up multiple Chrome instances in parallel using Docker - but again, this is something yet to be explored
I am developing WebUI automation tests using Karate 0.9.5.RC5 and it is working wonderfully. Does anyone know how to execute performance testing in Karate for WebUI Automation tests?
That's great to hear and thanks for the feedback. To be honest, we have focused on API perf testing and UI functional test automation so far. Maybe you can help us by experimenting and sharing what you find.
You must be aware of the Gatling integration for API performance testing. So we have some pieces of the puzzle in place.
So maybe a hybrid strategy is best:
identify the API calls being made by the UI, in the future we would like to derive them automatically from the Chrome network / devtools
manually convert the API calls to Karate tests, note that the VS Code plugin has an option to import from cURL
now you can convert the Karate tests to a performance test, and for most teams, this is sufficient
if needed, you can add some Karate calls to load HTML and static resources to make the load profile more realistic
finally, it may be possible to run a Karate UI test in parallel, just to measure "real user" experience and measure the HTML / JS load times etc on the browser side. we don't have this in place yet, but it should be possible to get all the page timings and perf-stats from Chrome
potentially you could look at spinning up multiple Chrome instances in parallel using Docker - but again, this is something yet to be explored
Hi, I want to get into test automation, and the two tools I found during my first web search are Selenium and Jmeter.
Which one do you think is the first to have a look at? Or do I need both tools as they're totally different?
What I would need is the possibility to do Clientside-Certificate-Authentication, filling forms with different Information, and checking result pages.
Apache JMeter is definitely tool for performance testing and load/stress tests. But you can use it also for functional tests as well (in your example: fill form ->check results but with checking if results are as expected - but better don't do functional testing with him)
For functional testing on the other hand there are Selenium and also Canoo web test.
So final answer will be to combine those two. (I was using JMeter for performance tests and canoo web test for functional testing, but I guess that Selenium is much better choice now)
Use Selenium for your functional tests
Use JMeter for stress tests, and measure performance
In both cases you can record a session, so you can start your Selenium or JMeter engine, do something in your browser and then stop recording. After that you can use Selenium or JMeter to run the session recorded again.
Selenium tests browser fields and buttons. In Selenium you can fill a input field and click a button, wait for the page load and then inspect the page.
Jmeter could be used for testing user-browser GET and POST communications. In Jmeter you can request an URL and post some parameters like the browser do and then inspect the page response.
PROS and CONS:
Selenium is good if you want to test javascript page functionalities.
Selenium is good if you want to have your test cases written in Java, Javascript, Python or simple html text files. Selenium can format your test cases in many programming languages. JMeter always uses an XML format for store test cases.
JMeter is good if you don't want to deal with browser versions. JMeter works in all browsers. Selenium has a wide list of supported browsers, but will always have browser requirements.
JMeter is good if you want to also record HTTP, SOAP and RESTFul protocols. JMeter can be used for record and test communications between servers. JMeter doesn't need a browser to run, Selenium does.
JMeter can run SQL queries, bash scripts, Java classes, ... from JMeter test. By the other hand, Selenium tests can be embedded in Java, Python, Javascript, ... programs.
Both supports xpath, html inspection, css inspection, ...
as mentioned in above replies, Selenium is a tool for testing functionality. Usually its described as a tool for automated testing, while on other hand JMeter is a tool used for performance testing.
I would suggest to start off with Selenium, since the most important aspect of any web application is that it's working correctly. Try to create the basic suite of couple of tests, with the most important automated tests that verify some functionality. Once you have the base automated testing knowledge at least, I would move on to JMeter and performance testing.
In my personal experience, performance testing requires much more knowledge about the system being tested than automated testing. Both JMeter and Selenium should not be complex to learn, but for performance testing you need to more about web application tested.
In my current project we are testing our ASP.NET GUI using WatiN and Mbunit.
When I was writing the tests I realized that it would be great if we also could use all of these for stresstesting. Currently we are using Grinder to stresstest but then we have to script our cases all over again which for many reasons isent that good.
I have been trying to find a tool that can use my existing tests to create load on the site and record stats, but so far i have found noting. Is there such a tool or is there an easy way to create one?
We have issues on our build server when running WatiN tests as it often throws timeouts trying to access the Internet Explorer COM component. It seems to hang randomly while waiting for the total page to load.
Given this, I would not recommend it for stress testing as the results will be inaccurate and the tests are likely to be slow.
I would recommend JMeter for making threaded calls to the HTTP requests that your GUI is making
For load testing there is a tool which looks promising - LoadStorm. Free for 25 users. It has zero deployment needs as this is a cloud based service.
You could build a load controller for your stress testing. It could take your watin tests and run them in a multithreaded/multiprocessed way.
If you are comfortable using Selenium instead of WatiN, check out BrowserMob for browser-based load testing. I'm one of the Selenium RC authors and started BrowserMob to provide a new way to load test. By using real browsers, rather than simulated traffic, tests end up being much easier to script and maintain.