I am new to test and during my intership, I had to look for some good tools to automate functional tests.
So I made a lot of searches and decided to use Cucumber, linked with Selenium and SoapUI.
But the fact is that another search had been made before to automate load tests and Gatling (used with Jenkins) had been chosen for that.
Then I am asking to find a solution to gather the 2 solutions but it is quite hard to find any solution about that.
The only information I catch was using Taurus to have a single configuration file.
If you have any information which you think coul help me, I would be very thankful.
The purpose and techniques used by cucumber and gatling are very different so you will probably not want to push interoperation too far.
For example, you plan to use Selenium in your cucumber steps. That would not work well with the performance testing.
If you use jvm version of cucumber, you'll be able to reuse some of the utilities, for example test builders.
Related
I have python scripts which are based on Robot Framework and were generally developed for Functional Testing. How can I achieve the Load Testing?
How to approach this problem? should I use Jmeter in order to run load testing (if yes then how is it possible) or there are some other methods available?
If you're a programmer and your team has a good culture of software and code review.
Instead of Jmeter, you have another choice Taurus
Taurus can consume configuration files written in JSON or YAML. But it also supports Jmeter.
It will be easier for code review and construct your own load test.
Some material you can read:
YAMLTutorial:
https://gettaurus.org/docs/YAMLTutorial/
Automating Performance Tests with Taurus:
https://www.youtube.com/watch?v=DN0VNnG-lzs
Simple load test example with Taurus:
https://medium.com/#s6307/how-to-design-a-good-load-test-with-jmeter-and-taurus-like-a-pro-part-1-1d1b32fd77be
You might want to consider using rfswarm, as it's designed exactly for this purpose:
https://github.com/damies13/rfswarm/blob/master/README.md
In the documentation there is even some details about things to consider when adjusting your robot files for use in performance testing.
Good luck with your project and i hope this answer is not too late for you.
Dave.
When creating automated tests with selenium, I thought one would use easier cucumber with selenium or testng with selenium or just junit with selenium although using only junit is not very popular. I have recently found out that you could use cucumber with testng but I don't see what is the gain of doing this. If someone is using both of them together can you tell me why ?
EDIT:
Using Testng over junit has many advantages. My question is if i use cucumber does it still make a difference or not anymore.
P.S I am not trying to start this tool vs this tool war
The answer that you seem to be looking for, is one of interest in what Cucumber, as a tool, adds to existing test frameworks.
The answer:
Cucumber adds an extra level of communication between you (the development team) and the management team. You are able to link test cases to scenarios that are now understandable by the business, which means that everybody is on the same page. You can even use the BDD tool to start talking about behaviours of the feature:
What things should be included?
Do we need more information?
Lets add that to the file, so that we can test that use case later.
Any new functionality added to the feature later?
Need to understand which section has gone wrong quickly, without having to decipher code written by the intern that was in for 2 months in the summer?
Cucumber helps with all of this, and that's just scraping the surface.
TestNG, JUnit, Selenium? You imagine it, you can do it. With Cucumber as your helpful neighbourhood BDD tool, you can pull together your test suite and bolt an abstraction layer on top. The business will now be able to look at the test results. Where tests have failed, they will be able to describe exactly what section has gone wrong to other members of management, without having to go too far into technical details.
If you're wondering whether to use JUnit or TestNG for this, it is most likely to be your choice. Using whatever is the current test tool to bolt cucumber on top of is the best option if you have an existing suite.
Also, make sure you are using the right language for your team. For instance:
Are you introducing a team of manual testers to developing test automation?
Maybe you should use Ruby or JavaScript, as they are easier languages to pick up as a first language
Are you a development team, using cucumber to add an abstraction layer to your unit tests?
Use the language that you are using for development, with the unit test tool that you are using.
Are you developers in test, using cucumber for automating tests for your website?
Use the language that you and your team are most comfortable with, taking the language being used for development over any others that tie with this (based on a team vote).
I think it depends on what are your other tests (unit ones for example) and how you run them.
If your current tests are already using TestNG, then it will be easier to run Cucumber tests with TestNG engine.
At the opposite, if you already have JUnit tests, it could be easier to use JUnit for Cucumber run (but TestNG is able to run JUnit tests, so you can use TestNG in that case too).
And if you have no other tests, so the choice of the test runner will depend on your own taste.
Yes.. I understand your question. Even I had the same doubt as below:
We use selenium for automation testing. Since they don't provide proper reports, we add TestNG to it (and also for other features). But now, we have cucumber, which gives proper reports. So why do we need TestNG?
I realized, though we get proper results with cucumber, TestNG provides us with many other features which cucumber cannot; like setting priority, setting method dependency, timeouts, grouping , etc.
Though cucumber provides a tag feature, it does not provide all the features provided by TestNG. Maybe when cucumber incorporates all those features, we can eliminate TestNG.
is there any tool out there that i can used to set-up run automatically and i was goggling and i found selenium test runner? there are so many tools out there its hard to figured out which is best
I'm using C# and using MSTest as a test framework and I'm looking forward to see if I can get a way from testing in MSTEST
any help?
This is very subjective question. Every requirement will have its own correct answer. Anyhow I will try to address few requirements and will be updating as I learn more.
If you are automating web app browser tests (sans flash player and silverlight) I would say that selenium is the way to go. There are ways to automate flash and silverlight too, but that is answer for another question.
Selenium is anyways an automation too and your choice will rather is of which test framework to select. So here are few options:
1. Integrating with CI tools:
If you want to organize your tests as segregated atomic units and want them to be integrated to some CI server (e.g. TeamCity). I will recommend using NUnit to run your selenium tests.
2. Behavioral Tests
It is a new trend in the software development and how we test our products. Using behavioral (i.e. business specification) like language. In my experience it is also a very good format to write up acceptance tests. You can use selenium with something like Nbehave or SpecFlow
3. Centralize Test management and Execution
Now this might not fit for everyone but I have found FitNesse (and its c# binding) to be very useful in maintaining and executing selenium test cases.
Please note this answer may not be right and is certainly not complete given the scope of the question. I have nevertheless tried provide few pointers.
I have been writing tests for my Ruby code for a while, but as a frontend developer I am obviously interested in bring this into the code I write for my frontend code. There is quite a few different options which I have been playing around with:
CasperJS
Capybara & Rspec
Jasmine
Cucumber or just Rspec
What are people using for testing? And further than that what do people test? Just JavaScript? Links? Forms? Hardcoded content?
Any thoughts would be greatly appreciated.
I had the same questions a few months ago and, after talking to many developers and doing a lot of research, this is what I found out. You should unit test your JavaScript, write a small set of UI integration tests and avoid record and playback testing tools. Let me explain that in more detail.
First, consider the test pyramid. This is a interesting analogy created by Mike Cohn that will help you decide which kind of testing you should be doing. At the bottom of the pyramid are the unit tests, which are solid and provide fast feedback. These should be the foundation of your test strategy and thus occupy the largest part of the pyramid. At the top, you have the UI tests. Those are the tests that interact with your UI directly, like Selenium does for example. Although these tests might help you find bugs, they are more expensive and provide very slow feedback. Also, depending on the tool you use, they become very brittle and you will end up spending more time maintaining these tests than writing actual production code. The service layer, in the middle, includes integration tests that do not require an UI. In Rails, for instance, you would test your REST interface directly instead of interacting with the DOM elements.
Now, back to your question. I found out that I could greatly reduce the number of bugs in my project, which is a web application written in Spring Roo (Java) with tons of JavaScript, simply by writing enough unit tests for JS. In my application, there is a lot of logic written in JS and that is the kind of thing that I am testing here. I am not concerned about how the page will actually look or if the animations plays as they should. I test if the modules I write in JS will execute the expected logic, if element classes are correctly assigned and if error conditions are well handled. For these tests, I've been using Jasmine. This is a great tool. It is very easy to learn and has nice mocking capabilities, which are called spies. Jasmine-jQuery adds more great functionality if you are using jQuery. In particular, it allows you to specify fixtures, which are snippets of the HTML code, so you don't have to manually mock the DOM. I have integrated this tool with maven and these tests are part of my CI strategy.
You have to be careful with UI tests, specially if you rely on record/playback tools like Selenium. Since the UI changes often, these tests keep breaking and you will spend a lot of time finding out if the tests really failed or if they are just outdated. Also, they don't add as much value as unit tests. Since they need an integrated environment to run, you will mostly like run them only after you finished developing, when the cost of fixing things is higher.
For smoke/regression tests, however, UI tests are very useful. If you need to automate these, then you should watch out for some dangers. Write your tests, don't record them. Recorded tests usually rely on automatically generated xpaths that break for every little change you do on your code. I believe Cucumber is a good framework for writing these tests and you can use it along with WebDriver to automate the browser interaction. Code thinking about tests. In UI tests, you will have to make elements easier to find so you don't have to rely on complex xpaths. Adding class and id elements where you usually wouldn't will be frequent. Don't write tests for every small corner case. These tests are expensive to write and take too long to run. You should focus on the cases that explore most of your functionality. If you write too many tests at this level you will probably test the same functionality that you have previously tested on your unit tests (supposing you have written them).
In my current project I am using Spock and Geb to write the UI tests. I find these tools amazing. They are written in Groovy, which suits better my Java project.
There are lots of options and tools for that. But their choice depends on whether you have a web UI or it's a desktop app?
Supposing from the tools you've mentioned it's Web UI. I would suggest Selenium (aka WebDriver): http://seleniumhq.org/docs/
There is a variety of languages it supports (Ruby is in the list). It can be run against a variety of browsers, ad it's quite easy to use with lots of tutorials and tips available.
Oh, and it's free, of course :)
I though as this post gets a lot of likes, I would post my answer to my question as I do write lots of tests now and how you test front end has moved on a lot now.
So in terms of FE testing I spent lot of time using karma with Jasmine, although karma will work nicely with other test suites like mocha & qunit. While these are great and karma allows you to interface directly with browsers to run your tests. The downside is as your test suite gets large it can become quite slow.
So recently I have moved to Jest which is much faster and if your writing react app, using enzyme with snap shot testing give you really good coverage. Talking of coverage Jest has Istanbul coverage built in and set up and mocking is really easy simple to use. The downside it doesn't test in browser and it using something called jsdom which is fast, but does have a few nuisances. Personally I don't find this a big deal particularly when I compile my code through webpack/babel which means the cross browser bugs are fairly few and far between, so generally isn't an issue if you manually test anyway (and imo you should).
In terms of working within the rails stack, this much easy now that the webpacker gem is now available and using npm and node is generally much more excepted. I would recommend using nvm to manage your node versions
While this isn't strictly testing, I would also recommend using linting as this also picks up a lot of issues in your code. For JS I use eslint with prettier and scss/css I use stylelint
In terms on what to test, I think as Carlos talks about the test pyramid is still relevant, after all the theory doesn't change, just the tools. I would also add to be practical about tests, I would always test, but to what level and coverage will depend on the project. It is important to manage your time and spending hours/days testing a short lifecycle project. Larger/longer term projects the benefits of a larger test suite is obviously greater.
Anyway I hope that helps people that look at the question.
Is it better to write/record selenium tests in html format and run them directly in the server with "-htmlSuite" or to write the tests in java/C#/... and run them in the server using selenium-rc?
What is the recommended solution?
I would always recommend people writing their Selenium Tests in a programming language because it allows the tests to be a lot more expressive.
You can create common methods that all tests use and if that changes you can then update 1 method to get n tests passing because they all fail on that item. One example of this is the Page Object model which is a development method for tests that suggest you create a DSL for each page you interact with and then your tests read a lot more fluently to both technical and non-technical people.
If you write your tests in a programming language you can also take advantage of Selenium Grid which runs your tests in parallel to make them faster
I'd highly recommend you look at moving to Selenium in C#, Ruby,PHP or Python. I found many timing issues resolving once I got away from the HTML Selenese.
I might keep in mind how you plan on executing your recorded tests. For instance, it might be trivial to incorporate Selenium's generated JUnit tests into a pre-existing JUnit-based testing framework.