Below is the structure how my Feature Files are divide. I have created Folders based on the functionalities and then added the scenarios inside them.
Now, I have to tag few test cases among them as Smoke Test cases and get them executed.
The point here is I need a specific order for that as in eg
Add Asset
Run Test
Schedule Test
Delete Asset
Since I will add something first and then work on it and delete it at the end
I know by default Cucumber executes test cases alphabetically but that would not solve my problem.
How can I achieve that?
I am using Java
Cucumber features/scenarios are run in Alphabetical order by feature file name.
However, if you specifically specify features, they should be run in the order as declared. For example:
#Cucumber.Options(features={"automatedTestingServices.feature", "smoketest.feature"})
You can achieve by setting priority or dependency, supported in QAF which is TestNg implementation for BDD. Setting priority with scenarion should do the needful for example:
with QAF scenario in DeleteAssets.feature may look like below:
#priority:100
#or you can set dependencies like below
##dependsOnGroups:['create','schedule']
#delete #otherGroup
Scenario: Delete existing Asset
Given ...
Note: gherkin syntax doesn't supports meta-data so you need to use either qaf bdd or bdd2 syntax and appropriate factory to run tests.
Yes, you can set a priority in cucumber scenarios. but not for the whole scenarios we can do that. inside methods we have declared in step definition file, can achieve that. Just put a keyword "Order" in the step definition file over the method based on the order of the method it will run as priority.
Click here for reference
Related
I am working on the automation of test cases with the Cucumber JVM 1.2.2 framework and Selenium. Each test case corresponds to a Feature file.
I have multiple Features files organized in folders, and I need to be able to define the order of execution through a text file. For example, the text file can be like this:
file3.feauture
file1.feature
file5.feature
file2.feature
file4.feature
At this time the execution is called through a tag that is placed in the files that will be executed.
First of all, the version you are using is very old. We are currently on 4.7.4 (see: https://github.com/cucumber/cucumber-jvm).
Note that the group-id changed from "info.cukes" to "io.cucumber" with v2.
Second of all, it is recommended to have your tests be independent of each other.
From the Cucumber FAQ:
"Each scenario should be independent; you should be able to run them in any order or in parallel without one scenario interfering with another.
Each scenario should test exactly one thing so that when it fails, it fails for a clear reason. This means you wouldn’t reuse one scenario inside another scenario.
If your scenarios use the same or similar steps, or perform similar actions on your system, you can extract helper methods to do those things."
Cucumber Ran the feature file in Alphabetical order.
Rename your feature file like :
Cfile3.feauture
Afile1.feature
Efile5.feature
Bfile2.feature
Dfile4.feature
I am following POM approach with Testng for designing my framework . Consider a scenario wherein the test case got failed in the nth test step inside a #Test method.Can anyone suggest how can I skip the remaining test steps(from n+1 onwards)?
Since I am automating the manual test cases, each #Test belongs to 1 test case and so I cannot split up the steps into multiple #Test methods. When the test step is failed, it needs to skip the next steps in that #Test method and proceed to the next test case.
Also I need the count of test steps skipped in the result.
Kindly help.
Looks like you are basically looking for something around what a BDD tool such as Cucumber provides you. Cucumber lets you create a .feature file which contains one or more scenario (You can visualize each of your scenario to be one #Test annotated test method).
You could then leverage one of the Cucumber integrations i.e.,
Choose either JUnit (or)
Choose TestNG integration
and let one of these run your BDD tests.
Here when a particular step fails, then all subsequent steps would be aborted (which is what you are asking for)
Outside of Cucumber, I dont think you can have this done via any other mechanism. The reporting needs (such as how many steps were skipped etc.,) can be fulfilled by any cucumber based reports.
You should start from here : http://docs.cucumber.io/guides/10-minute-tutorial/
NUnit (and the like) has method attributes which allow tests to be run multiple times with different arrange values. Is something similar possible with SpecFlow?
What I am aiming for is a way to run the same scenario tests in a feature file with as many browser drivers as I can, in one test run.
You can use scenario outlines. In example of scenario outline you can mention driver name and you code logic should take action according to driver. Please see more details about scenario ouyline below
https://github.com/cucumber/cucumber/wiki/Scenario-outlines
Examples are one solution, but in your case a little cumbersome, as you have to specify them at every scenario.
In your case, please have a look at the targets feature of the SpecFlow+Runner. With that you can "multiply" your scenarios for different configurations. If you put the web driver that should be used in this configuration, you can test as many webdriver as you want.
Have a look at this example: https://github.com/techtalk/SpecFlow.Plus.Examples/tree/master/SeleniumWebTest
Full Disclosure: I am one of the developers of SpecFlow & SpecFlow+
Use scenario outlines and this tool if you want to use browsers as tags:
https://github.com/unickq/Unickq.SeleniumHelper
We use TestNG and Selenium WebDriver to test our web application.
Now our problem is that we often have several tests that need to run in a certain order, e.g.:
login to application
enter some data
edit the data
check that it's displayed correctly
Now obviously these tests need to run in that precise order.
At the same time, we have many other tests which are totally independent from the list of tests above.
So we'd like to be able to somehow put tests into "groups" (not necessarily groups in the TestNG sense), and then run them such that:
tests inside one "group" always run together and in the same order
but different test "groups" as a whole can run in any order
The second point is important, because we want to avoid dependencies between tests in different groups (so different test "groups" can be used and developed independently).
Is there a way to achieve this using TestNG?
Solutions we tried
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We considered writing a method interceptor. However, this has the disadvantage that running tests from inside an IDE becomes more difficult (because directly invoking a test on a class would not use the interceptor). Also, tests using dependsOnMethods cannot be ordered by the interceptor, so we'd have to stop using that. We'd probably have to create our own annotation to specify ordering, and we'd like to use standard TestNG features as far as possible.
The TestNG docs propose using preserve-order to order tests. That looks promising, but only works if you list every test method separately, which seems redundant and hard to maintain.
Is there a better way to achieve this?
I am also open for any other suggestions on how to handle tests that build on each other, without having to impose a total order on all tests.
PS
alanning's answer points out that we could simply keep all tests independent by doing the necessary setup inside each test. That is in principle a good idea (and some tests do this), however sometimes we need to test a complete workflow, with each step depending on all previous steps (as in my example). To do that with "independent" tests would mean running the same multi-step setup over and over, and that would make our already slow tests even slower. Instead of three tests doing:
Test 1: login to application
Test 2: enter some data
Test 3: edit the data
we would get
Test 1: login to application
Test 2: login to application, enter some data
Test 3: login to application, enter some data, edit the data
etc.
In addition to needlessly increasing testing time, this also feels unnatural - it should be possible to model a workflow as a series of tests.
If there's no other way, this is probably how we'll do it, but we are looking for a better solution, without repeating the same setup calls.
You are mixing "functionality" and "test". Separating them will solve your problem.
For example, create a helper class/method that executes the steps to log in, then call that class/method in your Login test and all other tests that require the user to be logged in.
Your other tests do not actually need to rely on your Login "Test", just the login class/method.
If later back-end modifications introduce a bug in the login process, all of the tests which rely on the Login helper class/method will still fail as expected.
Update:
Turns out this already has a name, the Page Object pattern. Here is a page with Java examples of using this pattern:
http://code.google.com/p/selenium/wiki/PageObjects
Try with depends on group along with depends on method. Add all methods in same class in one group.
For example
#Test(groups={"cls1","other"})
public void cls1test1(){
}
#Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true)
public void cls1test2(){
}
In class 2
#Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true)
public void cls2test1(){
}
#Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true)
public void cls2test2(){
}
There is an easy (whilst hacky) workaround for this if you are comfortable with your first approach:
At first we just put tests that belong together into one class, and used dependsOnMethods to make them run in the right order. This used to work in TestNG V5, but in V6 TestNG will sometimes interleave tests from different classes (while respecting the ordering imposed by dependsOnMethods). There does not seem to be a way to tell TestNG "Always run tests from one class together".
We had a similar problem: we need our tests to be run class-wise because we couldn't guarantee the test classes not interfering with each other.
This is what we did:
Put a
#Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } )
Annotation on an Abstract Test Class or Interface that all your Tests inherit from.
This will put all your methods in the "first group" (group as described in this paragraph, not TestNG-groups). Inside the groups the ordering is class-wise.
Thanks to Cedric Beust, he provided a very quick answer for this.
Edit:
The group dummyGroupToMakeTestNGTreatThisAsDependentClass actually has to exist, but you can just add a dummy test case for that purpose..
How do the names of methods affect execution of test cases in robotium?
Robotium seems to execute methods in a test class in alphabetical order. So if you precede the name of the methods by something like "test1...", test2..." etc, you can determine the proper order. However, the order should not matter for your test results. If it does, then your tests are not independent.
Methods prefixed with the name "test" are considered tests and are executed in alphabetical order. This is actually a functionality of JUnit, which the Android Test Framework is built on top of, and doesn't have anything to do specifically with Robotium.