We can select multiple scenario's by including the following on the command line:
-Dcucumber.options="--tags #S1,#S2,#S6"
And if I want to exclude #S6 I can with:
-Dcucumber.options="--tags ~#S6"
But if I want to include #S1, #S2 and exclude #S6 all tags are ignored with:
-Dcucumber.options="--tags #S1,#S2,~#S6"
and all tags are also ignored if I try to double up on the options with:
-Dcucumber.options="--tags #S1,#S2" -Dcucumber.options="--tags ~#S6"
Is there a command line way to include and exclude in the one command line?
The reason I would like to do this is to run all of a type of test but exclude tests that use some external system that may be down temporarily.
EDIT: Turns out I was not fully aware of the difference between AND and OR in the Cucumber world. This SO answer and this article was a good reference, look for "Logical OR" and "Logical AND":
so to run S1 OR S2 AND NOT S3
mvn test -Dkarate.options="--tags #S1,#S2 --tags ~#S3"
Using the Java (or Runner) API, just use commas for OR and separate strings for AND:
Results results = Runner.path("classpath:com/myco/some.feature")
.tags("#S1,#S2", "~#S3")
.parallel(5);
For really advanced cases, also see: https://stackoverflow.com/a/67224240/143475
Related
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
Consider there are 100 scenarios and I want to run 99 scenarios in prod environment and 100 on stage environment .
Is there a way to achieve this in karate ?
Things I tried
1. Create a feature file with 1 scenario and another feature file with remaining 99
2. Add tag to the 1 scenario file
3. Ignore that while running
But then when I use it in jenkins job I have to run one command to run on both machines so would not work
The best solution for this case is the karate.abort() API:
So in the "special" scenario #100 - you can do this:
Scenario:
* eval if (karate.env == 'prod') karate.abort()
# normal scenario steps
Please note that there are advanced options for tag selectors in Karate 0.9.0 onwards - but just stick to the solution above please :)
Tag the 100th scenario with #hundred and just run the following command when you want to run 99 scenarios
mvn test -Dkarate.options="--tags ~#hundred"
And simply run mvn test when you want to run all tests.
You can tag a scenario
#hundred
Scenario: the scenario only played in one case
Given <...>
But you can also tag a Feature
#hundred
Feature: The feature containing the scenario only played in one case
Background:
* <..>
Scenario: <...>
Edit after your answer :
You could use a second runtime variable :
mvn test -Dkarate.options="--tags ~#{variable2}" -Dkarate.env={variable}
Or even use the same runtime variable :
mvn test -Dkarate.options="--tags ~#{variable}" -Dkarate.env={variable}
And that maybe wouldn't be the best solution, but you can add #Prod to the 99 scenarios, and #Stage to all of them, and switch the command to this :
mvn test -Dkarate.options="--tags #{variable}" -Dkarate.env={variable}
It's a bit longer to do, but at least the tag(s) on each feature/scenario will correspond to the case(s) in which they are launched
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
It is possible to execute one or several groups of tests in Codeception. There are multiple ways to assign a test to a group (tag with #group annotations, add a group configuration in a yaml or add a group txt file).
Is there a way to list all tests which are assigned to a specific group?
I tried codecept dry-run, but this does not accept the group parameter, and codecept run executes all tests (which I dont want).
There is no built-in command in Codeception for that.
If you only specify groups via #group tags you can use grep to find the test files for a group. If you use other ways to define groups this method will not work.
For example to search for all tests files with the group shop:
grep -ro '#group shop' tests/
You may need to replace tests/ with the path where your tests are located.
Similar command to list all groups:
grep -rho '#group.*' tests/ |sort |uniq
I'm asking for help on how to run a feature file scenario just by name. I've been trying for a while and it does not come out. I know that can be done by tags or by line number, but I wonder if we can run a cucumber test by name, more or less with this nomenclature.
Given a file named "features/test.feature" with:
Feature:
Scenario: My first scenario
Given this step is blah blah blah
Scenario: My second scenario
Given this step too blah blah
I want to run a scenario by name from the console or with gradle, maybe similar this way
cucumber features/test.feuture::My second scenario
Or maybe with gradle
./gradlew cucumber::My second scenario
You didn't describe how you start cucumber so I can't help you with that.
When used from the CLI accepts --name REGEXP. This will only run scenarios whose names match REGEXP.
The #CucumberOptions annotation accepts name="REGEXP".
Cucumber < v6.0.0 looks at the environment. For maven you can add -Dcucumber.options=--name REGEXP. I don't know the equivalent for gradle. Take note that the escape characters maybe shell/build system dependent.
Cucumber v6.0.0 and above looks at the environment. For maven you can add -Dcucumber.filter.name="REGEXP".
See:
https://cucumber.io/docs/reference/jvm#running
https://github.com/cucumber/cucumber-jvm/tree/main/core
From cucumber 6.x, you can run a scenario with below CLI commands:
// Specify a scenario by its line number
$ cucumber-js features/my_feature.feature:3
// Specify a scenario by its name matching a regular expression
$ cucumber-js --name "topic 1"
But, these are time-consuming and repetitive. You can save a lot of time by using a dedicated VSCode Extension called Cucumber-Quick. This extension will allow you to run a scenario/feature just by right-clicking on them. It can save you from all the hustle.
You would call the scenario by its line number.
So assuming that the second scenario starts on line 5 in your feature file you could run:
cucumber features/test.feature:5