I have my tests set up in the following structure using Robotframework:
Directory1:
- SuiteA
- SuiteB
- SuiteC
Directory2:
- SuiteA
- SuiteB
I want to execute all of the test suites from Directory1 first before executing the suites in Directory2 using "pabot". The test suites from Directory2 are dependent on test suites from Directory1 so the execution must be in this order. How can I order the execution this way?
I was reading online about using an argument file to specify the order of execution but I was not able to figure that out. I also read the documentation from Robotframework and named my test suites accordingly with the "number__testsuitename" naming convention but that did not work either.
Any ideas? Thanks in advance!
Basically, you need to modify the .pabotsuitenames file with the tests/suites that you want to run in parallel and also add the #WAIT flag in case that you need to run something sequentially, for example:
datasources:7a60d412cbd2efd728502cddce7ce4cd4b57aa42
commandlineoptions:65f95c924ba97541f47949701c4e3c51192a5b43
suitesfrom:no-suites-from-option
file:984cbf6a94ce4d05453079d2eedf2399a3cae664
--suite Directory 1 Name.Suite A Name
--suite Directory 1 Name.Suite B Name
--suite Directory 1 Name.Suite C Name
#WAIT
--suite Directory 2 Name.Suite A Name
--suite Directory 2 Name.Suite B Name
pabot does provide options to control the level of parallelism and the order of execution. It is described in readme file of the project:
.pabotsuitenames file contains the list of suites that will be executed. File is created during pabot execution if not already there. The file is a cache that pabot uses when re-executing same tests to speed up processing. This file can be partially manually edited. First 4 rows contain information that should not be edited - pabot will edit these when something changes. After this come the suite names.
There are three possibilities to influence the execution:
The order of suites can be changed.
If a directory (or a directory structure) should be executed sequentially, add the directory suite name to a row.
You can add a line with text #WAIT to force executor to wait until all previous suites have been executed.
Pabot Github
pabot is not working very well with a suite structure e.g. "01__B_Test" because it will be turned into “B_Test”. This means that suites will be executed in alphabetical order and not in numbering order. To fix this, all suites would have to be included in .pabotsuitenames to adjust the order again. With hundreds of suites, this is more than cumbersome.
Related
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
Consider there are 100 scenarios and I want to run 99 scenarios in prod environment and 100 on stage environment .
Is there a way to achieve this in karate ?
Things I tried
1. Create a feature file with 1 scenario and another feature file with remaining 99
2. Add tag to the 1 scenario file
3. Ignore that while running
But then when I use it in jenkins job I have to run one command to run on both machines so would not work
The best solution for this case is the karate.abort() API:
So in the "special" scenario #100 - you can do this:
Scenario:
* eval if (karate.env == 'prod') karate.abort()
# normal scenario steps
Please note that there are advanced options for tag selectors in Karate 0.9.0 onwards - but just stick to the solution above please :)
Tag the 100th scenario with #hundred and just run the following command when you want to run 99 scenarios
mvn test -Dkarate.options="--tags ~#hundred"
And simply run mvn test when you want to run all tests.
You can tag a scenario
#hundred
Scenario: the scenario only played in one case
Given <...>
But you can also tag a Feature
#hundred
Feature: The feature containing the scenario only played in one case
Background:
* <..>
Scenario: <...>
Edit after your answer :
You could use a second runtime variable :
mvn test -Dkarate.options="--tags ~#{variable2}" -Dkarate.env={variable}
Or even use the same runtime variable :
mvn test -Dkarate.options="--tags ~#{variable}" -Dkarate.env={variable}
And that maybe wouldn't be the best solution, but you can add #Prod to the 99 scenarios, and #Stage to all of them, and switch the command to this :
mvn test -Dkarate.options="--tags #{variable}" -Dkarate.env={variable}
It's a bit longer to do, but at least the tag(s) on each feature/scenario will correspond to the case(s) in which they are launched
I'm asking for help on how to run a feature file scenario just by name. I've been trying for a while and it does not come out. I know that can be done by tags or by line number, but I wonder if we can run a cucumber test by name, more or less with this nomenclature.
Given a file named "features/test.feature" with:
Feature:
Scenario: My first scenario
Given this step is blah blah blah
Scenario: My second scenario
Given this step too blah blah
I want to run a scenario by name from the console or with gradle, maybe similar this way
cucumber features/test.feuture::My second scenario
Or maybe with gradle
./gradlew cucumber::My second scenario
You didn't describe how you start cucumber so I can't help you with that.
When used from the CLI accepts --name REGEXP. This will only run scenarios whose names match REGEXP.
The #CucumberOptions annotation accepts name="REGEXP".
Cucumber < v6.0.0 looks at the environment. For maven you can add -Dcucumber.options=--name REGEXP. I don't know the equivalent for gradle. Take note that the escape characters maybe shell/build system dependent.
Cucumber v6.0.0 and above looks at the environment. For maven you can add -Dcucumber.filter.name="REGEXP".
See:
https://cucumber.io/docs/reference/jvm#running
https://github.com/cucumber/cucumber-jvm/tree/main/core
From cucumber 6.x, you can run a scenario with below CLI commands:
// Specify a scenario by its line number
$ cucumber-js features/my_feature.feature:3
// Specify a scenario by its name matching a regular expression
$ cucumber-js --name "topic 1"
But, these are time-consuming and repetitive. You can save a lot of time by using a dedicated VSCode Extension called Cucumber-Quick. This extension will allow you to run a scenario/feature just by right-clicking on them. It can save you from all the hustle.
You would call the scenario by its line number.
So assuming that the second scenario starts on line 5 in your feature file you could run:
cucumber features/test.feature:5
Difference between go test's two flags -parallel and -test.parallel and which flag gets precedence?
-parallel n
Allow parallel execution of test functions that call t.Parallel.
The value of this flag is the maximum number of tests to run
simultaneously; by default, it is set to the value of GOMAXPROCS.
Note that -parallel only applies within a single test binary.
The 'go test' command may run tests for different packages
in parallel as well, according to the setting of the -p flag
(see 'go help build').
Above documentation says that the number of tests that are run in parallel are equal to GOMAXPROCS if nothing is provided, but the behavior is not like that for me. Because I am running tests on a machine that has only 4 cores. But for me 8 tests run in parallel, so the behavior is more like following:
-test.parallel int
maximum test parallelism (default 8)
So what is the difference between the two? When to use which flag.
More Information
I am running all tests on a single package which has 9 tests, all of them are run parallely and all those exist in single test function.
The -test. flags are generated by go test command. The go test command produces a pkg.test binary on the fly and runs it with modified arguments. All the recognized arguments passed to go test will be converted. So, in your case: -parallel n becomes -test.parallel n.
So this command:
go test -parallel 6
creates:
pkg.test -test.parallel 6
First of all I should point out I'm new to Atlassian's Bamboo and continuous integration in general. This is the first project where I've used either.
I've created a raft of unit tests using the tSQLt framework. I've also configured Bamboo to:
Get a fresh copy of the repository from BitBucket
Drop & re-create the build DB
Use Red-Gate SQL Compare to deploy the DB objects from source to the build DB
Run the tSQLt tests
Output the results of the tests in XML format to a file called TestResults.xml
I've checked and can confirm that the TestResults.xml file is created.
In Bamboo I then added a JUnit Parser task to consume the contents of this TestResults.xml file. However when that task runs it returns this error:
Failed to parse test result file
At first I thought it might have meant that Bamboo could not find the file. I changed the task that created the results file to output a file called TestResults2.xml. When I did that the JUnit Parser returned this error:
Failing task since test cases were expected but none were found.
So I'm assuming that the first error message means Bamboo is finding the file, it just can't parse the file.
I have no idea where to start working out what exactly is the problem. Has anyone got any ideas?
I had a similar problem, but turned out to be weird behavior from bamboo needing file stamps being modified to have visibility of the JUnit file.
In Windows enviornment you just need to add "script task" before the "JUnit task"
powershell (ls *.xml).LastWriteTime = Get-Date
Reference
https://jira.atlassian.com/browse/BAM-12768
I have had several cases of this and was able to fix it by removing single quotes and greater than / less than characters from test names inside the *.rb file.
Example
test "make sure 'go_to_world' is removed from header and length < 23"
change to remove single quotes and < symbol
test "make sure go_to_world is removed from header and length less than 23"
Very common are contractions: "won't don't shouldn't", or possessives: "the vessel's data".
And also < or > characters.
I think there is a bug in the parser that just doesn't escape those characters in a test title appropriately.