I was wondering if we can create tests depending on each other. For example, if the first fails, run the second on and so on and skip downwards tests wherever the tests succeed.
The goal would be to save CPU consumption on superfluous tests if the major one passes.
Any idea on how to achieve something similar? or how to build a test with these criteria?
dbt has many ways to select subsets of tasks. the docs give a good selection of test selection examples.
My advice would be to group your tests into distinct phases using either tags or YAML selectors. You might even get away with having your phase one tests be --schema and phase two be --data.
In your orchestrator (Jenkins, Circle CI, GitHub Actions) you'd add your control flow (if-then) logic. Below is a hacky bash script example.
echo 'phase one: running'
if dbt test --tag phase_one; then
echo 'phase one: passed'
else
echo 'phase one: failed'
echo 'phase two: running,
if dbt test --tag phase_two; then
echo 'phase two: passed'
else
echo 'phase two: failed'
fi
fi
Related
Consider there are 100 scenarios and I want to run 99 scenarios in prod environment and 100 on stage environment .
Is there a way to achieve this in karate ?
Things I tried
1. Create a feature file with 1 scenario and another feature file with remaining 99
2. Add tag to the 1 scenario file
3. Ignore that while running
But then when I use it in jenkins job I have to run one command to run on both machines so would not work
The best solution for this case is the karate.abort() API:
So in the "special" scenario #100 - you can do this:
Scenario:
* eval if (karate.env == 'prod') karate.abort()
# normal scenario steps
Please note that there are advanced options for tag selectors in Karate 0.9.0 onwards - but just stick to the solution above please :)
Tag the 100th scenario with #hundred and just run the following command when you want to run 99 scenarios
mvn test -Dkarate.options="--tags ~#hundred"
And simply run mvn test when you want to run all tests.
You can tag a scenario
#hundred
Scenario: the scenario only played in one case
Given <...>
But you can also tag a Feature
#hundred
Feature: The feature containing the scenario only played in one case
Background:
* <..>
Scenario: <...>
Edit after your answer :
You could use a second runtime variable :
mvn test -Dkarate.options="--tags ~#{variable2}" -Dkarate.env={variable}
Or even use the same runtime variable :
mvn test -Dkarate.options="--tags ~#{variable}" -Dkarate.env={variable}
And that maybe wouldn't be the best solution, but you can add #Prod to the 99 scenarios, and #Stage to all of them, and switch the command to this :
mvn test -Dkarate.options="--tags #{variable}" -Dkarate.env={variable}
It's a bit longer to do, but at least the tag(s) on each feature/scenario will correspond to the case(s) in which they are launched
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
We can select multiple scenario's by including the following on the command line:
-Dcucumber.options="--tags #S1,#S2,#S6"
And if I want to exclude #S6 I can with:
-Dcucumber.options="--tags ~#S6"
But if I want to include #S1, #S2 and exclude #S6 all tags are ignored with:
-Dcucumber.options="--tags #S1,#S2,~#S6"
and all tags are also ignored if I try to double up on the options with:
-Dcucumber.options="--tags #S1,#S2" -Dcucumber.options="--tags ~#S6"
Is there a command line way to include and exclude in the one command line?
The reason I would like to do this is to run all of a type of test but exclude tests that use some external system that may be down temporarily.
EDIT: Turns out I was not fully aware of the difference between AND and OR in the Cucumber world. This SO answer and this article was a good reference, look for "Logical OR" and "Logical AND":
so to run S1 OR S2 AND NOT S3
mvn test -Dkarate.options="--tags #S1,#S2 --tags ~#S3"
Using the Java (or Runner) API, just use commas for OR and separate strings for AND:
Results results = Runner.path("classpath:com/myco/some.feature")
.tags("#S1,#S2", "~#S3")
.parallel(5);
For really advanced cases, also see: https://stackoverflow.com/a/67224240/143475
I'm asking for help on how to run a feature file scenario just by name. I've been trying for a while and it does not come out. I know that can be done by tags or by line number, but I wonder if we can run a cucumber test by name, more or less with this nomenclature.
Given a file named "features/test.feature" with:
Feature:
Scenario: My first scenario
Given this step is blah blah blah
Scenario: My second scenario
Given this step too blah blah
I want to run a scenario by name from the console or with gradle, maybe similar this way
cucumber features/test.feuture::My second scenario
Or maybe with gradle
./gradlew cucumber::My second scenario
You didn't describe how you start cucumber so I can't help you with that.
When used from the CLI accepts --name REGEXP. This will only run scenarios whose names match REGEXP.
The #CucumberOptions annotation accepts name="REGEXP".
Cucumber < v6.0.0 looks at the environment. For maven you can add -Dcucumber.options=--name REGEXP. I don't know the equivalent for gradle. Take note that the escape characters maybe shell/build system dependent.
Cucumber v6.0.0 and above looks at the environment. For maven you can add -Dcucumber.filter.name="REGEXP".
See:
https://cucumber.io/docs/reference/jvm#running
https://github.com/cucumber/cucumber-jvm/tree/main/core
From cucumber 6.x, you can run a scenario with below CLI commands:
// Specify a scenario by its line number
$ cucumber-js features/my_feature.feature:3
// Specify a scenario by its name matching a regular expression
$ cucumber-js --name "topic 1"
But, these are time-consuming and repetitive. You can save a lot of time by using a dedicated VSCode Extension called Cucumber-Quick. This extension will allow you to run a scenario/feature just by right-clicking on them. It can save you from all the hustle.
You would call the scenario by its line number.
So assuming that the second scenario starts on line 5 in your feature file you could run:
cucumber features/test.feature:5
Is it possible to split testing process in go for some package ?
go test package - uses all function Test* in all *_test.go files in package. If you have a lot of tests and try to TDT its rather boring to receive always all test logs.
See 'go help test' and 'go help testflag'. Quoting from the later:
...
-run regexp
Run only those tests and examples matching the regular
expression.
...