It is possible to execute one or several groups of tests in Codeception. There are multiple ways to assign a test to a group (tag with #group annotations, add a group configuration in a yaml or add a group txt file).
Is there a way to list all tests which are assigned to a specific group?
I tried codecept dry-run, but this does not accept the group parameter, and codecept run executes all tests (which I dont want).
There is no built-in command in Codeception for that.
If you only specify groups via #group tags you can use grep to find the test files for a group. If you use other ways to define groups this method will not work.
For example to search for all tests files with the group shop:
grep -ro '#group shop' tests/
You may need to replace tests/ with the path where your tests are located.
Similar command to list all groups:
grep -rho '#group.*' tests/ |sort |uniq
Related
I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
I'm running some tests using Elixir and ExUnit and I would like to pass a filter on the command line that I can select tests that have one tag but not the other.
Example:
Test A tags: #foo, #bar
Test B tags: #foo
I'd like to run just the test B with tag #foo, but not the A one with #foo and #bar, how do I achieve this?
mix test task accepts any number of file names, as well as --include and --exclude configs.
More info about tags and filters might be found in the documentation.
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
I am using dbt 0.18.1 and I follow the documentation about tags however I am curious to know how to run multi-tag selector as arguments.
According to this:
https://github.com/fishtown-analytics/dbt/pull/1014
Select using a mix of tags, fqns, and parent/child selectors:
$ dbt run --model tag:nightly+ salesforce.*+
Unfortunately this is not really a "mix of tags".
I have tags of [mixpanel_tests, quality] and I wish to run models that have both tags included (not separated). If I run dbt run -m tag:quality -t blabla
I would have executed all models that have QUALITY in the array of tags regardless if its single argument or multiple argument however I wish to run ONLY quality marked. How to do that?
How do I specify 2 tags or 3 tags selector to run models with the mentioned tags (i.e mixpanel_tests, quality - but only those models that have both tags defined). More or less an AND clause rather than an OR clause.
Hmm I hope it is clear. How to have multitag selector that executes only the combination of tags given?
Check out the intersection operator. It's new in dbt v0.18, and it's for this use case exactly.
dbt run -m tag:mixpanel_tests,tag:quality
We can select multiple scenario's by including the following on the command line:
-Dcucumber.options="--tags #S1,#S2,#S6"
And if I want to exclude #S6 I can with:
-Dcucumber.options="--tags ~#S6"
But if I want to include #S1, #S2 and exclude #S6 all tags are ignored with:
-Dcucumber.options="--tags #S1,#S2,~#S6"
and all tags are also ignored if I try to double up on the options with:
-Dcucumber.options="--tags #S1,#S2" -Dcucumber.options="--tags ~#S6"
Is there a command line way to include and exclude in the one command line?
The reason I would like to do this is to run all of a type of test but exclude tests that use some external system that may be down temporarily.
EDIT: Turns out I was not fully aware of the difference between AND and OR in the Cucumber world. This SO answer and this article was a good reference, look for "Logical OR" and "Logical AND":
so to run S1 OR S2 AND NOT S3
mvn test -Dkarate.options="--tags #S1,#S2 --tags ~#S3"
Using the Java (or Runner) API, just use commas for OR and separate strings for AND:
Results results = Runner.path("classpath:com/myco/some.feature")
.tags("#S1,#S2", "~#S3")
.parallel(5);
For really advanced cases, also see: https://stackoverflow.com/a/67224240/143475