I'm using gcov and I'd like to gather coverage results on a per test case basis.
QUESTION
When I execute the googletest executable, is it possible to pass in an argument on command line that says execute only the Nth test case?
I'd even be ok with passing in the name of the fixture+case; I can just use Python with regex to pull out all the tests.
Obviously I can accomplish the same thing by having one test case per .cpp file but that sounds ... stupid.
googletest allows to run a single test case or even the subset of the tests. You can specify a filter string in the GTEST_FILTER environment variable or in the --gtest_filter command line option and googletest will only run the tests whose full names (in the form of TestSuiteName.TestName) match the filter. More information about the format of the filter string can be found in Running a Subset of the Tests section. Also googletest supports --gtest_list_tests command line option to print the list of all test cases. It can be useful in your case.
Related
Is there a straightforward way when using ctest to get the number of tests passed (and/or failed) within a script, e.g., BASH, without grep-ping through a generated output file?
a straightforward way ... without grep-ping
No, I believe there is not.
You can also "grep" the count the lines Test failed. and Test passed. from CMake the_build_dir/Testing/Temporary/LastTest.log.
You could potentially generate ctest XML report to a dashboard and then parse the XML reports (instead of sending them). It's nowhere as straightforward, as ctest script has to be written that configures, builds and tests the project and then separate XML tool needs to parse the result.
You can also run a cdash server and let that ctest script upload the results to cdash and then query cdash server with simple curl 'https://your.cdash.server/api/v1/index.php?project=TheProjectName' | jq '.buildgroups[] | select(.id == 2).builds[] | { "pass": .test.pass, "fail": .test.fail, }. The querying is simple, but.. it needs to run a cdash server and also test with ctest script, it's not near straightforward..
Btw, it's easy to get the number of failed tests - it's just wc -l the_build_dir/Testing/Temporary/LastTestsFailed.log.
We can select multiple scenario's by including the following on the command line:
-Dcucumber.options="--tags #S1,#S2,#S6"
And if I want to exclude #S6 I can with:
-Dcucumber.options="--tags ~#S6"
But if I want to include #S1, #S2 and exclude #S6 all tags are ignored with:
-Dcucumber.options="--tags #S1,#S2,~#S6"
and all tags are also ignored if I try to double up on the options with:
-Dcucumber.options="--tags #S1,#S2" -Dcucumber.options="--tags ~#S6"
Is there a command line way to include and exclude in the one command line?
The reason I would like to do this is to run all of a type of test but exclude tests that use some external system that may be down temporarily.
EDIT: Turns out I was not fully aware of the difference between AND and OR in the Cucumber world. This SO answer and this article was a good reference, look for "Logical OR" and "Logical AND":
so to run S1 OR S2 AND NOT S3
mvn test -Dkarate.options="--tags #S1,#S2 --tags ~#S3"
Using the Java (or Runner) API, just use commas for OR and separate strings for AND:
Results results = Runner.path("classpath:com/myco/some.feature")
.tags("#S1,#S2", "~#S3")
.parallel(5);
For really advanced cases, also see: https://stackoverflow.com/a/67224240/143475
I have a CTestList.cmake file containing this function definition and call:
function(add_test_r testname )
add_test(
${testname} python executeRegressionTestCase.py ${testname}
)
endfunction(add_test_r)
add_test_r(Test01)
I need to support testing through CMake directly, cmake ..; make test, and through CTest without running CMake first. When CMake builds the test target and I run make test, the function above executes fine. However, when I run the same file with CTest, ${testname} is empty. Are function variables not supported in CTestList when running with CTest or is there something I'm missing?
I don't have a definitive answer, but after some testing it looks like CMake does a preprocessing step for the input files. Specifically, it reads in CTestList.cmake, evaluates the variables and functions, and generates an "expanded" CTestTestfile.cmake. Running CTest with a steering script does not run the preprocessing step so the variables and functions are not expanded. A few people on the internet have suggested that the CTestTestfile.cmake used in the steering script should be generated by CMake in the first place. Unfortunately, thats not the use case I'm looking for, but it may help someone else with running into this problem.
I have a program on my computer, let's say C:/Tools/generate_v23_debug.exe
I have a FindGenerate.cmake file which allows CMake to find that exact path to the executable.
So in my CMake code, I do:
find_program(Generate)
if (NOT Generate_FOUND)
message(FATAL_ERROR "Generator not found!")
So CMake has found the executable. Now I want to call this program in a custom command statement. Should I use COMMAND Generator or COMMAND ${GENERATOR_EXECUTABLE}? Will both of these do the same thing? Is one preferred over the other? Is name_EXECUTABLE a variable that CMake will define (it's not in the FindGenerate.cmake file), or is it something specific to someone else's example code I'm looking at? Will COMMAND Generator be expanded to the correct path?
add_custom_command(
OUTPUT blahblah.txt
COMMAND Generator inputfile1.log
DEPENDS Generator
)
find_program stores its result into the variable given as a first argument. You can verify this by inserting some debug output:
find_program(GENERATOR Generate)
message(${GENERATOR})
Note that find_program does not set any additional variables beyond that. In particular, you mentioned Generate_FOUND and GENERATOR_EXECUTABLE in your question and neither of those gets introduced implicitly by the find_program call.
The second mistake in your program is the use of the DEPENDS option on the add_custom_command. DEPENDS is used to model inter-target dependencies at build time and not to manipulate control flow in the CMakeLists. For example, additional custom command can DEPEND on the output of your command (blahblah.txt), but a custom command cannot DEPEND on the result of a previous find operation.
A working example might look something like this:
find_program(GENERATOR Generate)
if(NOT GENERATOR)
message(FATAL_ERROR "Generator not found!")
endif()
add_custom_command(
OUTPUT blahblah.txt
COMMAND ${GENERATOR} inputfile1.log
)
P.S.: You asked why the code examples were not properly formatted in your question. You indented everything correctly, but you need an additional newline between normal text and code paragraphs. I edited your question accordingly.
First of all I should point out I'm new to Atlassian's Bamboo and continuous integration in general. This is the first project where I've used either.
I've created a raft of unit tests using the tSQLt framework. I've also configured Bamboo to:
Get a fresh copy of the repository from BitBucket
Drop & re-create the build DB
Use Red-Gate SQL Compare to deploy the DB objects from source to the build DB
Run the tSQLt tests
Output the results of the tests in XML format to a file called TestResults.xml
I've checked and can confirm that the TestResults.xml file is created.
In Bamboo I then added a JUnit Parser task to consume the contents of this TestResults.xml file. However when that task runs it returns this error:
Failed to parse test result file
At first I thought it might have meant that Bamboo could not find the file. I changed the task that created the results file to output a file called TestResults2.xml. When I did that the JUnit Parser returned this error:
Failing task since test cases were expected but none were found.
So I'm assuming that the first error message means Bamboo is finding the file, it just can't parse the file.
I have no idea where to start working out what exactly is the problem. Has anyone got any ideas?
I had a similar problem, but turned out to be weird behavior from bamboo needing file stamps being modified to have visibility of the JUnit file.
In Windows enviornment you just need to add "script task" before the "JUnit task"
powershell (ls *.xml).LastWriteTime = Get-Date
Reference
https://jira.atlassian.com/browse/BAM-12768
I have had several cases of this and was able to fix it by removing single quotes and greater than / less than characters from test names inside the *.rb file.
Example
test "make sure 'go_to_world' is removed from header and length < 23"
change to remove single quotes and < symbol
test "make sure go_to_world is removed from header and length less than 23"
Very common are contractions: "won't don't shouldn't", or possessives: "the vessel's data".
And also < or > characters.
I think there is a bug in the parser that just doesn't escape those characters in a test title appropriately.