How to force cmake to write test output after make test - testing

I'm building fortran project with cmake and I can't find solution to print to console FRUIT test results, they look something like these:
Test module initialized
. : successful assert, F : failed assert
7.00000000000000 -3.60000000000000 7.00000000000000
FFF
Start of FRUIT summary:
Some tests failed!
-- Failed assertion messages:
[_not_set_]:Expected [7.00000000000000], Got [1.00000000000000]
[_not_set_]:Expected [-3.60000000000000], Got [2.00000000000000]
[_not_set_]:Expected [7.00000000000000], Got [6.00000000000000]
-- end of failed assertion messages.
Total asserts : 3
Successful : 0
Failed : 3
Successful rate: 0.00%
Successful asserts / total asserts : [ 0 / 3 ]
Successful cases / total cases : [ 0 / 0 ]
-- end of FRUIT summary
The output I'm getting with make test looks like:
make test
Running tests...
Test project /home/konrad/Desktop/fortran
Start 1: unit_tests
1/1 Test #1: unit_tests ....................... Passed 0.01 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.01 sec
And since passing cmake tests doesn't mean passing FRUIT ones, I want to print FRUIT file everytime I run tests (just for the sake of making it work). I've tried adding printing commands at the end of test command (like this less), adding
-P ${CMAKE_TEST_DIR}/unit_tests.txt
at the end of add_test, building custom after build commands (which I can't make to run after make test so if you knew how to do that would solve it as well, seems like make test or test is not really a target)
Last part of my cmake file with all the testing code:
add_executable(task ${TASK_SOURCES})
add_executable(tests ${TEST_SOURCES})
enable_testing()
set(run_command "${CMAKE_BINARY_DIR}/tests")
set(UNIT_TEST_NAME "unit_tests.txt")
file(MAKE_DIRECTORY ${CMAKE_TEST_DIR})
add_test( NAME unit_tests
COMMAND sh -c
"rm -f ${CMAKE_TEST_DIR}/${UNIT_TEST_NAME} \
&& ${run_command} \
>> ${CMAKE_TEST_DIR}/${UNIT_TEST_NAME} \
&& less ${CMAKE_TEST_DIR}/${UNIT_TEST_NAME}"
)

I have solved a lack of tests output with custom CMake target that will invoke ctest in verbose mode etc.
e.g.
enable_testing()
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
--force-new-ctest-process
--verbose
--output-on-failure
)

The output of cmake tests is captured to a file (Testing/Temporary/LastTest.log in my current project).
cmake tests rely on the return code of the test program, with one test program per test.
If you wish to run a program that is a "driver" for your tests, I recommend to use add_custom_target. This commands will add a target that runs a command of your choice.
For instance:
add_custom_target(Name unit_tests tests)
add_dependencies(unit_tests tests)
I am not sure whether the add_dependencies line is needed in this case though (as tests is a target managed by cmake).
Then, you can run
make unit_tests
and it will run your test driver.

Related

CTest misparsing add_test calls

I am trying to run some tests using CTest.
My CTestTestFile.cmake contains a single test
add_test(NAME test1
COMMAND python script.py args
)
but when I try run ctest it outputs the following
Start 1: NAME
Could not find executable test1
Looked in the following places:
test1
test1.exe
Release/test1
Release/test1.exe
Debug/test1
Debug/test1.exe
MinSizeRel/test1
MinSizeRel/test1.exe
RelWithDebInfo/test1
RelWithDebInfo/test1.exe
Deployment/test1
Deployment/test1.exe
Development/test1
Development/test1.exe
Unable to find executable: test1
1/1 Test #1: NAME .............................***Not Run 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.14 sec
The following tests FAILED:
1 - NAME (Not Run)
This suggests CTest thinks I'm using the short version of the command, so that my test's name is NAME, the command is test1 and python script.py args are the arguments to the command, when this is of course not the case, as I'm obviously using the long version of the command.
I have CMake version 3.24.1 installed.
The CTest runtime, unlike CMake, accepts only the "basic" signature documented at the end of the usual add_test documentation:
add_test(<name> <command> [<arg>...])
Notice that CTest thought your test was named NAME.
So you have to write:
add_test(test1 "python" "script.py" "args")

Execute process at install stage, every time

I would like to run program to perform do extra installation tasks from CMake. My attempted solution, based on INSTALL(CODE ...) is (this is a real MWE):
macro(MY_EXTRA_STUFF ARG)
execute_process(...)
endmacro()
install(CODE "MY_EXTRA_STUFF(${SOME_ARG})")
but CMake complains when I run ninja install (or make install, depending on generator in use):
[0/1] Install the project...
-- Install configuration: ""
CMake Error at cmake_install.cmake:41 (MY_EXTRA_STUFF):
Unknown CMake command "MY_EXTRA_STUFF".
FAILED: CMakeFiles/install.util
cd /tmp && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.
Is there a way to smuggle my own code into the install stage? The code is too long to fit inside install(CODE "...") nicely. A bonus to do it without an external file. Thanks!
The code passed to install(CODE) is executed as standalone CMake code, thus it shouldn't use definitions (functions,macros, variables) from the rest of CMakeLists.txt.
That is, install(CODE) behaves similar as install(SCRIPT) with a standalone script containing given code.
The thing is that configuration stage (when you call cmake to configure your project) and installation stage, which, as you can see, calls /usr/bin/cmake -P cmake_install.cmake, are separate cmake invocations. These invocations parse different files, so they unaware about context of each other.

ctest run only existing tests

I have some tests in my build that are optional. My CMakeLists.txt looks approximately like:
add_custom_target(all-tests)
add_executable(A ...)
add_dependencies(all-tests A)
add_test(NAME A COMMAND ...)
add_executable(B ...)
add_dependencies(all-tests B)
add_test(NAME B COMMAND ...)
## optional
add_executable(C EXCLUDE_FROM_ALL ...)
add_test(NAME C COMMAND ...)
The idea being that I can run
$ make all-tests
$ ctest
To both compile all my unit tests and then run them. The problem is, since C is optional, it doesn't build (all desired). And since it didn't build, it doesn't exist, and gets reported as having failed:
The following tests FAILED:
6 - C (Not Run)
Errors while running CTest
Is there a way to ignore this failure / have CTest not try to run tests that don't exist / otherwise express the idea of an optional test? I have a class of these optional tests that have similar names, so it'd be nice to be able to use ctest -R to run them, if they are built, rather than having to configure some script or other approach.
Make the optional tests depend on the existence of the built test executables by setting the REQUIRED_FILES property:
add_test(NAME C COMMAND ...)
add_executable(C EXCLUDE_FROM_ALL ...)
set_property(TEST C PROPERTY REQUIRED_FILES "$<TARGET_FILE:C>")
The use of the generator expression TARGET_FILE requires CMake 3.0 or later.

gulp-npm-test running Jest tests (all passing), but claiming to fail

I've installed gulp-npm-test following their documentation, that is, in my gulp directory I've got a file test.js that looks like this:
var gulp = require('gulp')
require('gulp-npm-test')(gulp)
var gulp = require('gulp-npm-test')(gulp, {
withoutNpmRun: false
})
But when I run gulp test I get the output like the following:
[17:12:41] Using gulpfile ~/my-project/gulpfile.js
[17:12:41] Starting 'test'...
> my-project#0.0.1 test /Users/wogsland/my-project
> jest
PASS frontend/tests/components/atoms/InputText.test.js
.
.
(many more Jest test passes, no fails)
.
.
FAIL gulp/tasks/test.js
● Test suite failed to run
Your test suite must contain at least one test.
at onResult (node_modules/jest/node_modules/jest-cli/build/TestRunner.js:192:18)
Test Suites: 1 failed, 135 passed, 136 total
Tests: 135 passed, 135 total
Snapshots: 209 passed, 209 total
Time: 92.237s
Ran all test suites.
npm ERR! Test failed. See above for more details.
What am I missing here? I've go a bunch of tests
Looks like the matcher that figures out which files are tests, matches everything that includes in the filename test.js. The default is [ '**/__tests__/**/*.js?(x)', '**/?(*.)(spec|test).js?(x)' ]. So either adapt this one, its testMatch in your jest settings. Or use testPathIgnorePatterns to exclude the /gulp folder.

Calling Unittest++ from cmake created makefile

I recently started learning cmake, and have run into a small issue. I got both my executable and the unit tests to compile from the generated makefile without issue. If I run ./test in the build directory, the tests created in UnitTest++ run and complete as expected, printing the results. Is there any way to get make test to simply run the test executable rather than running it inside ctest framework or should I go about this a different way?
Here is a minimal working example of my code:
src/main/main.c is a simple empty main function
src/test/testMain.cpp:
#include <UnitTest++/UnitTest++.h>
TEST(FailSpect)
{
CHECK(false);
}
int main()
{
UnitTest::RunAllTests();
}
CMakeLists.txt:
cmake_minimum_required( VERSION 2.6 )
project( myProject)
enable_testing()
set( myProjectMain
src/main/main.c
)
set( myProjectSrc
)
set( myProjectTestSrc
src/test/testMain.cpp
)
add_executable( myExecutable ${myProjectMain} ${myProjectSrc} )
add_executable( testSuite ${myProjectTestSrc} ${myProjectSrc} )
target_link_libraries( testSuite UnitTest++ )
add_test( testExe testSuite )
make test output:
Running tests...
Start processing tests
Test project /myProjectDir/build
1/ 1 Testing testExe Passed
100% tests passed, 0 tests failed out of 1
./testSuite output:
/myProjectDir/src/test/testMain.cpp:5: error: Failure in FailSpect: false
FAILURE: 1 out of 1 tests failed (1 failures).
Test time: 0.00 seconds.
I have sorted out how to do this. First remove the lines:
enable_testing()
and
add_test(testExe testSuite)
and replace them by the line:
add_custom_target(test ./testExe
DEPENDS ./testExe)
at the end of the CMakeLists.txt file. Now make (all) builds both the tests and the main program. If everything is built already, then make test will just check that the tests are built and run them, producing:
[100%] Built target testExe
/myProjectDir/src/test/testMain.cpp:5: error: Failure in FailSpect: false
FAILURE: 1 out of 1 tests failed (1 failures).
Test time: 0.00 seconds.
[100%] Built target test
If the tests are out of date (after a make clean for instance), then make test will produce:
[100%] Building CXX object CMakeFiles/testExe.dir/src/test/testMain.cpp.o
Linking CXX executable testExe
[100%] Built target testExe
/myProjectDir/src/test/testMain.cpp:5: error: Failure in FailSpect: false
FAILURE: 1 out of 1 tests failed (1 failures).
Test time: 0.00 seconds.
[100%] Built target test