I'm trying to get familiar with sanitizers as ASAN, LSAN etc and got a lot of useful information already from here: https://developers.redhat.com/blog/2021/05/05/memory-error-checking-in-c-and-c-comparing-sanitizers-and-valgrind
I am able to run all sort of sanitizers on specific files, as shown on the site, like this:
clang -g -fsanitize=address -fno-omit-frame-pointer -g ../TestFiles/ASAN_TestFile.c
ASAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer ./a.out >../Logs/ASAN_C.log 2>&1
which generates a log with found issue. Now I would like to extend this to run upon building the project with cmake. This is the command to build it at the moment:
cmake -S . -B build
cd build
make
Is there any way I can use this script with adding the sanitizers, without having to alter the cmakelist.txt file??
For instance something like this:
cmake -S . -B build
cd build
make -fsanitize=address
./a.out >../Logs/ASAN_C.log 2>&1
The reason is that I want to be able to build the project multiple times with different sanitizers (since they cannot be used together) and have a log created without altering the cmakelist.txt file (just want to be able to quickly test the whole project for memory issues instead of doing it for each file created).
You can add additional compiler flags from command line during the build configuration:
cmake -D CMAKE_CXX_FLAGS="-fsanitize=address" -D CMAKE_C_FLAGS="-fsanitize=address" /path/to/CMakeLists.txt
If your CMakeLists.txt is configured properly above should work. If that does not work then try adding flags as environment variable:
cmake -E env CXXFLAGS="-fsanitize=address" CFLAGS="-fsanitize=address" cmake /path/to/CMakeLists.txt
Currently I have a pipeline that builds a C++ program currently like this:
build:
stage: build
script:
- rm -rf .git/modules/docs .git/modules/libraries/fc ./docs ./libraries/fc
- git submodule sync
- git submodule update --init --recursive
- rm -rf build
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Release ..
- make -j$(nproc)
This build must still build, but I also would like to build this in parallel but with a different cmake option;
cmake -DBOOST_ROOT="$BOOST_ROOT" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTNET=1
I have read about the parallel option thats included in gitlab-ci, but haven't had success with incorporating this.
Any insight it greatly appreciated! Will update if solved prior to answers
You need to have two jobs. This article has some good ideas of how to set it up.
Now, Gitlab infers dependencies and assumes that you want to run them in order, so if you add a needs: [] list, it helps it build a graph. If you want two of them to run at the same time, then you remove their dependencies.
If you have something before this build, like a test or compare, you can use needs: ["test"] or needs: ["prepare"] or whatever jobs you want to run before this build step, but you can use [] to tell the CI no dependencies are needed and to run them as soon as possible.
build:
stage: build
needs: []
script:
- .. common stuff
- cmake -DCMAKE_BUILD_TYPE=Release ..
- make # I'd probably remove this in a CI situation -j$(nproc)
build2:
stage: build
needs: []
script:
- .. common stuff
- cmake -DCMAKE_BUILD_TYPE=Release AND OTHER OPTIONS ..
- make # I'd probably remove this in a CI situation -j$(nproc)
You can make use of parallel:matrix jobs. This feature runs one job multiple times but with a different variables set each time.
In your case it would look similar to this:
build:
stage: build
script:
- rm -rf .git/modules/docs .git/modules/libraries/fc ./docs ./libraries/fc
- git submodule sync
- git submodule update --init --recursive
- rm -rf build
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=${DCMAKE_BUILD_TYPE}
- make -j$(nproc)
parallel:
matrix:
# Initial state of your job
- DCMAKE_BUILD_TYPE="Release .."
# Other options...
- DCMAKE_BUILD_TYPE=Release
DBOOST_ROOT="$BOOST_ROOT"
DBUILD_TESTNET=1
This technique assumes that your jobs use the same variable subset, but with different values for each execution.
You can find more info in the official docs and here is another example, docker builds in this scenario, but the principle should be clear.
I'm redefining my project's cmake configuration based on what I have learned from Daniel Pfeifer's “Effective CMake" talks, but I still cannot get my head around ctest.
Currently I have the following setup for my tests:
enable_testing()
find_program(MEMCHECK_COMMAND valgrind)
# executable
add_executable(myTest ${SRC_LIST})
target_link_libraries(myTest ${LIB_LIST})
# test
add_test(NAME myTest_test
COMMAND ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_test
COMMAND ctest -R myTest_test --output-on-failure
DEPENDS myTest)
# memcheck
add_test(NAME myTest_memcheck
COMMAND ${MEMCHECK_COMMAND} ${CMAKE_CURRENT_BINARY_DIR}/myTest)
add_custom_target(myTest_memcheck
COMMAND ctest -R myTest_memcheck --output-on-failure
DEPENDS ${TEST_NAME})
and I could run make myTest_test and make myTest_memcheck and know that I didn't break anything.
It pains me that I have to do this for every test executable (and project is big so I have dozens of them) and even extracted as a function it just feels wrong.
I have read ctest docs (https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest) and dashboards seems almost perfect to me - if I could just run ctest -R myTest -T test, ctest -R myTest -T memcheck and ctest -R myTest -T coverage I would be very happy :) (and even more happy if those commands will trigger build if needed).
The problem is that include(CTest) creates targets that I do not need (Continuous/Nightly dashboards with Start/Update/Configure/Submit steps). It will be used solely as experimental builds and executed by devs during coding. For CI I have Jenkins with it's own magic. I would like to end up with:
- Dashboard/Build
- Dashboard/Test
- Dashboard/MemCheck
- Dashboard/Coverage
What should I do? Should I create manually my own dashboard (if it is possible) or drop this idea and stay with my custom targets?
Currently I have like ~4 scripts that just wrap around some simple CMake commands like "FILE(COPY ...)" or "CONFIGURE_FILE(...)" so I can use them with the -P flag in custom commands like:
ADD_CUSTOM_TARGET( ${target}_TMP_RESOURCES ALL
COMMAND ${CMAKE_COMMAND} -DFILES_LIST="${${target}_RESOURCES}" -DDESTINATION="${BUILD_INTERMEDIATE_DIR}/${target}/bin/assets" -DEXCLUDE_EXT=".svn .git CVS .DS_Store" -P ${ROOT_DIR}/cmake/scripts/copyFiles.cmake
DEPENDS "${target}_TMP_BIN_DIR"
COMMENT "Collecting resource files..." )
The -E flag looks like what I want, unfortunately it supports only very few platform commands according to the documentation.
So, is there a better way to use CMake commands in custom commands without running cmake in script mode and having to write these simple scripts?
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?
You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
You could call ctest directly, after cmaking and making your project.
ctest --verbose
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.
The second way is to get ctest to show you the content of log files after running tests:
place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
run make test again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section.
Good luck!
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
make check CTEST_OUTPUT_ON_FAILURE=TRUE
This makes test output more verbose:
make test ARGS="-V"
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
ctest -VV or ctest --extra-verbose
From documentation:
Enable more verbose output from tests.
Test output is normally suppressed and only summary information is
displayed. This option will show even more test output.
There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Edit:
As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml
and other files are generated too in Testing Folder