Access CMake cache variable from CTest script - cmake

My project links to a third-party library that comes with a valgrind suppression file, as well as a CMake script. The script stores the location of the suppression file in a CMake cache variable.
I have written a CTest script that runs continuous tests on my project and submit to a dashboard. I would like to use the suppression file during the memory-checking stage. Unfortunately, the CTest script does not seem to know anything about the CMake cache. How can I access the CMake cache variable from my CTest script?

You can't directly access CMake cache variables from the ctest -S script.
However, you could possibly:
include the third party CMake script in the ctest -S script with
"include" (after the update step, so the source tree is up to date)
read the CMakeCache.txt file after the configure step to pull out the
cache variable of interest
add code to your CMakeLists.txt file to write out a mini-script that
contains just the information you're looking for
For (1), the code would be something like:
include(${CTEST_SOURCE_DIRECTORY}/path/to/3rdParty/script.cmake)
This would only be realistically possible if the script does only simple things like set variable values that you can then reference. If it does any CMake-configure-time things like find_library or add_executable, then you shouldn't do this.
For (2):
file(STRINGS ${CTEST_BINARY_DIRECTORY}/CMakeCache.txt result
REGEX "^CURSES_LIBRARY:FILEPATH=(.*)$")
message("result='${result}'")
string(REGEX REPLACE "^CURSES_LIBRARY:FILEPATH=(.*)$" "\\1"
filename "${result}")
message("filename='${filename}'")
For (3):
In the CMakeLists.txt file:
file(WRITE "${CMAKE_BINARY_DIR}/mini-script.cmake" "
set(supp_file \"${supp_file_location}\")
")
In your ctest -S script, after the ctest_configure call:
include("${CTEST_BINARY_DIRECTORY}/mini-script.cmake")
message("supp_file='${supp_file}'")
# use supp_file as needed in the rest of your script

The tests should be configured in your CMakeLists.txt file via the command ADD_TEST(). The CTest script then simply calls ctest_test() to run all the configured tests. By doing it this way, you get access to the cache variables, plus you can just run make test at any point to run the tests. Save the CTest script for the nightly testing and/or Dashboard submissions.

Related

Have CMake execute a file which is not CMakeLists.txt

Is that true that you can't customize the name of your CMakeLists.txt file? I read in a few places that make suffers from the same problem, but that's completely not true, you sure can:
~$ make -f whatever_name_you_feel_like
Can't you do this with CMake?
My situation is as follows: The project leader wants to have a certain CMakeLists.txt file run in the CI workflow and another when developing. I thought it would be possible to just keep 2 CMake files and tell cmake which one to execute.
It's not possible to use file with a name different to CMakeLists.txt, but I'm almost certain that's not actually what you want to do anyways.
I assume the cl version and the development version are mostly similar and only some details change. In this case you should not duplicate the logic. Instead add one or multiple options to your cmake project that can set when you set up the build dir and can even be changed without reconfiguring the whole project from scratch. Basically you add a cache variable to CMakeLists.txt which allows the user to overwrite the default value via -D command line option. The value can also be modified after the initial configuration using cmake-gui.
cmake_mimimum_required(VERSION 3.0.2)
project(MyProject)
# option set to true by default
set(MY_PROJECT_COMMAND_LINE_BUILD 1 CACHE BOOL "Use the command line configuration for MyProject")
#logic common to both configurations
add_executable(MyProg foo.cpp bar.cpp)
if(MY_PROJECT_COMMAND_LINE_BUILD)
#logic only for command line build
target_compile_definitions(MyProg PRIVATE COMMAND_LINE_BUILD)
else()
# logic only for non-command line build
target_compile_definitions(MyProg PRIVATE DEVELOPMENT_BUILD)
endif()
Ironically you could set up both from the command line:
Command line build
cmake -S sourceDir -B buildDir
Development build
cmake -D MY_PROJECT_COMMAND_LINE_BUILD:BOOL=0 -S sourceDir -B buildDir
If you don't want to enter the cache values in the command line every time you set up the project, you could also use a cmake script file to initialize the cache values using the -C command line option.
cmake -C developmentVersion.cmake -S sourceDir -B buildDir
developmentVersion.cmake:
set(MY_PROJECT_COMMAND_LINE_BUILD 0 CACHE BOOL "Use the command line configuration for MyProject")
Theoretically you could the whole CMakeLists.txt file in an if else endif structure and use include in one of the alternatives to competely replace the standard logic in the CMakeLists.txt file, but imho this is not a good idea.
Can't you do this with CMake?
No, it's not possible.
The project leader wants to have a certain CMakeLists.txt file run in the CI workflow and another when developing.
One way: copy or symlink proper CMakeLists.txt before executing cmake.
Preferably one would use cmake scripting language:
# CMakeLists.txt
if (MODE STREUQAL "CI_WORKFLOW")
include(CMakeLists-ci-workflow.txt)
elseif (MODE STREQUAL "DEVELOPING")
include(CMakeLists-developing.txt)
else()
message("SUPER ERROR")
fi()
and then separate CMakeLists-ci-workflow.txt and separate CMakeLists-developing.txt and do cmake -D MODE=DEVELOPING or -D MODE=CI_WORKFLOW.
But overall, the idea of "separate CMakeLists.txt" sounds bad to me. Instead use CMAKE_BUILD_TYPE=Debug for developing and CMAKE_BUILD_TYPE=Release for release builds, and use other cmake variables to differentiate settings, instead of duplicating configuration.

How to incorporate unittest in cmake project structure

I'm working on an embedded project. As a part of this project, I have unit-tests that use gcc and Gtest. The question what is the best to approach to incorporate these unit-tests. My current implementation is that I have a different build type called unittest. I have a clause of CMAKE_BUILD_TYPE and decide which sources to use which targets to create. I see this is not a good design and this screws up multiconfiguration gnerators. What could be the elegant solution for this?
Thanks in advance for answering.
Create separate executables for testing and use ctest:
add_test to add the combination of executable+command line parameters as a test ctest runs and enable_testing() in the toplevel CMakeLists.txt.
This allows you to simply run ctest in the build dir and you can pass a configuration to test using -C command line option.
add_executable(MyTest test.cpp test_helper.cpp test_helper.h)
target_include_directories(MyTest PRIVATE .)
target_link_libraries(MyTest PRIVATE theLibToTest)
add_test(NAME NameOfTest COMMAND MyTest --gtest_repeat=1000)
enable_testing()
Running ctest from the build directory runs a test named NameOfTest. For multi configuration generators you simply specify the configuration to test with the -C command line option
ctest -C Release
Of course you can use add_test multiple times to add different test executables or the same test executable with different command line options.
Furthermore I recommend figuring out a way of storing results in a file, since this makes via the test parameters, since ctest's output probably won't do the trick. I'm not familiar enough with gtest to give advice on this.
Btw: ctest treats exit code 0 as success and any other exit code as failure, but I guess gtest produces executables that satisfy this property.
If you don't necessarily want to build the tests at the same time as the rest, you could exclude them from all, possibly adding a custom target that depends on all of the unit tests and is also excluded from all to allow building all of them at once.
You could also use a cache variable to toggle testing on and off:
# enabled via -D TEST_MY_PROJECT:BOOL=1 parameter for cmake
set(TEST_MY_PROJECT 0 CACHE BOOL "enable tests for my project")
if (TEST_MY_PROJECT)
# testing setup goes here
endif()

cmake/ctest generate Test.xml (without rebuilding?) / could not load cache

I have a cmake based (C++) project which includes tests created via add_test().
The build process is basically:
cmake
make all testsuite
make test
This has worked well for some time.
I am now trying to get my build to generate a Test.xml to be submitted as part of a pipeline build. It is not working for this project.
Q1 Can I tell cmake that I want a Test.xml to be generated when it (make test) runs the tests?
Currently I believe that a Test.xml can only be created by running ctest.
Presumably ctest aggregates the test results whereas cmake just runs them blindly?
Can someone confirm or refute this?
So I am currently trying to run ctest using:
ctest -vv -debug --output-on-failure -T test
As this does create the Test.xml file I need.
I had to add include(CTest) & include(Dart) to fix "could not find DartConfiguration.tcl".
However, even if I have previously run "make test" which guarantees the build is up to date it still tries to build and I get:
Error: could not load cache
For tests that run a compiled test (script based tests work fine).
The command it is executing is:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/test/<TestName>" --target "<test_exe>"
The CMakeCache.txt & CMakeLists.txt actually live in "<projectDir>/cmake" which is why the cache cannot be loaded.
The command should be:
/opt/cmake-3.18.1/bin/cmake "--build" "<projectDir>/cmake" --target "<test_exe>"
Q2 Is there some way to tell ctest it does not need to run cmake?
My suspicion is no. Does ctest have to invoke cmake to run the tests? Does it just monitor and aggregate the output?
If I look in DartConfiguration.tcl I have:
SourceDirectory: <projectDir/cmake
BuildDirectory: <projectDir/cmake
ConfigureCommand: "/opt/cmake-3.18.1/bin/cmake" "<projectDir/cmake"
MakeCommand: /opt/cmake-3.18.1/bin/cmake --build . --config "${CTEST_CONFIGURATION_TYPE}" -- -i
I am running ctest -T test rather than ctest -T build-and-test why is ctest trying to build at all?
I've tried setting Ctest_build_command but it seems to have no effect on the DartConfiguration.tcl generated.
Some clarifications:
ctest works for me for other projects which use out of source builds
This one cannot yet use an out of source build without massive refactoring.
The source root is actually "<projectDir>"
The CMakeLists.txt lives in "<projectDir>/cmake"
Program source code lives in "<projectDir>/src"
Test scripts and code live in "<projectDir>/test"
My build worked using:
enable_testing()
I never previously had to use:
include(CTest)
include(Dart)
One or other of these creates the DartConfiguration.tcl which is just a list of "key: value" pairs. I think this is used when ctest is invoked directly which was not the case here.
I'm not entirely clear on why and what are they actually for despite reading this.
Also posted on the cmake discourse channel - https://discourse.cmake.org/t/cmake-ctest-generate-test-xml-without-rebuilding-could-not-load-cache/2025
It turns out I had some pseudo tests used to build the test driver programs via cmake --build. These were explicitly and inexplicably using CMAKE_CURRENT_DIRECTORY instead of CMAKE_SOURCE_DIR. Oops.

Run custom shell script with CMake

I am having the following directory structure:
/CMakeLists.txt
/component-a/CMakeLists.txt
/...
/component-b/CMakeLists.txt
/...
/doc/CMakeLists.txt
/create-doc.sh
The shell script create-doc.sh creates a documentation file (doc.pdf). How can I use CMake to execute this shell script at build time and copy the file doc.pdf to the build directory?
I tried it by using add_custom_command in the CMakeLists.txt file inside the directory doc:
add_custom_command ( OUTPUT doc.pdf
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/create-doc.sh
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/)
Unfortunately the command is never run.
I also tried execute_process:
execute_process ( COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/create-doc.sh
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/ )
Now the script is executed during the configuration phase, but not at build time.
You got almost there with add_custom_command. This is indeed the correct way to tell CMake how to generate a file. However, CMake will only run that when something depends on that file.
When, as in your case, the file itself is the end product and is not used any further by subsequent build steps, the usual approach is to create a custom target to drive the relevant custom command(s):
add_custom_target(
BuildDocs ALL
DEPENDS doc.pdf
)
This (custom target driver for custom commands) is a very common idiom in CMake.
You can of course play around with arguments for add_custom_target (e.g. ALL, COMMENT) as it suits you.

CMake Header Generator Updates

In CMake I currently have a simple Python script to generate a header, but if I update the script itself CMake won't re-run the script. Is there a way I can get CMake to do this?
It seems you are directly invoking your code generation script when cmake is run. While it is possible solution but it is definitely not a right way to use code generators with cmake.
I recommend you to use add_custom_command for your case:
add_custom_command(
OUTPUT generated.h
COMMAND ${PYTHON_EXECUTABLE} generator.py
DEPENDS generator.py
)
And next you can simple put your header to the list of source files passed to add_library/add_executable commands. cmake will automatically track all the dependencies and invoke your script.
Term DEPENDS generator.py informs cmake that it should regenerate header if script is changed.
With this approach file generated.h will be generated only at build time (when you run make or execute a build command in IDE). In contrast if you are running your script at cmake time (with execute_process command) then you have to rerun cmake to regenerate your file. Which is possible but you need to use some tricks to introduce a non-standard dependency.