How do I include a literal double quote in a custom CMake command? - cmake

I'm trying to create a custom command that runs with some environment variables, such as LDFLAGS, whose value needs to be quoted if it contains spaces:
LDFLAGS="-Lmydir -Lmyotherdir"
I cannot find a way to include this argument in a CMake custom command, due to CMake's escaping rules. Here's what I've tried so far:
COMMAND LDFLAGS="-Ldir -Ldir2" echo blah VERBATIM)
yields "LDFLAGS=\"-Ldir -Ldir2\"" echo blah
COMMAND LDFLAGS=\"-Ldir -Ldir2\" echo blah VERBATIM)
yields LDFLAGS=\"-Ldir -Ldir2\" echo blah
It seems I either get the whole string quoted, or the escaped quotes don't resolve when used as part of the command.
I would appreciate either a way to include the literal double-quote or as an alternative a better way to set environment variables for a command. Please note that I'm still on CMake 2.8, so I don't have the new "env" command available in 3.2.
Note that this is not a duplicate of When to quote variables? as none of those quoting methods work for this particular case.

The obvious choice - often recommended when hitting the boundaries of COMMAND especially with older versions of CMake - is to use an external script.
I just wanted to add some simple COMMAND only variations that do work and won't need a shell, but are - I have to admit - still partly platform dependent.
One example would be to put only the quoted part into a variable:
set(vars_as_string "-Ldir -Ldir2")
add_custom_target(
QuotedEnvVar
COMMAND env LD_FLAGS=${vars_as_string} | grep LD_FLAGS
)
Which actually does escape the space and not the quotes.
Another example would be to add it with escaped quotes as a "launcher" rule:
add_custom_target(
LauncherEnvVar
COMMAND env | grep LD_FLAGS
)
set_target_properties(
LauncherEnvVar
PROPERTIES RULE_LAUNCH_CUSTOM "env LD_FLAGS=\"-Ldir -Ldir2\""
)
Edit: Added examples for multiple quoted arguments without the need of escaping quotes
Another example would be to "hide some of the complexity" in a function and - if you want to add this to all your custom command calls - use the global/directory RULE_LAUNCH_CUSTOM property:
function(set_env)
get_property(_env GLOBAL PROPERTY RULE_LAUNCH_CUSTOM)
if (NOT _env)
set_property(GLOBAL PROPERTY RULE_LAUNCH_CUSTOM "env")
endif()
foreach(_arg IN LISTS ARGN)
set_property(GLOBAL APPEND_STRING PROPERTY RULE_LAUNCH_CUSTOM " ${_arg}")
endforeach()
endfunction(set_env)
set_env(LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb")
add_custom_target(
MultipleEnvVar
COMMAND env | grep -E 'LDFLAGS|CFLAGS'
)
Alternative (for CMake >= 3.0)
I think what we actually are looking for here (besides the cmake -E env ...) is named Bracket Argument and does allow any character without the need of adding backslashes:
set_property(
GLOBAL PROPERTY
RULE_LAUNCH_CUSTOM [=[env LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb"]=]
)
add_custom_target(
MultipleEnvVarNew
COMMAND env | grep -E 'LDFLAGS|CFLAGS'
)
References
0005145: Set environment variables for ADD_CUSTOM_COMMAND/ADD_CUSTOM_TARGET
How to modify environment variables passed to custom CMake target?
[CMake] How to set environment variable for custom command
cmake: when to quote variables?

You need three backslashes. I needed this recently to get a preprocessor define from PkgConfig and apply it to my C++ flags:
pkg_get_variable(SHADERDIR movit shaderdir)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DSHADERDIR=\\\"${SHADERDIR}\\\"")

Florian's answer is wrong on several counts:
Putting the quoted part in a variable makes no difference.
You should definitely use VERBATIM. It fixes platform-specific quoting bugs.
You definitely shouldn't use RULE_LAUNCH_CUSTOM for this. It isn't intended for this and only works with some generators.
You shouldn't use env as the command. It isn't available on Windows.
It turns out the real reason OPs code doesn't work is that CMake always fully quotes the first word after COMMAND because it's supposed to be the name of an executable. You simply shouldn't put environment variables first.
For example:
add_custom_command(
OUTPUT q1.txt
COMMAND ENV_VAR="a b" echo "hello" > q1.txt
VERBATIM
)
add_custom_target(q1 ALL DEPENDS q1.txt)
$ VERBOSE=1 make
...
"ENV_VAR=\"a b\"" echo hello > q1.txt
/bin/sh: ENV_VAR="a b": command not found
So how do you pass an environment variable with spaces? Simple.
add_custom_command(
OUTPUT q1.txt
COMMAND ${CMAKE_COMMAND} -E env ENV_VAR="a b" echo "hello" > q1.txt
VERBATIM
)

Ok, I removed my original answer as the one proposed by #Florian is better. There is one additional tweak needed for multiple quoted args. Consider a list of environment variables as such:
set(my_env_vars LDFLAGS="-Ldir1 -Ldir2" CFLAGS="-Idira -Idirb")
In order to produce the desired expansion, convert to string and then replace ; with a space.
set(my_env_string "${my_env_vars}") #produces LDFLAGS="...";CFLAGS="..."
string(REPLACE ";" " " my_env_string "${my_env_string}")
Then you can proceed with #Florian's brilliant answer and add the custom launch rule. If you need semicolons in your string then you'll need to convert them to something else first.
Note that in this case I didn't need to launch with env:
set_target_properties(mytarget PROPERTIES RULE_LAUNCH_CUSTOM "${my_env_string}")
This of course depends on your shell.
On second thought, my original answer is below as I also have a case where I don't have access to the target name.
set(my_env LDFLAGS=\"-Ldir -Ldir2" CFLAGS=\"-Idira -Idirb\")
add_custom_command(COMMAND sh -c "${my_env} grep LDFLAGS" VERBATIM)
This technique still requires that the semicolons from the list->string conversion be replaced.

Some folks suggest to use ${CMAKE_COMMAND} and pass your executable as an argument, e.g:
COMMAND ${CMAKE_COMMAND} -E env "$(WindowsSdkDir)/bin/x64/makecert.exe" ...
That worked for me.

Related

How to delete files stored in a cmake list in an add_custom_target?

I have files stored in a list variable MatrixSSL_configure_files.
I'd like to remove all files in a add_custom_target, like so
add_custom_target( maintainer-clean-evio
COMMAND ${CMAKE_COMMAND} -E remove -f "${MatrixSSL_configure_files}"
)
This doesn't work because the list is semi-colon separated and the
list isn't expanded (the COMMAND simply tries to execute the files).
Of course I also tried
COMMAND rm -f ${MatrixSSL_configure_files}
with the same result.
Assuming the file names contain spaces, what would be the correct
way to do this? If not possible, assume they don't contain spaces :/
This is precisely what the COMMAND_EXPAND_LISTS option is for. From the add_custom_target documentation:
COMMAND_EXPAND_LISTS
Lists in COMMAND arguments will be expanded, including those created with generator expressions, allowing COMMAND arguments such as ${CC} "-I$<JOIN:$<TARGET_PROPERTY:foo,INCLUDE_DIRECTORIES>,;-I>" foo.cc to be properly expanded.
So something like this should allow the MatrixSSL_configure_files list to be expanded:
add_custom_target( maintainer-clean-evio
COMMAND ${CMAKE_COMMAND} -E remove -f "${MatrixSSL_configure_files}"
COMMAND_EXPAND_LISTS
)

Do not expand CMake list variable

I have a CMake script that runs some tests via add_test(), running under Windows (Server 2008, don't ask) in CMake 3.15. When these tests are called, the PYTHONPATH environment variable in the environment they run in seems to get reset to the environment default, and doesn't contain some paths that it needs to.
I therefore need to set PYTHONPATH when the tests are run to the value of the $ENV{PYTHONPATH} variable when CMake runs. This has a number of semicolon-separated paths, so CMake thinks it's a list and tries to expand it into a number of space-separated strings, which obviously ends badly.
I cannot work out how to stop CMake doing this. From everything I can see, you should be able to do just surround with quotes:
add_test(
NAME mytest
COMMAND cmake -E env PYTHONPATH="$ENV{PYTHONPATH}"
run_test_here)
...but it always does the expansion. I also tried setting with set_tests_properties:
set_tests_properties(mytest PROPERTIES
ENVIRONMENT PYTHONPATH="$ENV{PYTHONPATH}")
...but that didn't appear to do anything at all - PYTHONPATH at test time wasn't altered. I thought it was because it's an environment variable, but using a regular CMake variable via set() makes no difference, so I'm doing something wrong. Help please!
The following should work:
COMMAND cmake -E env "PYTHONPATH=$ENV{PYTHONPATH}"
You need to quote the full part of the command line, to make properly expanded message.
Tested with:
set(tmp "C:\\Python27\\Scripts;E:\\JenkinsMIDEBLD\\workspace\\...;...")
add_test(NAME MYtest1 COMMAND cmake -S . -E env "tmp=${tmp}")
add_test(NAME MYtest2 COMMAND cmake -S . -E env tmp="${tmp}")
After running ctest I get:
1: Test command: /bin/cmake "-S" "." "-E" "env" "tmp=C:\Python27\Scripts;E:\JenkinsMIDEBLD\workspace\...;..."
2: Test command: /bin/cmake "-S" "." "-E" "env" "tmp="C:\Python27\Scripts" "E:\JenkinsMIDEBLD\workspace\..." "...""
The first test has proper ; passed to var, while the second one passes space separated list.
This is how cmake parses quoted arguments. An argument is either fully quoted or not quoted at all - partial quotes are interpreted as a literal ". So assumnig that:
set(var a;b;c)
The following:
var="$var"
Is not a quoted argument and " are taken literally! It expands the $var list into space separated list and the " stay, there is one " between = and a, and there is additional " on the end. The var="$var" is equal to:
var=\"a b c\"
^^ ^^ - the quotes stay!
^^^^^^^ ^ ^^^ - these are 3 arguments, the last one is `c"`
Without quotes is:
var=$var
is equal to (notice the missing quotes):
var=a c c
To quotes argument you have to quote it all, with first and last character of the element beeing ":
"var=$var"
will expand to:
"var=a;b;c"
You can make this work with the ENVIRONMENT test property, but there's a catch:
Semicolon separates different environment variables to set; you need to escape semicolons in your environment variable. For example instead of
set_tests_properties(mytest PROPERTIES
ENVIRONMENT "PYTHONPATH=foo;bar")
you need to use
set_tests_properties(mytest PROPERTIES
ENVIRONMENT "PYTHONPATH=foo\\;bar")
The fact that an environment variable may contain semicolons makes some transformation necessary: since ; is used to separate list elements you can simply use list(JOIN) to replace those with "\\;".
The following example works with PATH, not PYTHONPATH, since I don't have python installed:
CMakeLists.txt
cmake_minimum_required(VERSION 3.12.4) # required for list(JOIN)
project(TestProject)
# get a version of the PATH environment var that can be used in the ENVIRONMENT test property
set(_PATH $ENV{PATH})
list(JOIN _PATH "\\;" _PATH_CLEAN)
# just use a cmake script so we don't need to require any program able to retrieve environment vars
add_test(NAME Test1 COMMAND "${CMAKE_COMMAND}" -P "${CMAKE_CURRENT_SOURCE_DIR}/test_script.cmake")
# we add another simpler var to test in the cmake script
set_tests_properties(Test1 PROPERTIES ENVIRONMENT "PATH=${_PATH_CLEAN};FOO=foo")
enable_testing()
test_script.cmake
message(STATUS "PATH=$ENV{PATH}")
if (NOT "$ENV{FOO}" STREQUAL "foo")
# the following command results in a non-0 exit code, if executed
message(FATAL_ERROR "FOO environment var should contain \"foo\" but contains \"$ENV{FOO}\"")
endif()

cmake 3.15 adding JOB_POOL to add_custom_command SOMETIMES

For users that are using cmake 3.15 or later and are also using Ninja as a generator, I want to set the new JOB_POOL argument to some large add_custom_command() blocks. For other users, I want to keep my add_custom_command() the same (no JOB_POOL).
In earlier steps, I check the version and the generator and set ${JOB_POOLS} and I also set a variable such that users who should use a pool will see (something like):
For historical reasons, I leave this here, although #Tsyvarev
points out that this is the source of my problem!
The double-quotes are NOT wanted here!
set(USE_POOL "JOB_POOL pool_A")
Users that are not using a pool will not have that variable set.
Now how to leverage that variable in my custom command...?
1.) Generator expressions don't work, just including the text with the previous line...
add_custom_command(
...
$<USE_POOL>
)
2.) I can't seem to simply place the variable in the command, again just including the variable contents on the previous line. For example, when ${JOB_POOL} is set to the string "JOB_POOL pool_A", this code...
For historical reasons, I leave this here, although #Tsyvarev
points out that this is the source of my problem!
Don't use a STRING! No double-quotes!
add_custom_command(
OUTPUT foo
DEPENDS bar
# Comment line here...
${USE_POOL}
COMMAND
${CMAKE_COMMAND} -E ...
)
gives this error...
ninja: error: '../path/to/src/dir/JOB_POOL pool_A', needed by 'path/to/src/dir/foo', missing and no known rule to make it
It simply considers the ${JOB_POOL} string to be another dependency!
3.) I can't use the "APPEND" feature of add_custom_command(). It's just ignored...
if (${USE_POOL})
add_custom_command(
...
APPEND
JOB_POOL pool_A
)
endif()
The only thing that seems to work is to put an "if" around my
entire command, which offends my sensibility since I don't like to duplicate so much code...
if(${USE_POOL})
add_custom_command(
...many lines...
JOB_POOL pool_A
)
else()
add_custom_command(
...many lines...
)
endif()
Do you have a better idea...?
Here's a standalone example for #tsyvarev:
cmake_minimum_required(VERSION 3.15)
project foo
set_property(GLOBAL PROPERTY JOB_POOLS pool_A=2)
# For historical reasons, I leave this here, although #Tsyvarev
# points out that this is the source of my problem!
# Don't use a STRING! No double-quotes!
set(USE_POOL "JOB_POOL pool_A")
add_custom_command(
OUTPUT foo.out
DEPENDS foo.in
${USE_POOL}
COMMAND
${CMAKE_COMMAND} -E copy foo.in foo.out
COMMENT "Converting foo.in -> foo.out"
VERBATIM
)
add_custom_target(foo-out
DEPENDS foo.out
)
% cmake -GNinja .
% ninja foo-out
ninja: error: 'JOB_POOL pool_A', needed by 'foo.out', missing and no known rule to make it
It considers the string to be a dependency... If I move the USE_POOL to after the comment, it considers it part of the comment... If I move it to after the command, it considers it part of the command...
Your JOB_POOL option serves for the user's choice. You may create another variable which contains sequence of related parameters for add_custom_command:
if(JOB_POOL)
set(JOB_POOL_PARAMS JOB_POOL pool_A) # Will add 'JOB_POOL pool_A' sequence of params
else()
set(JOB_POOL_PARAMS) # Will add nothing
endif()
Then use new variable directly in add_custom_command call:
add_custom_command(
...
${JOB_POOL_PARAMS} # Will expand to additional parameters when needed
)

How to preserve single quotes in a CMake cached variable?

I have a variable
SET(CODE_COVERAGE_EXCLUSION_LIST
""
CACHE STRING "List of resources to exclude from code coverage analysis")
It must contain a list of expressions such as : 'tests/*' '/usr/*'
When trying to set the default value to the above expressions, the single quotes are removed.
How to preserve them ?
Moreover, when I try to pass the exclusion list like this
cmake -DCODE_COVERAGE_EXCLUSION_LIST="'tests/*' '/usr/*'" ..
The initial and final single quotes are lost. How to preserve them as well ?
Finally, the same question applies when using cmake-gui.
EDIT : I tried to use backslash to escape the quotes :
SET(CODE_COVERAGE_EXCLUSION_LIST
" \'tests/*\' \'/usr/*\'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
It gave me the following error :
Syntax error in cmake code at
xxx.cmake:106
when parsing string
\'tests/*\' \'/usr/*\'
Invalid escape sequence \'
EDIT2 : code of the add_custom_target (and not add_custom_command, my bad)
ADD_CUSTOM_TARGET(${_targetname}
# Cleanup lcov
${LCOV_PATH} --directory . --zerocounters
# Run tests
COMMAND ${_testrunner} ${ARGV3}
# Capturing lcov counters and generating report
COMMAND ${LCOV_PATH} --directory . --capture --output-file ${_outputname}.info
COMMAND ${LCOV_PATH} --remove ${_outputname}.info 'tests/*' '/usr/*' ${CODE_COVERAGE_EXCLUSION_LIST} --output-file ${_outputname}.info.cleaned
COMMAND ${GENHTML_PATH} -o ${_outputname} ${_outputname}.info.cleaned
COMMAND ${CMAKE_COMMAND} -E remove ${_outputname}.info ${_outputname}.info.cleaned
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Resetting code coverage counters to zero.\nProcessing code coverage counters and generating report."
)
Turning my comments into an answer
First - taking the question why you need those quotes aside - I could reproduce your problem and found several possible solutions:
Adding spaces at the begin and end of your cached variable
cmake -DCODE_COVERAGE_EXCLUSION_LIST=" 'tests/*' '/usr/*' " ..
Using "escaped" double quotes instead of single quotes
cmake -DCODE_COVERAGE_EXCLUSION_LIST:STRING="\"tests/*\" \"/usr/\"" ..
Using your set(... CACHE ...) example by setting policy CMP0053 switch introduced with CMake 3.1:
cmake_policy(SET CMP0053 NEW)
set(CODE_COVERAGE_EXCLUSION_LIST
"\'tests/*\' \'/usr/*\'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
But when setting this in the code I could also just do
set(CODE_COVERAGE_EXCLUSION_LIST
"'tests/*' '/usr/*'"
CACHE STRING "List of resources to exclude from code coverage analysis : ")
The quoting issue seems only to be a problem when called from command line
Then - if I do assume you may not need the quotes - you could pass the paths as a list:
A semicolon separated CMake list is expanded to parameters again (with spaces as delimiter) when used in a COMMAND
cmake -DCODE_COVERAGE_EXCLUSION_LIST:STRING="tests/*;/usr/*" ..
with
add_custom_target(
...
COMMAND ${LCOV_PATH} --remove ${_outputname}.info ${CODE_COVERAGE_EXCLUSION_LIST} --output-file ${_outputname}.info.cleaned
)
would give something like
.../lcov --remove output.info tests/* /usr/* --output-file output.info.cleaned
I also tried to add the VERBATIM option, because "all arguments to the commands will be escaped properly for the build tool so that the invoked command receives each argument unchanged". But in this case, it didn't change anything.
References
add_custom_target()
CMake Language: Escape Sequences
0015200: Odd quoting issue when mixing single, double and escaped quotes to COMMAND

How to pass all variables to separate CMake instance?

I would like to figure out how to efficiently pass all of the CMake variables to another execution step of CMake.
There is a way to get all the variables, but I'm hoping there is an efficient option other than looping over every variable and appending the strings together with set() as follows:
get_cmake_property(_variableNames VARIABLES)
foreach (_variableName ${_variableNames})
if(NOT ${_variablename} STREQUAL BASIS_PROPERTIES_ON_TESTS_RE)
set(ALL_VARIABLES_COMMAND_LINE "${ALL_VARIABLES_COMMAND_LINE} -D ${_variableName}=\"${${_variableName}}\"\n")
endif()
endforeach()
execute_process (
COMMAND "${CMAKE_COMMAND}" ${COMMON_ARGS}
-D "PROJECT_INCLUDE_DIRS=${INCLUDE_DIRS}"
-D "BINARY_INCLUDE_DIR=${BINARY_INCLUDE_DIR}"
-D "EXTENSIONS=${EXTENSIONS}"
${ALL_VARIABLES_COMMAND_LINE}
-D "CMAKE_FILE=${CMAKE_FILE}"
-P "${BASIS_MODULE_PATH}/ConfigureIncludeFiles.cmake"
RESULT_VARIABLE RT
)
The problem with that method is that it will mess with escape characters and in some cases fail to execute the program.
Note: I currently write all the variables to disk and load them back in from there, but that operation takes 1/2 second. Since I need to run this script for over 100 independent packages the additional configuration time for that technique is too high.
Well, instead of passing the whole list of variables as command-line parameters, I would form a new (temporary) CMake script (with file(WRITE...)) which sets all required variables and then execute that script with another instance of CMAke (execute_process(COMMAND ${CMAKE_COMMAND} -P /path/to/script...)). Thus you'd avoid the necessity to fight against the bash escaping.
Certainly when writing the file you would need to escape variables values too, but in case of of CMake it's a much simpler task. It seems that the basic escaping should be as follows:
set(VAR "A String with \" and \\ ")
message(STATUS "VAR=[${VAR}]")
string(REPLACE "\\" "\\\\" VAR_ ${VAR}) # escape \
string(REPLACE "\"" "\\\"" VAR_ ${VAR_}) # escape "
file(WRITE file.cmake "SET(VAR \"${VAR_}\")\n" )
file(APPEND file.cmake "message(STATUS \"VAR=[\${VAR}]\")\n")
this cmake script creates another cmake script named file.cmake which prints the same variable as the first script.
I am assuming that you are using the ConfigureIncludeFiles module from the CMake BASIS project. One way to improve configuration time for your project is to run ConfigureIncludeFiles as an include script.
Instead of running ConfigureIncludeFiles in a CMake subprocess started with execute_process, run the the module in the main CMake process as a CMake script with the include command for each package that needs to be configured, i.e.:
# set up parameters for package 1
set (PROJECT_INCLUDE_DIRS ${PACKAGE1_INCLUDE_DIRS})
set (BINARY_INCLUDE_DIR ${PACKAGE1_BINARY_INCLUDE_DIR})
set (EXTENSIONS ${PACKAGE1_EXTENSIONS})
set (CMAKE_FILE ${PACKAGE1_CMAKE_FILE})
include(ConfigureIncludeFiles)
...
# set up parameters for package 2
set (PROJECT_INCLUDE_DIRS ${PACKAGE2_INCLUDE_DIRS})
set (BINARY_INCLUDE_DIR ${PACKAGE2_BINARY_INCLUDE_DIR})
set (EXTENSIONS ${PACKAGE2_EXTENSIONS})
set (CMAKE_FILE ${PACKAGE2_CMAKE_FILE})
include(ConfigureIncludeFiles)
Because the module runs in the main CMake process, it will implicitly have access to all CMake variables defined and there is no need to pass the variables explicitly.