I have a large makefile used to control a large engineering flow.
Before the first target, I have many variables, conditional branches and functions definitions to set up the environment and control some options.
Even inside targets, I may have some parts from different logical roots.
Let's say I have something like this:
var1 := val1
var2 ?= val2
...
# fn1 defintion
...
if conditions ...
...
...
target1: depends
....
....
.....
.
My question is:
Is it possible to split the makefile into different parts at any position (i.e inside or outside targets) - Like if all included parts are "concatenated" together in a single file this file will be exactly the same as my makefile?
Something like this:
make1.mk:
var1 := val1
var2 ?= val2
...
make2.mk:
# fn1 defintion
...
make3.mk:
if conditions ...
...
...
make4.mk:
target1: depends
....
make5.mk:
....
....
makefile:
include make1 make2 make3 make4 make5 ...
Thank you!
As #Etan Reisner commented, GNU make's include directive operates conceptually similar to a compiler pre-processor, reading each included makefile and processing their content inline at the point where they were included, as if you had generated the main makefile from a template that cats each included file in place. (Unlike a pre-processor, where this actually happens, the pre-processor outputting intermediate files containing source file modifications, make just processes each included file inline in memory when it encounters them.)
So, you almost had it right in your OG psudocode, but need to include actual filenames:
include make1.mk make2.mk make3.mk make4.mk make5.mk
Better to leverage make's support of shell globbing:
include *.mk
..making sure to name the files in appropriate alphanumeric order per your shell's globbing rules so the glob expansion will result in make inserting file content in the desired sequence.
Make will also interpolate variables in the included _filename_s. Imagine the following possible use case for this - imagine that in addition to having a bunch of common makefile content you want to always include, your make system also has different operating "modes" or target categories, each requiring a different set of included files.
Let's say your make system supports some operations in your local dev environment and some in a remote cloud environment in AWS EKS. In this case you could do something like this:
include common.mk $(MAKE_MODE).mk
Like any make variable, MAKE_MODE could be specified at the command line:
make myapp MAKE_MODE=local
or maybe it makes sense for MAKE_MODE to be determined automatically, say based on which target(s) were invoked:
EFFECTIVE_MAKECMDGOALS=$(or $(MAKECMDGOALS), $(.DEFAULT_GOAL))
LOCAL_TARGETS=docker-build-push docker-run docker-shell
REMOTE_TARGETS=eks-run eks-shell
ifneq ($(filter $(LOCAL_TARGETS),$(EFFECTIVE_MAKECMDGOALS)),$())
MAKE_MODE=local
else
ifneq ($(filter $(REMOTE_TARGETS),$(EFFECTIVE_MAKECMDGOALS)),$())
MAKE_MODE=remote
endif
endif
include common.mk $(MAKE_MODE).mk
Another, more common, use-case for conditional make includes is to "flatten" make operations in large, hierarchical projects to avoid the pitfalls of recursive make, where a top-level make invokes make in project subdirectories.
Related
If I have a similar to the following structure project tree (each subproject is added via add_subdirectory() to the parent project):
CMakeLists.txt (TOP-LEVEL)
|
+ -- subproj1
| |
| + -- CMakeLists.txt (subproj1)
| |
| + -- subsubproj1
| |
| + -- CMakeLists.txt (subsubproj1)
|
+ -- subproj2
|
+ -- CMakeLists.txt (subproj2)
I want to expose a variable set inside subsubproj1 in subproj2. The nature of this variable is irrelevant but in my case it points at ${CMAKE_CURRENT_SOURCE_DIR}/include that is the include directory of subsubproj1, which (in my case) is a library used by subproj2. Currently I am re-assigning the variable in each intermediate CMakeLists.txt between the one (here subproj1) where the variable was assigned a value to the top-level with PARENT_SCOPE enabled:
CMakeLists.txt (subsubproj1)
# Expose MY_VAR to subproj1
set(MY_VAR
"Hello"
PARENT_SCOPE
)
CMakeLists.txt (subproj1)
# Expose MY_VAR to top level project thus making it visible to all
set(MY_VAR
${MY_VAR}
PARENT_SCOPE
)
This can be applied to an arbitrary nested project tree.
My question is what is the common practice of doing what I have described above? I can declare MY_VAR as a top-level variable to begin with but what if for some reason I don't want to make it visible (as in written text) there. In which case is PARENT_SCOPE no longer an option and should be replaced with just a straight declaration of that variable in the top-level CMakeLists.txt?
Targets
The nature of this variable is irrelevant but in my case it points at ${CMAKE_CURRENT_SOURCE_DIR}/include that is the include directory of subsubproj1, which (in my case) is a library used by subproj2.
No, the nature is not irrelevant.
Using variables to communicate include directories in CMake is a horrible anti-pattern from the 2.6.x days. You should not use a hammer to drive in a screw.
Non-IMPORTED projects are always global, so you can link to them safely. In subsubproj1 you would write:
add_library(myproj_subsubproj1 INTERFACE)
add_library(myproj::subsubproj1 ALIAS myproj_subsubproj1)
target_include_directories(
myproj_subsubproj1
INTERFACE
"$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>"
)
Then in subproj2, you would write:
target_link_libraries(myproj_subproj2 PRIVATE myproj::subsubproj1)
Worse options
The following options are worse because they forego the declarative parts of the CMake language and make your scripts dependent on subproject inclusion order. This is a significant complexity increase that (in my experience) is not warranted in build code.
Nevertheless, here are the imperative tools CMake provides:
1. Using a CACHE variable
The cache is a disk-persisted store for global variables. Setting one in any directory makes the value visible to all directories.
Note that there are a few potential drawbacks to this:
Prior to CMake 3.21, creating a new cache entry would delete a normal variable of the same name, leading to tricky situations where builds could become non-idempotent (bad!). See https://cmake.org/cmake/help/latest/policy/CMP0126.html
The user can overwrite your cache variables at the command line, so you cannot rely on either the defined-ness or the value of the variable when your CMake program starts running.
If you can live with this, then you can write:
# On CMake <3.21, honor normal variables. Can remove
# check if on CMake >=3.21
if (NOT DEFINED MyProj_INADVISABLE_VARIABLE)
set(MyProj_INADVISABLE_VARIABLE
"${CMAKE_CURRENT_SOURCE_DIR}/include>"
CACHE STRING "Doc string...")
# If you want to hint to your users that they should
# not edit this variable, include the following line:
mark_as_advanced(MyProj_WEIRD_VARIABLE)
endif ()
If you do not want to allow your users to override it, then you may consistently use an INTERNAL cache variable:
set(MyProj_INADVISABLE_VARIABLE "..." CACHE INTERNAL "...")
As long as you initialize it to a known value early on, then this will work okay as a global variable, but might incur disk traffic on writes.
2. Directory property
A slightly better approach is to use a custom directory property to communicate a value. In subsubproj1:
set_property(DIRECTORY "." PROPERTY inadvisable_prop "foo")
Then in subproj2:
get_property(value DIRECTORY "../subproj1/subsubproj1"
PROPERTY inadvisable_prop)
Note that it is not an error to get a non-existent property, so be on the lookout for types.
You could also use a GLOBAL property instead of a directory property, but global variables in general are a headache waiting to happen. You might as well set it on the directory to decrease the chances of unintended scoping bugs.
In our project, we have a large number of configurations stemming from a large variety of target hardware types multiplied by few modes.
To avoid unneeded details let's just assume that the configurations have form <hw>_<mode> were
<hw> is one of: A, B or C,
<mode> is one of: 1, 2 or 3.
Furthermore, to remain close to actual case let's assume that A_3 and C_1 are unsupported exceptions. (However, I don't think it matters here.)
Which leaves us with 3 x 3 - 2 = 7 supported configurations.
Now, we would like to make settings (amongst others also the path to compiler and sysroot) depend on the configuration. Also, some sources should be included only in some configurations. And we would prefer to do it based on parts of the configuration.
For example, we would like to use /this/g++ for all A_* configurations and /that/g++ for all other. Or we would like to add mode2.cpp file for all *_2 configurations but not others.
It is a simple task if we use CMAKE_BUILD_TYPE. We can split it with regex (string(REGEX MATCH) and have variables with each part. Then simple if does the job.
However, such approach is not friendly with multi-config generators (it seems currently those are only Visual Studio and Xcode). To play nicely with multi-config generators, AFAIK, we would have to use generator expressions.
The problem is, however, that I see no way to extract parts for the configuration (CONFIG) in the generator expressions.
For example, I can do this:
add_executable(my_prog
source_1.cpp
# ...
source_n.cpp
$<$<CONFIG:A_2>:mode2.cpp>
$<$<CONFIG:B_2>:mode2.cpp>
$<$<CONFIG:C_2>:mode2.cpp>
)
but this doesn't look like a maintainable approach considering that sooner or later we will be adding new hardware types (or removing obsolete ones).
Is there any way to do some form of matching in generator expression?
The only workaround I found out so far is to use an approach like this:
set(CONFIG_IS_MODE_2 $<OR:$<CONFIG:A_2>,$<CONFIG:B_2>,$<CONFIG:C_2>>)
add_executable(my_target
source_1.cpp
# ...
source_n.cpp
$<${CONFIG_IS_MODE_2}:mode2.cpp>
)
which at least allows centralizing those expressions and when new hardware type is added there is a single place to update. However, still, there are many variables to update.
Is there any better solution?
With target_sources() command and a function() you could still use a regex to match your configurations.
This would look something like in this example code:
cmake_minimum_required(VERSION 3.0)
project(TestConfigRegEx)
function(my_add_sources_by_config_regex _target _regex)
foreach(_config IN LISTS CMAKE_CONFIGURATION_TYPES CMAKE_BUILD_TYPE)
if (_config MATCHES "${_regex}")
target_sources(${_target} PRIVATE $<$<CONFIG:${_config}>:${ARGN}>)
endif()
endforeach()
endfunction()
file(WRITE main.cpp "int main() { return 0; }")
file(WRITE modeRelease.cpp "")
add_executable(my_target main.cpp)
my_add_sources_by_config_regex(my_target Release modeRelease.cpp)
But that gives me an error from CMake version 3.11.1 Visual Studio 15 2017 generator side:
Target "my_target" has source files which vary by configuration. This is
not supported by the "Visual Studio 15 2017" generator.
Config "Debug":
.../main.cpp
Config "Release":
.../main.cpp
.../modeRelease.cpp
Strange enough it still generates the solution.
Alternatives
The classic one would be adding a define containing the configuration and handle the differences in the C/C++ code with #if checks
You differentiate not per configuration but with additional targets (like my_target and my_target_2)
Imagine a code generator which reads an input file (say a UML class diagram) and produces an arbitrary number of source files which I want to be handled in my project. (to draw a simple picture let's assume the code generator just produces .cpp files).
The problem is now the number of files generated depends on the input file and thus is not known when writing the CMakeLists.txt file or even in CMakes configure step. E.g.:
>>> code-gen uml.xml
generate class1.cpp..
generate class2.cpp..
generate class3.cpp..
What's the recommended way to handle generated files in such a case? You could use FILE(GLOB.. ) to collect the file names after running code-gen the first time but this is discouraged because CMake would not know any files on the first run and later it would not recognize when the number of files changes.
I could think of some approaches but I don't know if CMake covers them, e.g.:
(somehow) define a dependency from an input file (uml.xml in my example) to a variable (list with generated file names)
in case the code generator can be convinced to tell which files it generates the output of code-gen could be used to create a list of input file names. (would lead to similar problems but at least I would not have to use GLOB which might collect old files)
just define a custom target which runs the code generator and handles the output files without CMake (don't like this option)
Update: This question targets a similar problem but just asks how to glob generated files which does not address how to re-configure when the input file changes.
Together with Tsyvarev's answer and some more googling I came up with the following CMakeList.txt which does what I want:
project(generated)
cmake_minimum_required(VERSION 3.6)
set(IN_FILE "${CMAKE_SOURCE_DIR}/input.txt")
set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${IN_FILE}")
execute_process(
COMMAND python3 "${CMAKE_SOURCE_DIR}/code-gen" "${IN_FILE}"
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
INPUT_FILE "${IN_FILE}"
OUTPUT_VARIABLE GENERATED_FILES
OUTPUT_STRIP_TRAILING_WHITESPACE
)
add_executable(generated main.cpp ${GENERATED_FILES})
It turns an input file (input.txt) into output files using code-gen and compiles them.
execute_process is being executed in the configure step and the set_property() command makes sure CMake is being re-run when the input file changes.
Note: in this example the code-generator must print a CMake-friendly list on stdout which is nice if you can modify the code generator. FILE(GLOB..) would do the trick too but this would for sure lead to problems (e.g. old generated files being compiled, too, colleagues complaining about your code etc.)
PS: I don't like to answer my own questions - If you come up with a nicer or cleaner solution in the next couple of days I'll take yours!
I need to use a property from an external project. So far I'm successfully doing that this way:
ExternalProject_Add(
some_ep_name
...
)
# Get source dir
ExternalProject_Get_Property(some_ep_name SOURCE_DIR)
# Set source dir value into less generically named variable
set(SOME_EP_NAME_SOURCE_DIR "${SOURCE_DIR}")
This works, but it seems unnecessarily verbose, and it annoys me a little. I was hoping I could use a generator expression, like so:
"$<TARGET_PROPERTY:some_ep_name,SOURCE_DIR>"
But it seems like this doesn't work. Before I give up, I wanted to check if I was doing something wrong or if anyone knows a better way.
All "properties" of ExternalProject are known at configuration time. So they don't require support of generator expressions, which main intention is usage for values not known at configuration time (but known at build time).
If you found "unnecessarily verbose" having several lines of code for save external project' property into the variable, you may create a macro/function for incorporate all these lines. Then calling the macro/function will use only single line of code:
function(ExternalProject_Property_to_Var VAR eproject prop)
ExternalProject_Get_Property(${eproject} ${eprop})
set(${VAR} ${${eprop}} PARENT_SCOPE)
endfunction()
...
ExternalProject_Property_to_Var(SOME_EP_NAME_SOURCE_DIR some_ep_name SOURCE_DIR)
The problem I'm having at the moment is that I simply wish to manage my list of source files by grabbing everything and removing the few odds and ends that I do not need. I was hoping that Cmake provided nice built-in tools for this.
So I might start with:
file(GLOB A "Application/*.cpp")
I feel like I want to create another list of files to be removed and I want to tell CMake: Remove from list A items that are in list B.
If this were Python I might do something like:
C = [f for f in A if f not in B]
I may have screwed that syntax up but I'm wondering if there is built-in support for managing these lists of files in a more elegant way?
Even if I could do something like my Python example, A is list of absolute paths so constructing B is clunky.
And why absolute paths anyway? It seems like this will break your build as soon as you relocate the source.
You can do that by using the list command with the REMOVE_ITEM option:
list(REMOVE_ITEM <list> <value> [<value> ...])
Have a look:
file(GLOB FOO *)
set (ITEMS_TO_REMOVE "item;item2;item3")
message(STATUS "FOO is ${FOO}")
list(REMOVE_ITEM FOO ${ITEMS_TO_REMOVE})
message(STATUS "FOO is now ${FOO}")
Keep in mind that the paths returned by file(GLOB) are absolute, you might want to build your list of items to remove by prepending ${CMAKE_CURRENT_LIST_DIR} to each one of them:
set (ITEMS_TO_REMOVE "${CMAKE_CURRENT_LIST_DIR}/item;
${CMAKE_CURRENT_LIST_DIR}/item2;
${CMAKE_CURRENT_LIST_DIR}/item3")
If you like Python, you can generate your list of source files with execute_process. There is also possiblity to work with lists.
But I would recommend you to "hardcode" your list of source files.
File command documentation states:
We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate.