When running cmake on a large project, can it call a given cmake function in parallel?
What I am concerned about is that when this (user defined) function stores something in the cache then that would collide with the parallel execution; how does cmake deal with this when functions are indeed executed in parallel, or do I have to make sure that even invocations of the same function will never use the same (cached) variable name with different values?
EDIT (see comments):
I wrote the following function,
function(CW_SYS_CACHELINE_SIZE)
if (NOT DEFINED CACHE{cw_cv_sys_cacheline_size})
set(CMAKE_TRY_COMPILE_CONFIGURATION "Release")
try_run(RUN_WORKS
COMPILE_WORKS
${CMAKE_CURRENT_BINARY_DIR}/cw_utils_sys_cacheline_size
${CW_SYS_CACHELINE_SIZE_MODULE_PATH}/CW_SYS_CACHELINE_SIZE.c
COMPILE_OUTPUT_VARIABLE COMPILE_OUTPUT
RUN_OUTPUT_VARIABLE RUN_OUTPUT)
if (NOT COMPILE_WORKS)
message(FATAL_ERROR "Failed to compile test program
CW_SYS_CACHELINE_SIZE.c: ${COMPILE_OUTPUT}")
elseif (NOT RUN_WORKS EQUAL 0)
message(FATAL_ERROR "Failed to run test program CW_SYS_CACHELINE_SIZE.c: ${RUN_OUTPUT}")
else ()
set(cw_cv_sys_cacheline_size ${RUN_OUTPUT} CACHE INTERNAL "")
endif ()
endif()
endfunction()
Since this function doesn't take arguments, the internal values of the used variables are expected to be the same every time; but I have other very similar functions that do almost the exact same thing, except that they compile and test a different .c file.
So, each of those functions can have a different value of RUN_WORKS, which is (as it turns out) written to the cache as INTERNAL. Suppose that I am not terminating the program but doing something else with the value of RUN_WORKS here. Then it could be that I first run this function where RUN_WORKS is true, and then run the other function which sets RUN_WORKS to false.
If then I run cmake again (or inadvertently call the first function again),
then it has no way of knowing if the current value of RUN_WORKS in the cache has anything to do with a previous run of THIS function (and/or with the same arguments, if it has some). So, it CANNOT use the value of the cached variable. BUT - cmake is caching it. This worries me; why would it be caching anything if it isn't planning to reuse its value later? I'd feel a lot better if it didn't cache these variables.
So, my reasoning was: if anything is being cached (INTERNAL or not) I must make sure it is 100% unique (aka, that cmake will always write the same value to the same variable, or it wouldn't have made sense to cache it in the first place). But how to do that with this RUN_WORKS variable? I'm just horribly confused about how to write safe code like this :/
... can it call a given CMake function in parallel?
No, configuration stage of CMake, when it processes CMakeLists.txt and, among other things, executes its functions, is performed strictly sequentially.
Only build stage, when libraries and executables are compiled and COMMAND's (in add_custom_command/add_custom_target) are executed can be performed in parallel.
Moreover, during the build stage CMake isn't running at all: this stage is controlled by the build tool, selected as a generator for CMake.
Related
I would like to compile a library as shared or static based on other variable eg:
add_library(MyLibrary $<$<IF:${BUILD_SHARED_LIBS},SHARED,STATIC> ${SOURCES})
For clarity I expect this to be equivalent with the following:
if(BUILD_SHARED_LIBS)
add_library(MyLibrary SHARED ${SOURCES})
elseif()
add_library(MyLibrary STATIC ${SOURCES})
endif()
AFAIK, you cannot do that with generator expressions (no way to query that aspect according to doc), since BUILD_SHARED_LIBS is there for exactly that reason; to allow you to select its value during configuration (using the -D command line option). This will work only if you do not set the library type explicitly as in your code, but like this
add_library(MyLibrary ${SOURCES})
Actually, that is the recommended practice. If you need to influence its value in association with some other condition, you can override it with the usual if()/else() logic, making sure that you print at least an informative message() for the user.
However, an even better approach would be to push those decisions to the user (via options) and check for the illegal combinations, issuing a message(FATAL_ERROR). Even if that condition is determined automatically, this is still a tactic that has merit.
Say I have an option "ENABLE_Foo" in CMake:
option(ENABLE_Foo "Tell cmake to enable the package Foo" ON)
I want to detect whether the user specified -DENABLE_Foo=ON or -DENABLE_Foo=OFF, or whether the specified initial value ON is being used.
The reason I want to tell whether the user turned the option on or whether it is turned on by default is because Foo requires an additional package and I want to cover these cases:
1) User did not specify a value for the option ENABLE_Foo:
a) Package Foo is found -> use Foo
b) Package Foo is not found -> silently turn off Foo
2) User specified a value for the option ENABLE_Foo:
a) User said -DENABLE_Foo=ON:
i) Package Foo is found -> use Foo
ii) Package Foo is not found -> fatal error message
b) User said -DENABLE_Foo=OFF -> don't use Foo and don't try to find it
If there is no way to tell whether the option value came from user input or from the initial value, are there other ways to achieve what I have outlined above?
If the user gives -DENABLE_Foo=ON on the command line, an entry for the respective option will be added to the CMake cache. While it is possible to inspect this value before invoking the option command, you cannot distinguish whether the value was originally set by the user on the command line, or by the option command itself on a previous CMake run.
Still, achieving the behavior you described is possible.
The main issue is that you cannot model the configuration options you want with just one flag. You are actually dealing with two different options.
The first is whether support for Foo is desired at all, the second is whether Foo is to be considered an optional or a mandatory dependency. Note that the value of the latter is irrelevant in case support for Foo is disabled. One way to handle this would be to remove the option completely in this case. This allows for the following approach:
if(REQUIRE_Foo)
# REQUIRE_Foo implies ENABLE_Foo
unset(ENABLE_Foo CACHE)
endif()
option(ENABLE_Foo "Tell cmake to enable support for package Foo" ON)
if(ENABLE_Foo)
option(REQUIRE_Foo "Abort the build if Foo is not found." OFF)
find_package(Foo) # assuming that FindFoo is a well-behaving find script,
# which will set Foo_FOUND appropriately
else()
unset(REQUIRE_Foo CACHE)
set(Foo_FOUND FALSE)
endif()
if(REQUIRE_Foo AND NOT Foo_FOUND)
message(FATAL_ERROR "Required library Foo could not be found")
endif()
In case the user wants to require Foo support (your case 2a) they would invoke CMake with -DREQUIRE_Foo=TRUE. In case they want to disable Foo completely (your case 2b) they would use -DENABLE_Foo=FALSE. If the user specifies nothing, you get the behavior from your case 1.
Assuming that the rest of the build system is already prepared to handle the case where Foo was not found (case 1b from your question), this should just work without further changes.
Note that even if you could detect whether the flag was set by the user, it would still be undesirable to implement the original behavior from your question.
Assuming we could detect it, the initial value of REQUIRE_Foo in the above code would be set to true if and only if ENABLE_Foo=ON was set on the command line. Doing this implicitly without also adding the REQUIRE_Foo to the set of configuration options is problematic. A user would experience different behaviors on CMake runs even though the build options are the same. You should avoid magical behavior like this. It will only confuse users and give you a hard time debugging failing CMake runs.
Always make build options that depend on configuration given by the user visible to the user.
I have a CMake setup where the accessibilty of one variable will depend whether another one is set or not. Small snippet:
option(build-compiler "Build the Nap Compiler" ON)
set(include_interrupt_dirs CACHE INTERNAL "interrupts/intr_4" FORCE)
if(build-compiler)
option(enable-runtime-compilation
"Build in the runtime code compilation link in intr_2 & intr_3)" ON)
if(enable-runtime-compilation)
list(APPEND include_interrupt_dirs "interrupts/intr_2" "interrupts/intr_3" )
endif()
endif()
I use cmake-gui for configuring the project, and what I would like to achieve is:
if the user selects the build-compiler the enable-runtime-compilation should also be presented. This part is done.
if the user deselects the build-compiler the enable-runtime-compilation should be hidden from the GUI. This is not working.
Do you have any idea how to make it work?
Using unset(var [CACHE]) is subtly tricky. If you just unset the variable, it will remain in the cache (although it will not be visible to the script, it will still be visible to the user). If you also delete it from the cache, then you lose the value that was there.
In my use case, I wanted to hide variables based on some condition. I found it that deleting the variables from the cache might get confusing as when reinstated, they would return to their default state rather than to what the user might have set before.
I prefer to hide the variables using mark_as_advanced(FORCE var) and unhide using mark_as_advanced(CLEAR var). It does exactly what you need-it hides the variable from the GUI but it still exists in the cache. You might use this together with the "soft" unset (the one without CACHE) to make sure that the hidden variable is not still being used in the configuration.
Additionally, there is CMakeDependentOption which is intended specifically for this use case (an option that is only available if some set of condition evaluates as true). This is apparently available since CMake 3.0.2.
You can use unset(var CACHE) to remove the variable from the cache:
if(build-compiler)
option(enable-runtime-compilation
"Build in the runtime code compilation link in intr_2 & intr_3)" ON)
if(enable-runtime-compilation)
list(APPEND include_interrupt_dirs "interrupts/intr_2" "interrupts/intr_3" )
endif()
else()
unset(enable-runtime-compilation CACHE)
endif()
Assume I have a compilation unit consisting of three functions, A, B, and C. A is invoked once from a function external to the compilation unit (e.g. it's an entry point or callback); B is invoked many times by A (e.g. it's invoked in a tight loop); C is invoked once by each invocation of B (e.g. it's a library function).
The entire path through A (passing through B and C) is performance-critical, though the performance of A itself is non-critical (as most time is spent in B and C).
What is the minimal set of functions which one should annotate with __attribute__ ((hot)) to effect more aggressive optimization of this path? Assume we cannot use -fprofile-generate.
Equivalently: Does __attribute__ ((hot)) mean "optimize the body of this function", "optimize calls to this function", "optimize all descendant calls this function makes", or some combination thereof?
The GCC info page does not clearly address these questions.
Official documentation:
hot
The hot attribute on a function is used to inform the compiler that the function is a hot spot of the compiled program. The function is optimized more aggressively and on many target it is placed into special subsection of the text section so all hot functions appears close together improving locality.
When profile feedback is available, via -fprofile-use, hot functions are automatically detected and this attribute is ignored.
The hot attribute on functions is not implemented in GCC versions earlier than 4.3.
The hot attribute on a label is used to inform the compiler that path following the label are more likely than paths that are not so annotated. This attribute is used in cases where __builtin_expect cannot be used, for instance with computed goto or asm goto.
The hot attribute on labels is not implemented in GCC versions earlier than 4.8.
2007:
__attribute__((hot))
Hint that the marked function is "hot" and should be optimized more aggresively and/or placed near other "hot" functions (for cache locality).
Gilad Ben-Yossef:
As their name suggests, these function attributes are used to hint the compiler that the corresponding functions are called often in your code (hot) or seldom called (cold).
The compiler can then order the code in branches, such as if statements, to favour branches that call these hot functions and disfavour functions cold functions, under the assumption that it is more likely that that the branch that will be taken will call a hot function and less likely to call a cold one.
In addition, the compiler can choose to group together functions marked as hot in a special section in the generated binary, on the premise that since data and instruction caches work based on locality, or the relative distance of related code and data, putting all the often called function together will result in better caching of their code for the entire application.
Good candidates for the hot attribute are core functions which are called very often in your code base. Good candidates for the cold attribute are internal error handling functions which are called only in case of errors.
So, according to these sources, __attribute__ ((hot)) means:
optimize calls to this function
optimize the body of this function
put body of this function to .hot section (to group all hot code in one location)
After source code analysis we can say that "hot" attribute is checked with (lookup_attribute ("hot", DECL_ATTRIBUTES (current_function_decl)); and when it is true, the functions's node->frequency is set to NODE_FREQUENCY_HOT (predict.c, compute_function_frequency()).
If the function has frequency as NODE_FREQUENCY_HOT,
If there is no profile information and no likely/unlikely on branches, maybe_hot_frequency_p will return true for the function (== "...frequency FREQ is considered to be hot."). This turns value of maybe_hot_bb_p into true for all Basic Blocks (BB) in the function ("BB can be CPU intensive and should be optimized for maximal performance.") and maybe_hot_edge_p true for all edges in function. In turn in non -Os-modes these BB and edges and also loops will be optimized for speed, not for size.
For all outbound call edges from this function, cgraph_maybe_hot_edge_p will return true ("Return true if the call can be hot."). This flag is used in IPA (ipa-inline.c, ipa-cp.c, ipa-inline-analysis.c) and influence inline and cloning decisions
I have the following two files:
a.tcl:
set condition false
source b.tcl
b.tcl:
if {$condition} {
puts "hello"
}
When I run a.tcl, it prints "hello". Is this a correct practice for accessing variable defined in a.tcl? What is the scope of $condition in b.tcl? Thank you.
The scope of condition is global. The source command evaluates the script read from the specified file in the context it's run; in your case this context is also global, hence your puts works.
The question about practice is more complicated as it hightly depends on what you actually do.
The way the source command works is pretty much exactly as if it was reading the file into a string and then passing that to eval (the sole subtlety is to do with info script). That means that the scope that the source was done in will be the one that the outermost level of the script is evaluated in, and so that you could have condition be a local variable there:
proc funkystuff {condition} {
source b.tcl
}
funkystuff true
That will work (and is in fact vital for how Tcl's package definition scripts work; they're evaluated in a context where there is a local variable $dir that describes where the package definition is located) but it can most certainly lead to code that is confusing!
Because of this, it's good practice to write your scripts so that the code inside them makes no assumptions about what context it is evaluated in. The easiest way to do that is often to put the code in the script inside a namespace, where the name of the namespace is fully qualified.
namespace eval ::foobar {
# Do stuff here...
}
It's also a good thing to try to write code that isn't excessively parameterized on sourcing, instead saving that for either which version of the code you load (e.g., one file for Linux, another for Windows) or what parameters you pass to the commands. Of course you don't have to work that way, but it does help make your code robust and easy to understand.
Finally, the scope used for the main script to a Tcl interpreter is always evaluated at the global level (i.e., in the :: namespace with no parent scope).