CMake invocation of GLSLC with respect to includes/dependencies - cmake

I'm using glslc to compile GLSL shaders with #includes (not part of the core spec IIRC but supported in shaderc, which is the engine behind glslc, distributed with the LunarG Vulkan SDK) into SPIR-V for Vulkan and GL 4.5. glslc emits gcc-style depsfiles ([my_shader].[ext].d) files containing dependency info.
My project is built with cmake/ninja/MSVC 2017.
Today, I use a cmake custom_command to invoke glslc when a shader has changed on disk, as a post-build step to my primary target. However, this doesn't catch the changes in included files (isn't at all aware of the .d files or their contents), so rebuilding shaders when an included glsl file is changed can trip up myself and other people on my team.
It looks like ninja can invoke arbitrary compilers, and since ninja knows how to handle the depsfiles, I should be able to coerce ninja into running glslc -- unsure about other build systems since right now we're standardized on ninja.
So how can I tell cmake to configure ninja to use glslc for a specific target? Or is there a paradigmatic way to get this done? It looks like a cmake pull request to add support for glslc as a compiler didn't make it in to cmake circa 2016, so whatever I do is going to be a workaround.

CMake understands depfiles when used in conjunction with ninja.
DEPFILE
Specify a .d depfile for the Ninja generator. A .d file holds dependencies usually emitted by the custom command itself. Using DEPFILE with other generators than Ninja is an error.
add_custom_command(
OUTPUT ${source}.h
DEPENDS ${source}
COMMAND
glslc
-MD -MF ${source}.d
-o ${source}.h -mfmt=num
--target-env=opengl
${CMAKE_CURRENT_SOURCE_DIR}/${source}
DEPFILE ${source}.d
)
-M Generate make dependencies. Implies -E and -w.
-MM An alias for -M.
-MD Generate make dependencies and compile.
-MF <file> Write dependency output to the given file.
-MT <target> Specify the target of the rule emitted by dependency
generation.
EDIT: Getting fancier
find_package(Vulkan COMPONENTS glslc)
find_program(glslc_executable NAMES glslc HINTS Vulkan::glslc)
function(compile_shader target)
cmake_parse_arguments(PARSE_ARGV 1 arg "" "ENV;FORMAT" "SOURCES")
foreach(source ${arg_SOURCES})
add_custom_command(
OUTPUT ${source}.${arg_FORMAT}
DEPENDS ${source}
DEPFILE ${source}.d
COMMAND
${glslc_executable}
$<$<BOOL:${arg_ENV}>:--target-env=${arg_ENV}>
$<$<BOOL:${arg_FORMAT}>:-mfmt=${arg_FORMAT}>
-MD -MF ${source}.d
-o ${source}.${arg_FORMAT}
${CMAKE_CURRENT_SOURCE_DIR}/${source}
)
target_sources(${target} PRIVATE ${source}.${arg_FORMAT})
endforeach()
endfunction()
add_executable(dummy dummy.c)
compile_shader(dummy
ENV opengl
FORMAT num
SOURCES
dummy.vert
dummy.frag
)

Related

generating debug info with emscripten / ninja / cmake on complex project

I am trying to debug a port of some c and c++ code to WASM. I worked out how to source level debug in the browser with a simple 10 line .c program but now I want to make that work with a non trivial code base. (mixed c and c++). The wasm code works in a simple app but not in my more complex use case, hence the need to debug it
I use CMake to generate ninja build files
Here is where I am setting flags in my CMakeLists.txt
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} \
-s MODULARIZE=1 \
-s SINGLE_FILE=1 \
-s EXPORT_NAME=aubio \
-g --bind")
this was basically copied from the original codebase (the -g instead of -Oz is mine), I am not a CMake nor ninja nor emscripten wizard. When I build this I can see that the CXX flags are not passed to the emc++ compile passes, only to the 'linker' phase
I am not even sure where the -g (perhaps with source-map) needs to be for a multi-file project 'linked' into a single file. Should it be on the compile passes or the link pass , or maybe both. But certainly at the moment I do not get any symbols anywhere, no 'map' file(s) (embedded DWARF?, since the browser plugin claims to support it)
set(CMAKE_BUILD_TYPE Debug)
seems to embed DWARF to WASM file.
I don't know why set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g") nor target_compile_options(foo PUBLIC -g) doesn't work in Release build.

Configure cmake to work with homebrew libraries instead system-provided libraries

I find myself going against the grain configuring cmake paths with ccmake over and over again as with every change of for ex. compiler some of my library paths get lost.
In particular paths to (unlinked) lapack, lapacke, gsl get either lost or set to system defaults instead the ones I've installed with brew.
There has to be a way to tell cmake to "ignore" system libraries and instead look in homebrew paths (say. /opt/homebrew/lib, /opt/homebrew/include etc.).
I'd prefer not to link those libraries as this is not recommend and I'm not experienced in switching environments.
[EDIT] MRE:
git clone https://gitlab.physik.uni-muenchen.de/AG-Scrinzi/tRecX.git
cd tRecX
cmake . -DCMAKE_BUILD_TYPE=Parallel
make -j 8
I add the following to .bash_profile/.zshrc:
export LDFLAGS="-L/opt/homebrew/opt/lapack/lib -L/opt/homebrew/opt/lapack/lib"
export CPPFLAGS="-I/opt/homebrew/opt/lapack/include -I/opt/homebrew/opt/openblas/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/lapack/lib/pkgconfig /opt/homebrew/opt/openblas/lib/pkgconfig"
then I try:
cmake . -DCMAKE_PREFIX_PATH=/opt/homebrew -DCMAKE_FIND_FRAMEWORK=NEVER -DCMAKE_FIND_APPBUNDLE=NEVER -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE -DCMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH=FALSE -DMPI_CXX_COMPILER=/opt/homebrew/bin/mpicxx -DMPI_C_COMPILER=/opt/homebrew/bin/mpicc -DCMAKE_CXX_COMPILER=/opt/homebrew/bin/g++-11 -DCMAKE_C_COMPILER=/opt/homebrew/bin/gcc-11
The most common solution is to just set CMAKE_PREFIX_PATH to /opt/homebrew. CMake will then look preferentially in /opt/homebrew for everything. Since you're on Apple, you might need to set CMAKE_FIND_FRAMEWORK and CMAKE_FIND_APPBUNDLE to LAST or NEVER, too.
You can skip the standard platform search paths by setting CMAKE_FIND_USE_CMAKE_SYSTEM_PATH to FALSE at the command line, in a preset, or in a toolchain file. You might also wish to disable looking at the PATH environment variable by setting CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH to FALSE.
Finally, if you're in a cross-compiling scenario or toolchain file, you can change the definition of the system directories by setting CMAKE_SYSROOT. Note that the sysroot will have to contain the language runtime libraries (e.g. glibc) and will be passed to the --sysroot flag (or equivalent). Just be aware of those effects, too.
All of this is documented here:
https://cmake.org/cmake/help/latest/command/find_package.html#search-procedure
https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_FRAMEWORK.html#variable:CMAKE_FIND_FRAMEWORK
https://cmake.org/cmake/help/latest/variable/CMAKE_FIND_APPBUNDLE.html#variable:CMAKE_FIND_APPBUNDLE
The following homebrew.cmake toolchain file worked for me:
set(HOMEBREW_PREFIX "/usr/local"
CACHE PATH "Path to Homebrew installation")
set(CMAKE_C_COMPILER "${HOMEBREW_PREFIX}/bin/gcc-11")
set(CMAKE_CXX_COMPILER "${HOMEBREW_PREFIX}/bin/g++-11")
set(CMAKE_PREFIX_PATH
"${HOMEBREW_PREFIX}"
# These libraries are keg-only and not loaded into
# the root prefix by default (to avoid clashes).
"${HOMEBREW_PREFIX}/opt/lapack"
"${HOMEBREW_PREFIX}/opt/openblas"
"${HOMEBREW_PREFIX}/opt/gcc/lib/gcc/11"
)
list(TRANSFORM CMAKE_PREFIX_PATH APPEND "/include"
OUTPUT_VARIABLE CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES)
set(CMAKE_C_STANDARD_INCLUDE_DIRECTORIES "${CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES}")
set(CMAKE_FIND_FRAMEWORK NEVER)
set(CMAKE_FIND_APPBUNDLE NEVER)
set(CMAKE_FIND_USE_CMAKE_SYSTEM_PATH FALSE)
set(CMAKE_FIND_USE_SYSTEM_ENVIRONMENT_PATH FALSE)
I built with the following commands:
$ ls
tRecX homebrew.cmake
$ cmake -G Ninja -S tRecX -B tRecX-build \
-DCMAKE_TOOLCHAIN_FILE=$PWD/homebrew.cmake \
-DCBLAS=/usr/local/opt/openblas/lib/libblas.dylib \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_BUILD_TYPE=Parallel
[ ... output clipped ... ]
Boost found -- full functionality
Build "Parallel" with C++ flags -D_USE_BOOST_ -O3 -pthread -D_USE_FFTW_, return to default by -UCMAKE_BUILD_TYPE
Compiler: /usr/local/bin/g++-11, change by -DCMAKE_CXX_COMPILER=[path_to_complier]
-- Linking to libraries Boost::system;Boost::filesystem;/usr/local/lib/libfftw3.dylib;/usr/local/opt/gcc/lib/gcc/11/libgfortran.dylib;alglib;/usr/local/lib/libarpack.dylib;Boost::system;Boost::filesystem;/usr/local/opt/lapack/lib/liblapacke.dylib;/usr/local/opt/openblas/lib/libblas.dylib;/usr/local/opt/lapack/lib/liblapack.dylib;/usr/local/opt/lapack/lib/libblas.dylib;m
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/alexreinking/Development/tRecX-build
$ cmake --build tRecX-build
I had to set CBLAS manually because libblas.dylib provides the OpenBLAS CBLAS interface, but the build system specifically looks for a library named libcblas. There's no other option in this case.
The code and build have issues with its linking model and dependencies. I was able to paper over these by setting -Wl,-undefined,dynamic_lookup. However, note that this will just defer linker errors to runtime and might impose a large startup cost.
If you can make commits to the project, I would store these settings in a preset, maybe name it homebrew-parallel or something:
-DCMAKE_TOOLCHAIN_FILE=$PWD/homebrew.cmake \
-DCBLAS=/usr/local/opt/openblas/lib/libblas.dylib \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-undefined,dynamic_lookup" \
-DCMAKE_BUILD_TYPE=Parallel
Then you could just run cmake --preset=homebrew-parallel

CMake get all flags, includes and defines programatically

I'm using CMake 3.12 with linux and I try to extract all flags, includes and defines programatically from a target.
I found some help online, but it was always limited to the default settings induced by CMake.
For example with the variables :
CMAKE_CXX_FLAGS
CMAKE_CXX_FLAGS_DEBUG
CMAKE_CXX_FLAGS_RELEASE
...
Here I'm missing external flags.
I' stumbled upon the file "flags.make" located in the build folder of the project:
# CMAKE generated file: DO NOT EDIT!
# Generated by "Unix Makefiles" Generator, CMake Version 3.12
# compile C with /usr/bin/gcc-4.8
C_FLAGS = -O3 -DNDEBUG -fPIC
C_DEFINES = -DLINUX -DLINUX64 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
C_INCLUDES =
Which is exactly what I need. Does anybody know how to access these variables within CMake?
Thank you!
Best regards
Fabian

How do I use CMake ExternalProject_Add or alternatives in a cross-platform way?

I would like to build a third-party project that already has CMake as part of my project's CMake strips. ExternalProject_Add is for this purpose, but I have found it can only be made to work with a specific generator, and I wanted it to work on many platforms easily.
For example, here is my external project with an added script for zlib, which has its own CMakeLists.txt:
set(USE_PROJECT_CMAKE_MODULE_PATH "-DCMAKE_MODULE_PATH=${MAKE_MODULE_PATH}")
ExternalProject_Add(ZLIB
SOURCE_DIR ${CMAKE_SOURCE_DIR}/external/zlib
DOWNLOAD_COMMAND ""
UPDATE_COMMAND ""
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
${USE_PROJECT_CMAKE_MODULE_PATH}
INSTALL_COMMAND "")
ExternalProject_Add_Step(ZLIB installInternally
COMMAND cd <BINARY_DIR> && make install
DEPENDEES install
ALWAYS 1)
ExternalProject_Get_Property(ZLIB install_dir)
if(UNIX)
set(ZLIB_NAME libz)
else(UNIX)
set(ZLIB_NAME zlib)
endif(UNIX)
add_library(zlib UNKNOWN IMPORTED)
set_property(TARGET zlib PROPERTY IMPORTED_LOCATION ${install_dir}/lib/${ZLIB_NAME}.a)
set(ZLIB_LIBRARIES zlib)
set(ZLIB_LIBRARIES_OPTIONAL ${ZLIB_LIBRARIES})
set(ZLIB_DIR ${install_dir} CACHE INTERNAL "zlib ROOT dir")
set(ZLIB_INCLUDE_DIRS ${install_dir}/include CACHE INTERNAL "zlib include dirs")
set(ZLIB_DEFINES "-msse2 -mfpmath=sse" CACHE INTERNAL "zlib defines")
The problem with this is that it works with make, but not with Xcode or Visual Studio. Perhaps there is some way to take the CMake build commands passed to my project and forward them to ExternalProject_Add.
How can I write ExternalProject_Add calls in a cross-platform way with minimal code complexity, or is there a better alternative?
Problems
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
This is enough for single-configuration projects. But for Xcode and Visual Studio, you need to set CMAKE_CONFIGURATION_TYPES plus call build . --config at the build stage. See my answer.
COMMAND cd <BINARY_DIR> && make install
This will work only for Makefile generators of course. To be cross-platform you can use:
--build . --target install --config inside INSTALL_COMMAND of ExternalProject_Add.
Take a look at this template file, and in particular the following lines:
ExternalProject_Add(
"${current_project}"
URL
#HUNTER_PACKAGE_URL#
URL_HASH
SHA1=#HUNTER_PACKAGE_SHA1#
DOWNLOAD_DIR
"#HUNTER_PACKAGE_DOWNLOAD_DIR#"
SOURCE_DIR
"#HUNTER_PACKAGE_SOURCE_DIR#"
INSTALL_DIR
"#HUNTER_PACKAGE_INSTALL_PREFIX#"
# Not used, just avoid creating Install/<name> empty directory
BUILD_COMMAND ""
# This command is empty because all necessary targets will
# be built on install stage
CMAKE_ARGS
"-G#CMAKE_GENERATOR#"
"-C#HUNTER_CACHE_FILE#"
"-C#HUNTER_ARGS_FILE#"
"-D${postfix_name}=${${postfix_name}}"
"-DCMAKE_BUILD_TYPE=${configuration}"
"-DCMAKE_CONFIGURATION_TYPES=${configuration}"
"-DCMAKE_INSTALL_PREFIX=#HUNTER_PACKAGE_INSTALL_PREFIX#"
"-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
INSTALL_COMMAND
"#CMAKE_COMMAND#"
--build .
--target install
--config ${configuration}
--
${jobs_option}
)
Alternative
or is there a better alternative?
Have you seen Hunter?
You can add zlib just like this:
hunter_add_package(ZLIB)
find_package(ZLIB CONFIG REQUIRED)
target_link_libraries(... ZLIB::zlib)
This code works everywhere. Third party dependencies will be downloaded automatically in the configuration step. Example of building with different generator/toolchains (build.py is just a CMake wrapper that sets CMAKE_TOOLCHAIN_FILE and -G/-B):
build.py --toolchain mingw --config Release # MinGW Makefiles
build.py --toolchain vs-12-2013 --config Debug # Visual Studio 12 2013
build.py --toolchain xcode --config Release # Xcode
build.py --toolchain libcxx --config Release # Makefile with -stdlib=libc++ toolchain
build.py --toolchain ios-8-2 --config Release # Xcode with iOS SDK 8.2 toolchain
You got full control what options, build types or number of jobs you want to have while building third-party packages. For instance, this is how you can build four types, Debug, Release, MinSizeRel, and RelWithDebInfo for zlib and link MinSizeRel to the current project:
> build.py --toolchain xcode --verbose --config MinSizeRel --fwd "HUNTER_CONFIGURATION_TYPES=Release;Debug;MinSizeRel;RelWithDebInfo"
/.../clang /.../lib/libz-MinSizeRel.a ... -o /.../_builds/xcode/MinSizeRel/foo
> ls -la /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz*
99056 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-MinSizeRel.a
307872 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-RelWithDebInfo.a
109536 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz.a
258904 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libzd.a
CMake ExternalProject_Add calls work cross-platform by default and will only fail to do so if one uses particular commands that are only available on a subset of operating systems.
Typically, CMAKE_ARGS is used to pass information to each superbuild unit within an external project build. The CMakeLists.txt files that control each miniature part of the overall build use CMake's declarative syntax (e.g., "add_library(library_name SHARED filename1.hpp filename1.cpp). CMake will convert such syntax to the commands that are specific to the particular build system you wish to use (e.g., make and Ninja).
The sample above re: zlib fails to be cross-platform in part because the ExternalProject_Add_Step contains "COMMAND cd && make install", which necessarily only works in situations where invoking "cd" is actually the correct way to change directories, and where invoking "make" is actually the correct way to build software.
CMake's -E option provides a way to invoke basic operations like changing/copying/making/removing directories without making such assumptions.
(By the way, if you're using IDEs such as Visual Studio or Xcode, you'll likely want to invoke one or more IDE generators when using CMake. For instance, setting
-G "Eclipse CDT4 - Unix Makefiles" -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE
will cause Eclipse projects to be generated in each build area, and also in the source code area that is shared for all builds. Of course, if you are using Xcode or Visual Studio, you'll have to substitute the appropriate flag for those IDEs. Alternatively, you could consider using Eclipse with Ninja on all platforms, though at the time of writing, I am not completely certain that Ninja is ready for prime-time on non-Linux, non-Windows operating systems.)

How to have cmake unpack my compiler

In order to ensure that my Linux builds are identical regardless of the distribution the build host uses, I have packaged up my compiler and the sysroot files into a relocatable tar file and checked that into source control.
So the first step in any build (or at least, a step that must be invoked before any compile step) must be to extract this tar file.
If I was using a makefile, this would be simple to do. However, the project is using cmake and I can't figure out any way to do it with cmake. It might even be that I need this extract step invoked before cmake starts to detect the compiler: I can hard-code the compiler name but if cmake fails if it can't find the compiler then I need the unpack to happen before that test.
Is this possible with cmake?
You can use execute_process to invoke cmake's cross-platform command mode (cmake -E tar). The command would be something like:
execute_process(COMMAND ${CMAKE_COMMAND} -E tar xvf MyCompiler.bz2)
The command which causes CMake to check for a valid compiler is project, so as long as you have your execute_process call before the project call, the unpacking will be done before the compiler check.