Short question:
Is it possible to set CMake to follow symlinks when copying files during an install, and if so how is this done?
Details: I am using CMake to build and install LLVM. In my LLVM source tree in the include directory I have a symbolic link to another subproject that is being developed against LLVM. Everything seems to work, except that I noticed that when I ran "cmake install" that it copied the include directory without following the symlinks. The problem that I have is that my symlinks have a relative path (because it is inside a git repo). So when the symlinks are copied (instead of followed and copying the contents) they no longer point to the correct files. For example I have dsa -> ../../llvm-poolalloc/include/dsa/ I would like to copy the contents of this link when I do the install rather than just copying the link. But I did not find a cmake flag for doing this yet.
I realize that this is probably not the idea way to structure my project, but I am working with something that's already in place and it would be preferable to not have to change too much of the directory structures because other people I am working with expect it to be this way. So I think that being able to follow symlinks might solve my problem without having to restructure the whole build system. But I am open to other suggestions for better ways to accomplish what I am trying to do.
Note that I am working in Linux (Ubuntu 10.04) and using LLVM 2.6 (that I am compiling from source along with llvm-gcc). Also I am using CMake version 2.8.
Edit:
Here is the source code from the CMakeLists.txt file that is associated with the install instruction:
install(DIRECTORY include
DESTINATION .
PATTERN ".svn" EXCLUDE
PATTERN "*.cmake" EXCLUDE
PATTERN "*.in" EXCLUDE
PATTERN "*.tmp" EXCLUDE
)
install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/include
DESTINATION .
)
the directory listing for the include directory is:
dsa -> ../../llvm-poolalloc/include/dsa/
llvm
llvm-c
poolalloc -> ../../llvm-poolalloc/include/poolalloc
What I want is for the dsa and poolalloc directories to be copied rather than just copying the symbolic links. The reason that I don't use absolute paths in the symbolic links is that I have them checked into a git repo. So my absolute path would differ from someone else working on the project when they do a checkout from the repo.
Hmm, let's try this:
get_filename_component(ABS_DIR include REALPATH)
install(DIRECTORY ${ABS_DIR}
DESTINATION .
PATTERN ".svn" EXCLUDE
PATTERN "*.cmake" EXCLUDE
PATTERN "*.in" EXCLUDE
PATTERN "*.tmp" EXCLUDE
)
If it wouldn't help, you can try to install not the include dir itself (which is symlink), but it's contents. But in your case you would need to came up with smart regex:
file(GLOB INCLUDES include/*) # filter there .svn and others
install(FILES ${INCLUDES}
DESTINATION include
)
Finally, make the symlink absolute.
Related
Update: turns out the problem is not related to the Jenkins agent but to CMake. It is easily reproducible from the command line.
It was reported here once before:
CMake cannot follow symlinks on Windows 10
However, the problem is not OS-related. We encounter it on Linux also.
I'm debugging the following error:
CMake Error at C:/jenkins/trial/workspace/WWB6-6.13.0/wwb-Pilot_Build_BB/wwb6/build/cmake_install.cmake:48 (file):
file INSTALL cannot read symlink
"C:/jenkins/trial/workspace/WWB6-6.13.0/wwb6-Build-Pilot-Windows/wwb6/wwb6/HelpFiles"
to duplicate at
"C:/jenkins/trial/workspace/WWB6-6.13.0/wwb6-Build-Pilot-Windows/wwb6/build/_CPack_Packages/win64/NSIS/WWB6 Setup/./Help".
The relevant line in the make file is:
file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/./Help" TYPE DIRECTORY FILES "C:/jenkins/trial/workspace/WWB6-6.13.0/wwb-Pilot_Build_BB/wwb6/wwb6/HelpFiles/")
The intent is to copy the contents of the HelpFiles directory into a new directory "Help" in the workspace. The HelpFiles directory in this scenario is a symbolic link to another directory in the source repository:
Directory of C:\jenkins\trial\workspace\WWB6-6.13.0\wwb6-Build-Pilot-Windows\wwb6\wwb6
05/17/2019 03:09 PM <SYMLINKD> HelpFiles [..\helpwwb6]
Traversing this symlink from the command line works fine:
C:\jenkins\trial\workspace\WWB6-6.13.0\wwb6-Build-Pilot-Windows\wwb6\wwb6>cd HelpFiles
C:\jenkins\trial\workspace\WWB6-6.13.0\wwb6-Build-Pilot-Windows\wwb6\wwb6\HelpFiles>
Anyone know of a workaround for this problem? The version of CMake we're running is 3.12.0.
Fortunately, there's an easy (although not elegant) workaround for this: replace the symlink with the real path in the CMakeLists file:
install ( DIRECTORY ${CMAKE_SOURCE_DIR}/../helpwwb6/
DESTINATION ${BINARY_INSTALL_LOCATION}/Help
COMPONENT Runtime
PATTERN ".svn" EXCLUDE
PATTERN ".git" EXCLUDE
)
Would rather see CMake work with symlinks though.
I use a CMakeLists.txt with the following install command:
install(DIRECTORY ./ DESTINATION include FILES_MATCHING PATTERN "*.h")
It correctly installs all "./*.h" files, but also copies the "./.git" directory structure (without any files).
The problem happens when using CMake 3.14.0 and did not happen with CMake 3.11.1.
Did the command change or is this a CMake bug? Should I use an explicit exclude for ".git" or can I somehow keep the whitelist approach, that will e.g. keep working when I actually need to install subfolders?
As of now, there does not seem to exist any straightforward solution other than explictly specifying to exclude your directory. The behaviour is not new to version 3.14.0 and was similar in 3.11.1. The fact that your .git directory wasn't copied might be due to another command in your CMakeLists...
As you suggest and based on this post and this thread in the old CMake forum, a solution for you would be:
install(DIRECTORY ./ DESTINATION include FILES_MATCHING PATTERN "*.h" PATTERN ".git*" EXCLUDE)
There is a ticket to add a feature to not include empty directories when using install(DIRECTORY ...), so you might keep an eye on it for when it is finally implemented.
Alternatively, you may use nested file(GLOB ...) followed by install(FILES ...), with the inherent drawbacks of globbing (see the note in the documentation).
I am building a project using ROS and thus, catkin_make to build my ROS nodes and libraries.
The problem I'm facing is:
I am using a git submodule in one package (package A) (and thus, I have a hierarchical include folder structure) and I have difficulties referencing a header file within that submodule.
In order to build the package B, which is dependent on package A, I have added the INCLUDE_DIRS statement to the catkin_package command in package A:
catkin_package(
INCLUDE_DIRS my-submodule/include
...
)
The content of that directory is:
my-submodule/my-header.h
(I have put the header files under a folder, named after the submodule, as many tutorials stated that within ROS you should use this convention).
The include statement in a file from package-B reads like this:
...
#include <my-submodule/my-header.h>
...
This works fine - package B is being built (as I am using one combined workspace to build this).
But: When I switch to the target system, where I only install package A, and then try to build package B (on that target system), it does not build because the include paths are not setup correctly.
The INSTALL statement for package A looks like this
install(DIRECTORY my-submodule/include
DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
FILES_MATCHING PATTERN "*.h"
PATTERN ".svn" EXCLUDE
)
This is mainly, because the installed folder structure on the target system looks like this:
.../ros/include/my-package-A/include/my-submodule/my-header.h
So, the install process actually puts that submodule's include-path under the package-A-include path (which is a different path structure compared to when I build the packages directly in one combined workspace).
And the CFLAGS for compilation only set the include directory to the folder:
.../ros/include
And thus, breaking my include statement in my package-B file:
#include <my-submodule/my-header.h>
Do you have any idea how to solve this?
I am sure there are more people than me, trying to reference header files from a submodule within a package.
Assuming you have a file my-submodule/include/my-submodule/my-header.h inside your package A, then two small changes to your install() statement should fix this:
install(DIRECTORY my-submodule/include/
DESTINATION ${CATKIN_GLOBAL_INCLUDE_DESTINATION}
FILES_MATCHING PATTERN "*.h"
PATTERN ".svn" EXCLUDE
)
First, add a slash to the path (.../include/ instead of .../include), which causes the contents of the include folder to be installed instead of the include folder itself (Otherwise you'll end up with ../ros/install/include/include/my-submodule/my-header.h)
Secondly, use ${CATKIN_GLOBAL_INCLUDE_DESTINATION} (which points to .../ros/install/include/) instead of ${CATKIN_PACKAGE_INCLUDE_DESTINATION} (which points to .../ros/install/my-package-A/include/) as destination.
The alternative would be to fix catkin, as
catkin_package(
INCLUDE_DIRS my-submodule/include
...
)
should theoretically already export my-submodule/include, so you can pick it up in package B with
find_package(catkin REQUIRED DEPENDS my-package-A)
catkin_package(
CATKIN_DEPENDS my-package-A
)
include_directories(${catkin_INCLUDE_DIRS})
Unfortunately, for some reason this is explicitely not possible when using catkin config --install. See https://answers.ros.org/question/335846/install_dirs-not-working-as-expected-when-using-install/.
I´m trying to run packet generator within a VS project, it crashes while compiling because of the use of absolute path on installation from Targets and Files.
ABSOLUTE path INSTALL DESTINATION forbidden (by caller): ...
I checked twice and all installation directories are relative. I set quite a lot of variables as sub-folders of ${PROJECT_BINARY_DIR} (which should be relative) such as:
set(INSTALL_DIR ${PROJECT_BINARY_DIR}/bin)
set(LIB_DIR ${PROJECT_BINARY_DIR}/bin/lib)
set(EXT_DIR ${PROJECT_BINARY_DIR}/bin/ext)
...
does CMAKE/CPACK interpret those variables as absolute paths?
If so, is there a way to make CPack working properly with those variables?
How do I use CPack when sub-relative path are involved?
thanks
Ok I see, the ${PROJECT_BINARY_DIR} is interpreted as an ABSOLUTE path, from there all sub-folder of it will be rejected.
To avoid this problem I surrounded the install variables in if else blocks, and if it is the case of packaging then a relative folder will be used as follows:
if(PACK)
set(INSTALL_DIR bin)
set(LIB_DIR bin/lib)
set(EXT_DIR /bin/ext)
...
else(PACK)
set(INSTALL_DIR ${PROJECT_BINARY_DIR}/bin)
set(LIB_DIR ${PROJECT_BINARY_DIR}/bin/lib)
set(EXT_DIR ${PROJECT_BINARY_DIR}/bin/ext)
...
endif(PACK)
this solves it, but it is really dirty, waiting for a better function on new CPack version.
ciao
This fatal error is meant to tell you installation root should be specified at the moment when user executes the installer. I guess somewhere in your cmake config might have code like this:
INSTALL (TARGET myApp DESTINATION ${SOME_INSTALL_PATH}/bin )
If you assign SOME_INSTALL_PATH an absolute path when cmake cache is generated, you incur the CMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION error, which gave you the "ABSOLUTE path INSTALL DESTINATION forbidden (by caller)" message.
To solve this problem, either always use relative path for installation DESTINATION or assign only package prefix to SOME_INSTALL_PATH variable.
For reference, following is the link to INSTALL command.
http://www.cmake.org/cmake/help/v3.0/variable/CMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION.html
There was also a similar question asked on the CMake mailing list.
http://public.kitware.com/pipermail/cmake/2013-May/054656.html
(apologies: cross-posted from CMake mailing list)
I'm trying to get my head round CMake's regex implementation; I have a folder containing 4 folders and 2 text files as follows:
build/
projectA/
CMakeLists.txt
extrafiles/
README
temp/
One line of CMakeLists.txt is:
set(CPACK_SOURCE_IGNORE_FILES "[^projectA]$")
In my source package that is then subsequently generated, build/, projectA/ and extrafiles are present, but temp/ and the 2 text files are not. I'm trying to get to a stage where the regex will ignore everything in the folder except for projectA/, README and CMakeLists.txt, but can't work out at the moment how the regex I've
supplied is giving those results.
I guess what this boils down to is how to match a whole string using regex. I realise that the docs say Matches any character(s) not inside the brackets which is where I guess I'm going wrong...
Further exploration
In trying to understand CMake's regex implementation, I thought I'd start from 1st principles and do some easy stuff.
If I do
set(CPACK_SOURCE_IGNORE_FILES projectA)
then the folder projectA doesn't appear in my source package (as expected); however, if I do
set(CPACK_SOURCE_IGNORE_FILES ^projectA$)
or
set(CPACK_SOURCE_IGNORE_FILES ^/projectA/$)
then projectA does appear. What is it about the ^ (beginning of line) and $ (end of line) that I'm not understanding?
Even more
As probably obvious, projectA is not actually the name of my project, but everything above holds true when I physically rename my project folder to projectA. But, when I replace
set(CPACK_SOURCE_IGNORE_FILES projectA)
with
set(CPACK_SOURCE_IGNORE_FILES <name of my project>)
and rename my actual project folder from projectA to its actual name, I end up with an empty tarball! Argh! I have absolutely no idea what strange tricks CMake is playing on me, but I just want to cry.
Any insight will be greatly appreciated!
SELF CONTAINED EXAMPLE
As requested by Fraser, a self contained example showing 2 of the 'features' I've described. However, I do know that I'm running CMake in a slightly non-standard way, in order to keep everything to do with individual builds together, so if there's any proof running CMake in a more standard way eliminates these problems I'd be interested to see them.
Step 1: creating files
Create tree:
cd ~
mkdir
cd projectA
mkdir projectA
Create C file, and save it as ~/projectA/projectA/helloworld.c:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
printf("!!!Hello World!!!\n"); /* prints !!!Hello World!!! */
printf("!!!Hello CMake!!!\n"); /* prints !!!Hello CMake!!! */
return 0;
}
create a file that won't need compiling, and save it as ~/projectA/test.sh:
#A non compiled program
echo "Hello world!"
create ~/projectA/CMakeLists.txt:
cmake_minimum_required (VERSION 2.6)
project (HelloWorld)
set(CMAKE_INSTALL_PREFIX "$ENV{HOME}/projectAinstall")
add_executable(helloworld projectA/helloworld.c)
install(TARGETS helloworld DESTINATION .)
include(InstallRequiredSystemLibraries)
set(CPACK_GENERATOR "TGZ")
set(CPACK_SOURCE_GENERATOR "TGZ")
include(CPack)
Step 2: compiling
In ~/projectA, run:
chris#chris:~/projectA$ cmake -H. -Bbuild
then:
make -C build && make -C build package && make -C build package_source
this results in 2 tarballs in the build folder. Moving these somewhere else and untarring them shows helloworld in the binary tarball (as expected), and everything from the ~/projectA/projectA in the source tarball, including test.sh which won't get compiled (which Fraser seemed surprised about)
Step 3: random tests
Modifying CMakeLists.txt to include
set(CPACK_SOURCE_IGNORE_FILES "projectA")
and rerunning the CMake / Make commands above results in an empty source tarball, but with the same binary tarball as above. I have now realised that changing the directory tree so that the top level directory is testproject (and so different to its child folder) doesn't result in an empty source tarball, and does only remove the files listed in CPACK_SOURCE_IGNORE_FILES
I don't think you can achieve what you're after using CPACK_SOURCE_IGNORE_FILES (although I'm not certain). As you rightly noted, CMake's regex handling allows for excluding groups of characters, but I don't think it allows for negating whole patterns. [See updated answer at the end of the edits.]
That being said, I guess you can list all the folders you wish to exclude in your install command. Not as robust as excluding everything except "projectA", but still here's the syntax:
install(DIRECTORY .
DESTINATION the_install_subdir
REGEX "build|extrafiles|temp+" EXCLUDE)
Regarding the empty tarball, I imagine that you maybe have <name of my project> both as your project's root dir and as a subdir? So in your example, if you called your project "projectA", then you'd have "projectA/build", "projectA/projectA", etc.
If so, the regex will work on the full path, and hence all files within your project will contain projectA/ within their paths.
As for the crying... well, I can only advise you to get a grip and pull yourself together! :-)
Edit: In response to the comments, here's a quick example of using the install command to achieve the goal:
install(DIRECTORY projectA
DESTINATION the_install_subdir)
install(FILES CMakeLists.txt README DESTINATION the_install_subdir)
Further Edit:
OK, your example helps a lot - I had indeed misunderstood what you were doing. I hadn't picked up that you were actually making 2 different targets ("package" and "package_source"). I had thought you were creating the binary package by doing something like
cpack -G DEB
and that you were creating the other package by doing
cpack -G TGZ
These both build the binary package. My mistake - I should have paid more attention. Sorry!
As for your specific questions:
Question 1
It seems to me that installing files / directories that aren't compiled but are at the same level as the folder containing all the compiled files (i.e. bin), and then ignoring the bin folder using CPACK_SOURCE_IGNORE_FILES results in an empty tarball - is this correct?
I take this to mean: "Should doing set(CPACK_SOURCE_IGNORE_FILES "${CMAKE_BINARY_DIR}") result in an empty tarball?" The answer is probably not.
Because CPACK_SOURCE_IGNORE_FILES represents a regex, I'm sure there are cases where the resultant regex could match every file in the project, and this would cause an empty tarball. However I imagine it's fairly unlikely.
If, rather than using the full path to your bin dir via the variable ${CMAKE_BINARY_DIR} you were to just give the folder name, there would be a much greater chance of an empty tarball. Say you call your bin dir "build" and have set(CPACK_SOURCE_IGNORE_FILES "build"). If your project lived in say ~/test_builds/projectA, then the regex "build" would match every file in the project since each contains "test_builds"; resulting in an empty tarball.
I think this is the crux of issue each time you've generated an empty tarball. Whatever the regex is trying to achieve, it actually ends up matching and excluding all files.
Question 2
It also seems that files in the CMAKE_SOURCE_DIR which aren't 'installed' don't end up in the binary tarball but do end up in the source tarball
Yes, the "package_source" is indeed a different target to the binary package. It by default contains all files in the ${CMAKE_SOURCE_DIR}, whereas the "package" target contains only items added via install commands. Here, the term "source files" is probably a slight misnomer since it means all files in the source tree - not just .c, .cc, .cxx, etc.
Original Question
I think there's a reasonably safe way to achieve your original aim after all! If you use file(GLOB ...) to generate a non-recursive list of all files/folders in your root, then remove those you wish to keep in the source package, you should be able to use the remaining list as the regex value of CPACK_SOURCE_IGNORE_FILES:
file(GLOB SourceIgnoreFiles "${CMAKE_SOURCE_DIR}/*")
set(SourceKeepFiles "${CMAKE_SOURCE_DIR}/projectA"
"${CMAKE_SOURCE_DIR}/CMakeLists.txt"
"${CMAKE_SOURCE_DIR}/README")
list(REMOVE_ITEM SourceIgnoreFiles ${SourceKeepFiles})
# Escape any '.' characters
string(REPLACE "." "\\\\." SourceIgnoreFiles "${SourceIgnoreFiles}")
set(CPACK_SOURCE_IGNORE_FILES "${SourceIgnoreFiles}")
Hopefully this should now work for you. Sorry again for the misdirections.
CMake tends to use absolute paths except in contexts where there's a strong argument for using relative paths. So I'm pretty sure it's running each regex in CPACK_SOURCE_IGNORE_FILES against absolute paths of files (which should answer your question "What is it about the ^ (beginning of line) and $ (end of line) that I'm not understanding?"). Anything that isn't matched by any regex in CPACK_SOURCE_IGNORE_FILES is not ignored.
What you want is probably something like:
set(CPACK_SOURCE_IGNORE_FILES
/build/
/extrafiles/
/temp/
)