Cython, CMake and out-of-source build results in double compilation - cmake

Question

I have a big project containing multiple projects that create library/executables and some even a python module using a library.
To keep it simple, I have an MWE included called myprogram that creates an executable and a shared library for use by Cython code to create a python package for that same code. I use CMake to compile the code but due to my setup, as detailed here, the build process of the python routine is executed twice (during build and install).
During build the output is:
Scanning dependencies of target pymyprogram
[ 80%] Generating build/timestamp
Compiling /builds/myprogram/src/myprogram/pymyprogram.pyx because it changed.
[1/1] Cythonizing /builds/myprogram/src/myprogram/pymyprogram.pyx
running build_ext
building 'pymyprogram' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/builds
creating build/temp.linux-x86_64-3.8/builds/myprogram
creating build/temp.linux-x86_64-3.8/builds/myprogram/src
creating build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/builds/myprogram/src/myprogram -I/builds/myprogram/include -I/builds/myprogram/buildRelease -I/opt/rh/rh-python38/root/usr/include/python3.8 -c /builds/myprogram/src/myprogram/pymyprogram.c -o build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram/pymyprogram.o
creating build/lib.linux-x86_64-3.8
gcc -pthread -shared -L/opt/rh/rh-python38/root/usr/lib64-Wl,-z,relro -Wl,-rpath,/opt/rh/rh-python38/root/usr/lib64 -Wl,--enable-new-dtags -g -L/opt/rh/rh-python38/root/usr/lib64-Wl,-z,relro -Wl,-rpath,/opt/rh/rh-python38/root/usr/lib64 -Wl,--enable-new-dtags -g build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram/pymyprogram.o -L/builds/myprogram/buildRelease/src/myprogram -L/opt/rh/rh-python38/root/usr/lib64 -lmyprogram -o build/lib.linux-x86_64-3.8/pymyprogram.cpython-38-x86_64-linux-gnu.so -Wl,-rpath=/builds/myprogram/lib/
[ 80%] Built target pymyprogram
And during install:
-- Installing: /builds/myprogram/bin/myprogram.x
running install
running build
running build_ext
building 'pymyprogram' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/builds
creating build/temp.linux-x86_64-3.8/builds/myprogram
creating build/temp.linux-x86_64-3.8/builds/myprogram/src
creating build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/opt/rh/rh-python38/root/usr/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/builds/myprogram/src/myprogram -I/builds/myprogram/include -I/builds/myprogram/buildRelease -I/opt/rh/rh-python38/root/usr/include/python3.8 -c /builds/myprogram/src/myprogram/pymyprogram.c -o build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram/pymyprogram.o
creating build/lib.linux-x86_64-3.8
gcc -pthread -shared -L/opt/rh/rh-python38/root/usr/lib64-Wl,-z,relro -Wl,-rpath,/opt/rh/rh-python38/root/usr/lib64 -Wl,--enable-new-dtags -g -L/opt/rh/rh-python38/root/usr/lib64-Wl,-z,relro -Wl,-rpath,/opt/rh/rh-python38/root/usr/lib64 -Wl,--enable-new-dtags -g build/temp.linux-x86_64-3.8/builds/myprogram/src/myprogram/pymyprogram.o -L/builds/myprogram/buildRelease/src/myprogram -L/opt/rh/rh-python38/root/usr/lib64 -lmyprogram -o build/lib.linux-x86_64-3.8/pymyprogram.cpython-38-x86_64-linux-gnu.so -Wl,-rpath=/builds/myprogram/lib/
running install_lib
creating /builds/myprogram/lib64
creating /builds/myprogram/lib64/python3.8
creating /builds/myprogram/lib64/python3.8/site-packages
copying build/lib.linux-x86_64-3.8/pymyprogram.cpython-38-x86_64-linux-gnu.so -> /builds/program/lib/python3.8/site-packages
running install_egg_info
Writing /builds/myprogram/lib/python3.8/site-packages/pymyprogram-1.1A-py3.8.egg-info
Also when I make changes to otherprogram.c but no changes to myprogram in either the library of pyx code, at install stage the library is build anyway which increases compilation time.
I therefore searched for a solution and found the method from PJ_Finnegan but have a few issues so wondered whether others have resolved that or could help out.
My working tree for the MWE is:
.
├── README.TXT
├── README.md
├── bin
├── CMakeLists.txt
├── buildDebug
│   ├── CMakeCache.txt
│   ├── CMakeFiles
│   ├── Makefile
│   ├── cmake_install.cmake
│   ├── install_manifest.txt
│   ├── src
│   └── utils
├── include
│   ├── version.h
│   └── version.h.txt
├── lib
│   └── libmyprogram.dylib
├─── src
│   ├── some other myprogram
│   │   ├── CMakeLists.txt
│   │   ├── othermyprogram.h
│   │   ├── othermyprogram.c
│   ├── myprogram
│   │   ├── CMakeLists.txt
│   │   ├── myprogram.h
│   │   ├── myprogram.c
│   │   ├── myprogram.pyx
│   │   ├── setup.py.in
│   └── versioning.cmake
The top level CMakeLists.txt contains add_subdirectory(src/myprogram) and the content of the CMakeLists.txt in src/myprograms is:
cmake_minimum_required(VERSION 2.8.8...3.20.5 FATAL_ERROR)
project (myprogram)
message ("-- Configuring: *** myprogram **")
if(APPLE)
set(CMAKE_MACOSX_RPATH 1)
endif()
set(CMAKE_C_FLAGS "${CFLAGS} -O0 -ggdb -fPIC")
set(CMAKE_C_FLAGS_DEBUG "${CFLAGS} -O0 -ggdb -fPIC")
set(CMAKE_C_FLAGS_RELEASE "${CFLAGS} -O3 -fPIC")
set(CMAKE_CXX_FLAGS "${CFLAGS} -O0 -ggdb")
set(CMAKE_CXX_FLAGS_DEBUG "${CFLAGS} -O0 -ggdb")
set(CMAKE_CXX_FLAGS_RELEASE "${CFLAGS} -O3")
# *** myprogram.so ***
SET(libmyprogram_SRCS
myprogram.c
)
ADD_LIBRARY(myprogram SHARED
${libmyprogram_SRCS}
)
ADD_DEPENDENCIES(myprogram versioning)
TARGET_LINK_LIBRARIES(myprogram
m
)
install(TARGETS myprogram
DESTINATION "lib")
if (APPLE)
install(CODE "execute_process(COMMAND ln -sf ${PROJECT_HOME}/lib/libmyprogram.dylib /usr/local/lib/libmyprogram.dylib)")
endif()
# *** myprogram.x ***
SET(myprogram_SRCS
myprogram.c
)
ADD_EXECUTABLE(myprogram.x
${myprogram_SRCS}
)
ADD_DEPENDENCIES(myprogram.x myprogram versioning)
TARGET_LINK_LIBRARIES(myprogram.x
m
)
set (EXECUTABLES "${EXECUTABLES}" myprogram.x)
# install executables and scripts
install (TARGETS ${EXECUTABLES}
RUNTIME DESTINATION "bin")
# *** pymyprogram.pyx ***
find_package(PythonInterp 3 REQUIRED)
if (CMAKE_BUILD_TYPE MATCHES "Debug")
message(STATUS "Python: version=${PYTHON_VERSION_STRING} interpreter=${PYTHON_EXECUTABLE}")
set(PYTHON_BUILD_FLAGS "--debug")
endif()
# Get location of user site-packages to install libs to
execute_process(COMMAND python3 -m site --user-site OUTPUT_VARIABLE PYTHON_INSTALL_DIR OUTPUT_STRIP_TRAILING_WHITESPACE)
if (PYTHONINTERP_FOUND)
set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/pymyprogram.pyx")
set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build/timestamp")
message ("-- Building Python Extension using: " ${PYTHON_EXECUTABLE})
configure_file(${SETUP_PY_IN} ${SETUP_PY})
add_custom_command(OUTPUT ${OUTPUT}
COMMAND ${PYTHON_EXECUTABLE} ${SETUP_PY} build ${PYTHON_BUILD_FLAGS} --build-lib ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_PROJECT_NAME}
COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
DEPENDS ${DEPS})
add_custom_target(pymyprogram ALL DEPENDS ${OUTPUT})
add_dependencies(pymyprogram myprogram versioning)
install(CODE "execute_process(COMMAND ${PYTHON_EXECUTABLE} ${SETUP_PY} install_lib --skip-build
--build-dir ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_PROJECT_NAME}
--install-dir=${PYTHON_INSTALL_DIR} --force
COMMAND ${PYTHON_EXECUTABLE} ${SETUP_PY} install_egg_info
--install-dir=${PYTHON_INSTALL_DIR})" DEPENDS ${DEPS})
else()
message (WARNING "-- Could not build *** pymyprogram **, could not locate PYTHON: ${PYTHON_EXECUTABLE} or improper version")
endif()
And the setup.py.in that is used:
import os, sys
from distutils.core import setup, Extension
from Cython.Build import cythonize
link_arguments = []
if (sys.platform == 'darwin'):
link_arguments.append("-Wl,-rpath")
link_arguments.append("-Wl,#loader_path/")
os.environ["CC"] = "gcc"
else:
link_arguments.append("-Wl,-rpath=${CMAKE_SOURCE_DIR}/lib/")
myprogram_extension = Extension(
name="pymyprogram",
sources=["${CMAKE_CURRENT_SOURCE_DIR}/pymyprogram.pyx"],
libraries=["myprogram"],
extra_link_args = link_arguments,
library_dirs=["${PROJECT_BINARY_DIR}"],
include_dirs=["${CMAKE_SOURCE_DIR}/include", "${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}"],
)
setup(name="pymyprogram",
author="My Name",
author_email="my-email",
version="1.1A",
description="Python wrapper for myprogram",
package_dir={ "": "${CMAKE_SOURCE_DIR}/lib/" },
license="MIT License",
ext_modules=cythonize([myprogram_extension], compiler_directives={'language_level' : "3"})
)
The code runs on different machines (Linux or macOS) and my module was previously build and installed using this method and either the --user or --prefix of the install command based on whether the code ran on Linux or macOS.
To avoid the double build step of the python module, I implemented a new method from PJ_Finnegan as mentioned above and shown in the code sample of my CMakeLists.txt. This method uses install_lib, to avoid the second build which allows the use of --build-dir which is great but no longer works with --user or --prefix which is unwanted as it will require me to define the path manually and that the path no longer depends on the python version automatically.
I also wonder what the difference between using install-lib vs install entails. I noticed no egg-info file being created, is this a problem? I therefore added the install_egg_info step as well.
Also, the the mypogram code is build twice, (the executable and library), and a third time for the python package. Is there some optimisation possible to reduce compile time?
Any other ways to build and install a python/Cython extension and separate standalone executable and help the install step find the build code? Perhaps using sdist and pip install?
EDIT
I've changed the INSTALL command in my CMAKE files to:
install(CODE "execute_process(COMMAND ${PYTHON_EXECUTABLE} ${SETUP_PY} install ${PYTHON_INSTALL_PREFIX} --skip-build --force WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})")
This reduces the install step to only installation and no more build as the build directory from the previous build step is now found.
Now the question remains, how to make the CMAKE install command not run every time and only upon changes in the *.pyx file. I know add_custom_target is always considered out-of-date so is there perhaps a workaround?