CUDA Separable Compilation with CMake, invalid device function - cmake

Question

I am developing a C++ application with cmake as the build system. Each component in the application builds into a static library, which the executable links to.
I am trying to link in some cuda code that is built as a separate static library, also with cmake. When I attempt to invoke the global function entry point in the cuda static library from the main application, everything seems to work fine - the cudaDeviceSynchronize that follows my global function invocation returns 0. However, the output of the kernel is not set and the call returns immediately.
I ran cuda-gdb. Despite the code being compiled with -g and -G, I was not able to break within the device function called by the kernel. So, I ran cuda-memcheck. When the kernel is launched, this message appears:
========= Program hit cudaErrorInvalidDeviceFunction (error 8) due to "invalid device function" on CUDA API call to cudaLaunchKernel.
I looked this up, and the NVIDIA docs/forum posts I read suggested this is usually due to compiling for the wrong compute capability. However, I'm running Titan V's, and the CC is correctly set to 7.0 when compiling.
I have set CUDA_SEPARABLE_COMPILATION on both the cuda library and the component in the main application that the cuda code links to per https://devblogs.nvidia.com/building-cuda-applications-cmake/. I've also tried setting CUDA_RESOLVE_DEVICE_SYMBOLS.
Here is the relevant portion of the cmake for the main application:
(kronmult_cuda is the component in the main application that links to the cuda library ${KRONLIB}. another component, kronmult, links to kronmult_cuda. Eventually, something that links to kronmult is linked to the main application).
find_package(CUDA 9.0 REQUIRED)
include_directories(${CUDA_INCLUDE_DIRS})
enable_language(CUDA)
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -arch sm_70 -g --ptxas-options=-O3")
set_source_files_properties( src/kronmult_cuda.cpp PROPERTIES LANGUAGE CUDA ) # no .cu extension
...
target_include_directories(kronmult_cuda PRIVATE ${KRON_PATH})
target_link_libraries(kronmult_cuda PRIVATE OpenMP::OpenMP_CXX PUBLIC ${KRON_LIB})
if (ASGARD_USE_CUDA)
set_target_properties(kronmult_cuda
PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
endif()
if(APPLE AND ASGARD_USE_GPU)
set_target_properties(kronmult_cuda
PROPERTIES
BUILD_RPATH ${CMAKE_CUDA_IMPLICIT_LINK_DIRECTORIES})
endif ()
target_link_libraries(kronmult PRIVATE kronmult_cuda)
...
Full CMakeLists: https://github.com/bmcdanie/ASGarD/blob/feature/kronmult/CMakeLists.txt.
relevant CMakeLists portion for cuda library:
project(kronmult LANGUAGES CXX CUDA)
set(KRONSRC
[list of all sources]
)
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -arch sm_70 -g --ptxas-options=-O3")
set_source_files_properties( ${KRONSRC} PROPERTIES LANGUAGE CUDA )
add_library(kron STATIC ${KRONSRC})
target_compile_features(kron PUBLIC cxx_std_11)
set_target_properties( kron
PROPERTIES CUDA_SEPARABLE_COMPILATION ON)
Full CmakeLists: https://github.com/project-asgard/kronmult/blob/master/CMakeLists.txt.
What am I missing here?
EDIT:
Output of cuda-gdb when I attempt to call kernel:
Thread 1 "asgard" hit Breakpoint 1, kronmult2_xbatched<double> (n=2, Aarray_=0x15551fa24800, lda=8, pX_=0x15551fa23c00, pY_=0x15551fa24400, pW_=0x15551fa24000, batchCount=128)
at /home/3bm/asgard/contrib/kronmult/src/kronmult-ext/kronmult2_xbatched.hpp:36
36 {
(cuda-gdb) step
__wrapper__device_stub_kronmult2_xbatched<double> (__cuda_0=#0x7fffffff9e1c: 2, __cuda_1=0x15551fa24800, __cuda_2=#0x7fffffff9e18: 8, __cuda_3=0x15551fa23c00,
__cuda_4=0x15551fa24400, __cuda_5=0x15551fa24000, __cuda_6=#0x7fffffff9e30: 128) at /tmp/tmpxft_0000ac33_00000000-5_kronmult_cuda.cudafe1.stub.c:40
40 /tmp/tmpxft_0000ac33_00000000-5_kronmult_cuda.cudafe1.stub.c: No such file or directory.
(cuda-gdb) step
__device_stub__Z18kronmult2_xbatchedIdEviPKPKT_iPPS0_S6_S6_i (__par0=2, __par1=0x15551fa24800, __par2=8, __par3=0x15551fa23c00, __par4=0x15551fa24400, __par5=0x15551fa24000,
__par6=128) at /tmp/tmpxft_0000ac33_00000000-5_kronmult_cuda.cudafe1.stub.c:39
39 in /tmp/tmpxft_0000ac33_00000000-5_kronmult_cuda.cudafe1.stub.c
(cuda-gdb) step
dim3::dim3 (this=0x7fffffff9d28, vx=1, vy=1, vz=1)
at /home/dg6/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/cuda-10.0.130-s6ervywpchxmerrju62il7xkeeamlfcv/include/vector_types.h:420
420 __host__ __device__ dim3(unsigned int vx = 1, unsigned int vy = 1, unsigned int vz = 1) : x(vx), y(vy), z(vz) {}
(cuda-gdb) step
dim3::dim3 (this=0x7fffffff9d34, vx=1, vy=1, vz=1)
at /home/dg6/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/cuda-10.0.130-s6ervywpchxmerrju62il7xkeeamlfcv/include/vector_types.h:420
420 __host__ __device__ dim3(unsigned int vx = 1, unsigned int vy = 1, unsigned int vz = 1) : x(vx), y(vy), z(vz) {}
(cuda-gdb) step
cudaLaunchKernel<char> (
func=0x5555555f94c0 <kronmult2_xbatched<double>(int, double const* const*, int, double**, double**, double**, int)> "UH\211\345H\203\354\060\211}\374H\211u\360\211U\370H\211M\350L\211E\340L\211M\330L\213E\330H\213}\340H\213M\350H\215U\370H\213u\360H\215E\374H\203\354\bL\215M\020AQM\211\301I\211\370H\211\307\350\355\343\377\377H\203\304\020\220\311\303UH\211\345H\203\354\060\211}\374H\211u\360\211U\370H\211M\350L\211E\340L\211M\330L\213E\330H\213}\340H\213M\350H\215U\370H\213u\360H\215E\374H\203\354\bL\215M\020AQM\211\301I\211\370H\211\307\350\267\345\377\377H\203\304\020\220\311\303UH\211\345H\203\354\060\211}\374H\211u\360\211U\370H\211M\350L\211E\340L\211", <incomplete sequence \330>..., gridDim=..., blockDim=...,
args=0x7fffffff9d40, sharedMem=0, stream=0x0)
at /home/dg6/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/cuda-10.0.130-s6ervywpchxmerrju62il7xkeeamlfcv/bin/..//include/cuda_runtime.h:202
202 return ::cudaLaunchKernel((const void *)func, gridDim, blockDim, args, sharedMem, stream);
(cuda-gdb) step
warning: Cuda API error detected: cudaLaunchKernel returned (0x8)

Answer 1

After the helpful hint from #talonmies, I suspected this was a device linking problem. I simplified my build process, included all CUDA files in one translation unit, and turned off SEPARABLE COMPILATION.
Still, I did not see a cmake_device_link.o in either my main application binary or the component that called into my cuda library. And, still had the same error. Tried setting CUDA_RESOLVE_DEVICE_SYMBOLS to no effect.
Finally, I tried building the component that calls into my cuda library as SHARED. I saw the device linking step when building the .so in my cmake output, and the program runs fine. I do not know why building SHARED fixes what I suspect was a device linking problem - will accept any answer that deciphers that?