Set multiple paths to CMAKE_Fortran_MODULE_DIRECTORY - cmake

Question

So I am trying to include an installed Fortran library to a CMake project. I know the directory where the module (.mod) file is located, but my project can't seem to find it unless I set CMAKE_Fortran_MODULE_DIRECTORY.
SET(LIB_INCLUDE_DIR /usr/local/include)
SET(EX_FILES main.f90 file1.f90 file2.f90)
INCLUDE_DIRECTORIES(${LIB_INCLUDE_DIR})
ADD_EXECUTABLE(test ${EX_FILES})
TARGET_LINK_LIBRARIES(test lib1)
Where the error is
use lib1
1
Fatal Error: Can't open module file 'lib1.mod' for reading at (1): No such file or directory
unless I include the line
SET(CMAKE_Fortran_MODULE_DIRECTORY ${LIB_INCLUDE_DIR})
And then it can find the file just fine. But I'm running into a little problem. By setting CMAKE_Fortran_MODULE_DIRECTORY, CMake tries to write all generated modules to this directory rather than the CMAKE_BINARY_DIR (where I would like it).
From the documentation, I know that CMAKE_Fortan_MODULE_DIRECTORY is meant to be set to specify where to write generated module files, but some compilers look to that directory to find modules. Is there any way to set multiple directories so that if it can't find in/write to one directory, it searches the second? When I try to set CMAKE_Fortran_MODULE_DIRECTORY to be multiple directories, it only looks at the first directory.
If it helps I am on a Ubuntu 14.04 LTS system using gfortran 4.8.4
EDIT:
So by Alexander Vogt's suggestion I ran with VERBOSE=1 and got
cd /home/user/Repos/build/src && gfortran -I/home/user/Repos/build/src -isystem /usr/local/include -c /home/user/Repos/test_project/src/main.f90 -o CMakeFiles/test.dir/main.f90.o
when I did NOT set CMAKE_Fortran_MODULE_DIRECTORY and
cd /home/user/Repos/build/src && gfortran -J/usr/local/include -I/home/user/Repos/build/src -isystem /usr/local/include -c /home/user/Repos/test_project/src/main.f90 -o CMakeFiles/test.dir/main.f90.o
when I did.
It seems the only difference is the -J flag, which sets exactly what CMAKE_Fortran_MODULE_DIRECTORY sets. Is there some flag that sets just where to look for compiled modules, and not where to put them?