I'm having trouble understanding if/how to share code among several Fortran projects without building libraries or duplicating source code.
I am using Eclipse/Photran with the Intel compiler (ifort) on a linux system, but I believe I'm having a bigger conceptual problem with modules than with the specific tools.
Here's a simple example: In ~/workspace/cow I have a source directory (src) containing cow.f90 (the PROGRAM) and two modules m_graze and m_moo in m_graze.f90 and m_moo.f90, respectively. This project builds and links properly to create the executable 'cow'. The executable and modules (m_graze.mod and m_moo.mod) are stored in ~/workspace/cow/Debug and object files are stored under ~/workspace/cow/Debug/src
Later, I create ~/workplace/sheep and have src/sheep.f90 as the program and src/m_baa.f90 as the module m_baa. I want to 'use m_graze, only: ruminate' in sheep.f90 to get access to the ruminate() subroutine. I could just copy m_graze.f90 but that could lead to code getting out of sync and doesn't take into account any dependencies m_graze might have. For these reasons, I'd rather leave m_graze in the cow project and compile and link sheep.f90 against it.
If I try to compile the sheep project, I'll get an error like:
error #7002: Error in opening the compiled module file. Check INCLUDE paths. [M_GRAZE]
Under Properties:Project References for sheep, I can select the cow project. Under Properties:Fortran Build:Settings:Intel Compiler:Preprocessor I can add ~/workspace/cow/Debug (location of the module files) to the list of include directories so the compiler now finds the cow modules and compiles sheep.f90. However the linker dies with something like:
Building target: sheep
Invoking: Intel(R) Fortran Linker
ifort -L/home/me/workspace/cow/Debug -o "sheep" ./src/sheep.o
./src/sheep.o: In function `sheep':
/home/me/workspace/sheep/src/sheep.f90:11: undefined reference to `m_graze_mp_ruminate_'
This would normally be solved by adding libraries and library paths to the linker settings except there are no appropriate libraries to link to (this is Fortran, not C.)
The cow project was perfectly capable of compiling and linking together cow.f90, m_graze.f90 and m_moo.f90 into an executable. Yet while the sheep project can compile sheep.f90 and m_baa.f90 and can find the module m_graze.mod, it can't seem to find the symbols for m_graze even though all the requisite information is present on the system for it to do so.
It would seem to be an easy matter of configuration to get the linker portion of ifort to find the missing pieces and put them together but I have no idea what magic words need to be entered where in the Photran UI to make this happen.
I confess an utter lack of interest and competence in C and the C build process and I'd rather avoid the diversion of creating libraries (.a or .so) unless that's the only way to make this work.
Ultimately, I'm looking for a pure Fortran solution to this problem so I can keep a single copy of the source code and don't have to manually maintain a pile of custom Makefiles.
So can this be done?
Apologies if this has already been documented somewhere; Google is only showing me simple build examples, how to create modules, and how to link with existing libraries. There don't seem to be (m)any examples of code reuse with modules that don't involve duplicating source code.
Edit
As respondents have pointed out, the .mod files are necessary but not sufficient; either object code (in the form of m_graze.o) or static or shared libraries must be specified during the linking phase. The .mod files describe the interface to the object code/library but both are necessary to build the final executable.
For an oversimplified toy problem such as this, that's sufficient to answer the question as posed.
In a larger project with more complex dependencies (in my case, 80+KLOC of F90 linking to the MKL version of LAPACK95), the IDE or toolchain may lack sufficient automatic or user-interface facilities to make sharing a single canonical set of source files a viable strategy. The choice seems to be between risking duplicate source files getting out of sync, giving up many of the benefits of an IDE (i.e. avoiding manual creation of make/CMake/SCons files), or, in all likelihood, both. While a revision control system and good code organization can help, it's clear that sharing a single canonical set of source files among projects is far from easy given the current state of Eclipse.
Some background which I suspect you already know: Typically (including ifort) compiling the source code for a Fortran module results in two outputs - a "mod" file that contains a description of the Fortran entities that the module defines that the compiler needs to find whenever it sees a USE statement for the module, and object code for the linker that implements the procedures and variable storage, etc., that the module defines.
Your first error (the one you solved) is because the compiler couldn't find the mod file.
The second error is because the linker hasn't been told about the object code that implements the stuff that was in the source file with the module. I'm not an Eclipse user by any means, but a brute force way of specifying that is just to add the object file (xxxxx/Debug/m_graze.o) as an additional linker option (Fortran Build > Settings, under Intel Fortran Linker > Command Line). (Other tool chains have explicit "additional object file" properties for their link stage - there may well be a better way of doing this for the Intel chain.)
For more involved examples you would typically create a library out of the shared code. That's not really C specific, the only Fortran aspect is that the libraries archive of object code needs to be provided alongside the mod files that the Fortran compiler generates.
Yes the object code must be provided. E.g., when you install libnetcdf-dev in Debian (apt-get install libnetcdf-dev), there is a /usr/include/netcdf.mod file that is included.
You can now use all netcdf routines in your Fortran code. E.g.,
program main
use netcdf
...
end
but you'll have link to the netcdf shared (or static) library, i.e.,
gfortran -I/usr/include/ main.f90 -lnetcdff
However, as user MSB mentioned the mod file can only be used by gfortran that comes with the distribution (apt-get install gfortran). If you want to use any other compiler (even a different version that you may have installed yourself) then you'll have to build netcdf yourself using that particular compiler.
So creating a library is not a bad solution.
Related
I have big project with cmake. It mostly works.
But recently some combination of compilation server vs test server broke. Investigation found that final compile/link command calls gcc (...) -licudata -licui18n -licuuc (...), this introduces dependency on shared library which is not present on test server.
How do I find out what in my project (my library, imported library, found library, whatever) adds those 3 flags to compile command?
I don't add them explicitly, so something is done automagically and I want to find it. compile_commands.json doesn't have them because linking flags don't belong in it. CMakeCache.txt has those flags in some obscure variable PC_LIBXML_STATIC_LIBRARIES:INTERNAL but removing them there doesn't affect compile/link command.
Note that this question is not about dealing with libicu specifically but about a method for investigation in general (though comments about eventual known problems with libicu would be appreciated too).
I found out that dependency graphs created by cmake can have more details that was configured for our project. Here are all options: https://cmake.org/cmake/help/latest/module/CMakeGraphVizOptions.html I expect GRAPHVIZ_EXTERNAL_LIBS, GRAPHVIZ_SHARED_LIBS are most important to set to true.
We enabled everything that was possible to enable, filtered out nothing and resulting graph was massive (to big for xdot - luckily .dot files are human readable), but showed that Boost::regex uses those 3 libraries.
I have a module module1 in a file called mymodule.f90. What should I do in order to make module1 usable like fortran intrinsic module?, i.e. it need only be called in a use statement (use module1) in any programs, subroutines, or functions that use it but I don't need to link /path/to/mymodule/ when compiling those procedures.
I use gfortran, but possibly in the future I will also have to use the Intel fortran compiler.
So maybe I'm misunderstanding you, but you want to use a module without having to tell the compiler where to find the .mod file (that contains the interface definitions for whatever module1 exports), or the linker where the object code can be found?
If so, for GFortran the solution is to download the GCC source code, add your own module as an intrinsic module, and then build your own custom version of GFortran. As a word of warning, unless you're familiar with the GFortran/GCC internals, while this isn't rocket science, it isn't trivial either.
For Intel Fortran, where you presumably don't have access to the source code of the compiler, I suppose you're out of luck.
My suggestion is to forget about this, and instead tell the compiler/linker where your .mod files and object files can be found. There are tools like make, cmake etc. that can help you automate this.
When you compile mymodule.f90 you will obtain an object file (mymodule.o) and a module file (mymodule1.mod). The compiler needs to have access to the module file when it compiles other files that use mymodule1, and the linker needs to have access to the object file when it generates the binary.
You don't need to specify the location of intrinsic modules because they are built in into the compiler. That will not be the case with your modules: you may be able to set up your environment in a way that the locations of your files allow the compiler to find the files without explicitly specifying their paths in compilation or linking commands, but the fact that you don't see it does not mean it's not happening.
For the Intel compiler, the answer is given by https://software.intel.com/en-us/node/694273 :
Directories are searched for .mod files in this order:
1 Directory of the source file that contains the USE statement.
2 Directories specified by the module path compiler option.
3 Current working directory.
4 Directories specified by the -Idir (Linux* and OS X*) or /include (Windows*) option.
5 Directories specified with the CPATH or INCLUDE environment variable.
6 Standard system directories.
For gfortran I have not found such a clear ordered list, but relevant information can be found in
https://gcc.gnu.org/onlinedocs/gfortran/Directory-Options.html
https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html#Directory-Options
It should be clear to you that a compiler won't be able to understand module files created by other compilers, or even by different enough versions of the same compiler. Therefore, you would need a copy of your "always available" module for each compiler you use, and if you are using multiple versions of a compiler you may need up to one per version - each of them in a different directory to avoid errors.
As you can see, this is not particularly practical, and it is indeed far from common practice. It is usually easier and more clear to the user to specify the path to the relevant module file in the compilation command. This is quite easy to set up if you compile your code using tools such as make.
Finally, remember that, if you make such arrangements for module files, you will also need to make arrangements for the corresponding object files at the linking stage.
As far as I understand using bazel I can only produce libtensorflow_cc.so and libtensorflow_framework.so.
I need to produce static libs that are position independent (-fPIC) because I'll link them to a dynamic lib of my own later.
I found this answer which suggest the use of a Makefile included in the project.
I successfully used it to replace the libtensorflow_cc.so but what can I do to replace libtensorflow_framework.so?
Not an actual answer, but too long for a comment.
I managed to do something like what you mention using Bazel on Windows. In particular, I wanted to make a single wrapper DLL with one or two headers (limited in functionality) that I could move around easily. I'll write a summary of the things that I did; it's rather convoluted an customized for our needs, but maybe you find something useful.
I pass --config=monolithic to the bazel build command (besides any other option that you need). That will avoid modularizing the library and thus remove the dependency to a libtensorflow_framework.so (see
tools/bazel.rc).
The goal that I build is not any of the ones in the TensorFlow repository. Instead, I add a very small program that uses my wrapper as a new Bazel target (a C++ file plus my headers headers and a BUILD file). So all of TensorFlow had to be compiled beforehand in order to compile this final dummy program.
When I get that done, I take advantage of the fact that Bazel does already compile every subgoal as a static library. I check a file under the bazel-bin directory generated for my dummy program goal with a name ending .params - there I find the path of all the static libraries that were used to compile it.
I copy all of these intermediate static libraries to somewhere else. Also, I copy a bunch of headers I will need to compile my final wrapper (TensorFlow own's, but also Eigen, Protobuf and Nsync now too). I put all of this in a build area I have prepared before.
I use NMake Makefile to produce my custom DLL, using the static libraries, the copied headers and my own thin wrapper.
And that's about it, I think. I have an ugly Bash script I run on MSYS2 that does everything for me. Usually with every new release I need to tweak one or two things (some option in the configure script, some additional headers I need to copy, etc.), but I do get it to work in the end. It's quite a lot of fiddling though, so I'm not necessarily saying you should use the same approach (but feel free to ask for details about any step if you want).
Using the -2.params files #jdehesa mentioned and bazel verbose output (-s switch), you can even create a link command to eventually statically link these intermediate static libraries. I automated this process for Windows/Linux/macOS and included it to the vcpkg package manager. To use it just run vcpkg install tensorflow:x64-windows-static. If you're interested in the sources, you'll find them here.
I'm trying to clean up a fortran make process for distribution. Currently, two libraries are made, and then the executable is compiled linking to the libraries and including the module files. I see from previous answers (Distribute compiled fortran library with module files) that you can't get rid of the module files and that they can be different for every machine and compiler. This is very annoying.
However, the code in my libraries are made up entirely of modules. It seems like I don't need the library part at all; I can just include the modules. I've tried this and it does compile and run on small examples.
Will this always work (when all I have are modules in the libraries)? Is it best practice? Should I instead consider rewriting my libraries NOT to use modules so I can avoid all these compiler dependencies and only distribute the lib*.a files? Is that what this document is referring to by using submodules (which no one supports static lib with many modules)
It really depends on the features you have in your library. Does it have only a couple of declarations? Then the .mod files would suffice, but why not distribute the source in such a simple case?
Are all your public procedures simple enough, so that they do not require an explicit interface and they are outside of modules? Then you don't need any .mod files.
Do you have a simple public module or an include file with the public API and the rest is private? You can then distribute the source of the API module or the include file. I would recommend to place just the interface blocks and other declarations in this module.
Be aware of one important problem. You can get away (using interface locks or similar) with avoiding the non-portable .mod files, but if the procedures are using some more advanced argument passing, their ABI is often NOT portable between different compilers or even some compiler versions. You would the be able to compile it and get mysterious crashes when calling your library.
Submodules can change it all, but actually I do not expect they will solve portability between compilers. The user of your library will still need the same compiler you had. It is true that interfacing the closed source software will be easier, but not more portable between compilers.
You can link either from a library lib*.a, or from object files. Both will be at least platform dependent and so more difficult to distribute than source code. library file might have the advantage of fewer files. In either case, linking from lib*a or object files, you can present your code to the user as a library of procedures to call. If you don't want to distribute your source code, then you will have to compile for however many platforms you support. Modules are a major advantage of modern Fortran, automating the checking of procedure actual and dummy arguments. Compared to, for example, C header files, they have the advantage of being automatic, but the disadvantage of producing a compiler-dependent intermediate file. If you are providing procedures to other programmers, it would seem a bad idea not to provide them with this interface checking. If you want to hide your source code, then you could write interface blocks describing the procedures and distribute only this source for them to compile.
I am trying to run sample Fortran code on Xcode 4.3 using a 64-bit compiler and it will not build correctly. The main problem is that despite my best efforts, I cannot get the separate .f90 files to interact with each other, thus code like
USE ElementModule, ONLY : ElementType
will not work. Does anybody have any answers regarding how to get the separate .f90 files to read each other. I'm aware you have to include specific modules, but my search hasn't given me any straight answers regarding what those specific modules are.
Normally when F90 code compiles, it generates 2 files: an object file and a mod file. When compiling subsequent modules, the mod files are used for the USE statements.
If you have circular dependency, then you have to build two or more times. Best to avoid circular dependency if you can avoid it.
The mod files are normally picked up by the same directive that tells the compiler where the include files are.