In CMake, what are the different possible values of CMAKE_SYSTEM_PROCESSOR? At least, the values for common processor families by AMD, Intel, Apple, Qualcomm and such?
I couldn't find this information in the CMake documentation.
According to the documentation, "when not cross-compiling, this variable has the same value as the CMAKE_HOST_SYSTEM_PROCESSOR variable". In the former scenario, the variable is set by the toolchain file, which I assume is what you're interested in doing.
In the latter case, the documentation says that CMAKE_HOST_SYSTEM_PROCESSOR is determined by inspecting the environment in the following way:
On Windows, the value of the PROCESSOR_ARCHITECTURE environment variable is used.
The options are: AMD64, IA64, ARM64, EM64T, X86. Source: this SuperUser answer.
On macOS, the value of uname -m is used by default. However, since this might vary based on whether you're using x86 or ARM CMake, version 3.19.2+ will use the value of CMAKE_APPLE_SILICON_PROCESSOR (either CMake or environment variable) instead, if it is set. It also normalizes Power Macintosh to powerpc.
As far as I am aware, the possible values here are x86_64, arm64, and powerpc.
On OpenBSD, it uses the arch -s command.
Not sure what the full list is here. Unfortunately, the man page doesn't have anything to say about this.
On Linux and related systems (Cygwin, MSYS, Android, GNU), it uses uname -m
This has many possible values. See: https://stackoverflow.com/a/45125525/2137996
Otherwise, it looks for the uname command and tries uname -p first. If it returns a non-zero exit status, it resorts to uname -m
No telling what this could be.
But what really matters is how CMake will use the value of CMAKE_SYSTEM_PROCESSOR. Here are the functions I'm aware of:
The default value of CPACK_SYSTEM_NAME is ${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR}.
The language modules Modules/CMake<LANG>Information.cmake all optionally include platform modules suffixed with -${CMAKE_SYSTEM_PROCESSOR}.cmake
In most cases, no such modules exist. On Android, very many platform modules exist with processor-specific variants.
The FindJNI module uses it to guide its search
It is passed to ARMClang via --mcpu (compile) and --cpu (link)
Related
After How to solve "FATAL: kernel too old" when running gem5 in syscall emulation SE mode? I managed to run a statically linked hello world under certain conditions.
But if I try to run an ARM dynamically linked one against the stdlib with:
./out/common/gem5/build/ARM/gem5.opt ./gem5/gem5/configs/example/se.py -c ./a.out
it fails with:
fatal: Unable to open dynamic executable's interpreter.
How to make it find the interpreter? Hopefully without copying my cross' toolchain's interpreter on my host's root.
For x86_64 it works if I use my native compiler, and as expected strace says that it is using the native interpreter, but it does not work if I use a cross compiler.
The current FAQ says it is not possible to use dynamic executables: http://gem5.org/Frequently_Asked_Questions but I don't trust it, and then these presentations mention it:
http://www.gem5.org/wiki/images/0/0c/2015_ws_08_dynamic-linker.pdf
http://research.cs.wisc.edu/multifacet/papers/learning_gem5_tutorial.pdf
but not how to actually use it.
QEMU user mode has the -L option for that.
Tested in gem5 49f96e7b77925837aa5bc84d4c3453ab5f07408e
https://www.mail-archive.com/gem5-users#gem5.org/msg15582.html
Support for dynamic linking has been added in November 2019
At: https://gem5-review.googlesource.com/c/public/gem5/+/23066
It was working for sure at that point, but then it broke at some point and needs fixing.....
arm 32-bit https://gem5.atlassian.net/browse/GEM5-461
arm 64-bit https://gem5.atlassian.net/browse/GEM5-828
If you have a root filesystem to use, for example one generated by Buildroot you can do:
./build/ARM/gem5.opt configs/example/se.py \
--redirects /lib=/path/to/build/target/lib \
--redirects /lib64=/path/to/build/target/lib64 \
--redirects /usr/lib=/path/to/build/target/usr/lib \
--redirects /usr/lib64=/path/to/build/target/usr/lib64 \
--interp-dir /path/to/build/target \
--cmd /path/to/build/target/bin/hello
Or if you are using an Ubuntu cross compiler toolchain for example in Ubuntu 18.04:
sudo apt install gcc-aarch64-linux-gnu
aarch64-linux-gnu-gcc -o hello.out hello.c
./build/ARM/gem5.opt configs/example/se.py \
--interp-dir /usr/aarch64-linux-gnu \
--redirects /lib=/usr/aarch64-linux-gnu/lib \
--cmd hello.out
You have to add any paths that might contain dynamic libraries as a separate --redirect as well. Those are enough for C executables.
--interp-dir sets the root directory where the dynamic loader will be searched for, based on ELF metadata which says the path of the loader. For example, buildroot ELF files set that path to /lib/ld-linux-aarch64.so.1, and the loader is a file present at /path/to/build/target/lib/ld-linux-aarch64.so.1. As mentioned by Brandon, this path can be found with:
readelf -a $bin_name | grep interp
The main difficulty with syscall emulation dynamic linking, is that we want somehow:
linker file accesses to go to a magic directory to find libraries there
other file accesses from the main application to go to normal paths, e.g. to read an input file in the current working directory
and it is hard to detect if we are in the loader or not, especially because this can happen via dlopen in the middle of a program.
The --redirects option is a simple solution for that.
For example /lib=/path/to/build/target/lib makes it so that if the guest would access the C standard library /lib/libc.so.6, then gem5 sees that this is inside /lib and redirects the path to /path/to/build/target/lib/libc.so.6 instead.
The slight downside is that it becomes impossible to actually access files in the /lib directory of the host, but this is not common, so it works in most cases.
If you miss any --redirect, the dynamic linker will likely complain that the library was not found with a message of type:
hello.out: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
If that happens, you have to find the libstdc++.so.6 library in the target filesystem / toolchain and add the missing --redirect.
It later broke at https://gem5.atlassian.net/browse/GEM5-430 but was fixed again.
Downsides of dynamic linking
Once I got dynamic linking to work, I noticed that it actually has the following downsides, which might or not be considerable depending on the application:
the dynamic linker has to run some instructions, and if you have a very minimal userland test executable, and are running on a low CPU like O3, then this startup can dominate runtime, so watch out for that
ExecAll does not show symbol names for stdlib functions, you just get offsets from some random nearest symbol e.g. #__end__+274873692728. Maybe something along these lines would work: Debugging shared libraries with gdbserver but not sure
dynamically jumping to a stdlib function for the first time requires going through the dynamic linking machinery, which can create problems if you are trying to control a microbench.
I actually already hit this once: the dynamic version of a program was doing something extra that and that compounded with a gem5 bug broke my experiment, and cost me a few hours of debugging.
Interpreters like Python and Java
Python and Java are just executables, and the script to execute an argument to the executable.
So in theory, you can run them in syscall emulation mode e.g. with:
build/ARM/gem5.opt configs/example/se.py --cmd /usr/bin/python --options='hello.py arg1 arg2'
In practice however hugely complex executable like interpreters are likely to have syscalls that not yet implemented given the current state of gem5 as of November 2019, see also: When to use full system FS vs syscall emulation SE with userland programs in gem5?
Generally it is not hard to implement / ignore uneeded calls though, so give it a shot. Related threads:
Java: Running Java programs in gem5(or any language which is not C)
Python: 3.6.8 aarch64 fails with "fatal: syscall unused#278 (#278) unimplemented.", test setup
Old answer
I have been told that as of 49f96e7b77925837aa5bc84d4c3453ab5f07408e (May 2018) there is no convenient / well tested way for running dynamically linked cross arch executables in syscall emulation: https://www.mail-archive.com/gem5-users#gem5.org/msg15585.html
I suspect however that it wouldn't be very hard to patch gem5 to support it. QEMU user mode already supports that, you just have to point to the root filesystem with -L.
The cross-compiled binary should have an .interp entry if it's a dynamic executable.
Verify it with readelf:
readelf -a $bin_name | grep interp
The simulator is setup to find this section in main executable when it loads the executable into the simulated address space. If this sections exists, a c-string is set to point to that text (normally something like /lib64/ld-linux-x86-64.so.2). The simulator then calls glibc's open function with that c-string as the parameter. Effectively, this opens the dynamic linker-loader for the simulator as a normal file. The simulator then maps the file into the simulated address space in phases with mmap and mmap_fixed.
For cross compilation, this code must fail. The error message follows it directly if the simulator cannot open the file. To make this work, you need to debug the opening process and possibly the subsequent pasting of the loader into the address space. There is mechanism to set the program's entry point into the loader rather than directly into the code section of the main binary. (It's done through the auxiliary vector on the stack.) You may need to play around with that as well.
The interesting code is (as of 05/29/19) in src/base/loader/elf_object.cc.
I encountered this problem after I just cross compiled the code. You can try to add "--static" after the command.
I am porting some code over to windows and my cmake checks for the package Libavahi using
find_package(Libavahi)
I have the headers, dll, etc. but I'm not sure where to place these such that cmake will find them.
Where can I put these files to be found by cmake? They're in a folder called usr.
I see that the module path is specified using:
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
but I'm wondering if there is a default location that will be searched as well
The CMake manual fully specifies the rather complicated search order for the different find_* commands. Unfortunately, since Windows lacks a default directory structure à la /usr/local/lib, it is hard to come up with reasonable defaults here.
One of the most reliable ways of managing directories is through environment variable hints. You simply add an $ENV{MY_VAR} to the HINTS section of the find command and then document that environment variable in your project's readme. Most users that are capable of compiling a C++ program know how to use environment variables, and it is way more convenient than having to give the path on the command line every time (although it never hurts to leave that as an additional option).
For find_package CMake offers a special mechanism on Windows called the package registry. CMake maintains a list of package information in the Windows registry under HKEY_CURRENT_USER\Software\Kitware\CMake\Packages\. Packages build from source can register there using the export command. Other projects build later on the same machine will then be able to find that package without additional configuration. This is quite powerful if you need to build a lot of interdependent projects from source on the same machine.
Update: Starting with version 3.12, CMake now implicitly considers the <PackageName>_Root environment variable a HINT for every find_package call.
In the newer versions of cmake, you can use the --debug-find option to list the directories that cmake is searching through. Somethin like:
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=ON --debug-find .
I'm thinking to write a simple configure script (similar to autoconf one) which execs cmake. But before doing that I want to check if anyone knows of such an effort already. I wasn't able to find anything on google.
It should be able to support the basic autoconf configure flags (prefix, exec-prefix, bindir mostly).
Reason to do it is of course that there's a certain user expectancy to be able to do ./configure && make
Also not really an answer but too long for a comment:
After reading up about cmake / cpack, I can at least tell you this. Cmake expects to be present on the platform. Therefore CPack cannot generate the same type of ./configure scripts as autotools. The Autotools expect some shell to be present, which is essentially the same as cmake to be present. However since cmake also targets the Win environment, it cannot rely on a shell. That being said, CPack can provide source packages, which need to be installed with cmake in the usual manner.
Also this does not solve your problem, I do not recommend to write a tool for cmake. Cmake is able to use all these type of prefixes you are interested in. If the user wants to compile your program from scratch, he has to know at least the basics (e.g. setting variables) of your build system. This is also true for autotools. If you want to spare him the pain, you can provide binary .sh, .deb or .rpm packages, which can be easily built with cmake / cpack.
I've been setting up a test-suite along with advancing the application my team is writing and today I came across a problem in how I was going to test scripts that get run through our application. We allow the user to set a seed for the Random module to allow repeatability in results (this is very important in a scientific application), I use this in our test suite to compare the stdout/stderr from a set of scripts with 'approved' runs.
In updating those scripts I noticed that all the scripts failed on certain machines. I soon discovered this was due to the Random module from version 3.12.0 changing the core function for generating random bits. Currently we run a number of versions of OCaml (including 3.13) across a number of environments (win32, osx, linux) and we would prefer to test against different versions of OCaml.
I would like to replace the Random module from 3.12.1 into our distribution to allow for consistency regardless of the version of OCaml the user is compiling against. But, the naive approach to drop the Random module in the source directory reports that the compiler found two files that define a module named Random.
Any suggestions on solving this issue? I realize I can rename Random to XRandom and then use that module to define what I need or include the standard library random module, but that would require changing every function call and allows developers to continue to (accidentally) use Random instead of the overloaded version. Is there a way to select a specific random module during compilation? Or maybe some other option I'm unaware of.
EDIT:
I just took the Random module from OCaml 3.12.1 and dropped it into my project, on compilation via OCamlbuild I got the following error message during linking (this the same error with OCaml 3.13.0+dev8, and pretty much what I expected when I did it),
Error: Files random.cmx and /opt/ocaml-3.12.1/lib/ocaml/stdlib.cmxa
both define a module named Random
And the linking line is,
/opt/ocaml-3.12.1/bin/ocamlopt.opt dynlink.cmxa unix.cmxa str.cmxa bigarray.cmxa -I +camlp4 camlp4fulllib.cmxa -cclib -lcside -cclib -L. -ccopt -I -ccopt /usr/lib -cclib -L. -cclib -Wl,--no-as-needed -cc gcc -cclib -llapack -cclib -lblas -cclib -lgfortran -cclib -lz -cclib -lreadline -cclib -lm -cclib '' compileFlags.cmx all_sets.cmx random.cmx status.cmx timer.cmx sexpr.cmx xml.cmx bitSet.cmx lz.cmx fileStream.cmx methods.cmx utl.cmx cost_matrix.cmx alphabet.cmx primes.cmx fileContents.cmx numerical.cmx version.cmx scripting.cmx libgrappa.a libgzcaml.a libzlibstubs.a libcside.a -o scripting.native
You can indeed use your own XRandom module, and add module Random = XRandom at the top of your test files.
Unfortunately, I'm afraid there is no good and clean solution to your problem ...
I'm sitting in an environment which I have no real control over (it's not just me, so basically, I can't change the environment or it won't work for anyone else), the only thing I can affect is how the binary is built.
My problem is, the environment specifies an LD_LIBRARY_PATH containing a libstdc++ which is not compatible with the compiler being used. I tried compiling it statically, but that doesn't seem possible for g++ (version 4.2.3, seems to have been work done in this direction in later versions though which are not available, -static-libstdc++ or something like that).
Now I've arrived at using rpath to bake the absolute path name into the executable (would work, all machines it's supposed to run on are identical). Unfortunately it appears as though LD_LIBRARY_PATH takes precedence over rpath (resetting LD_LIBRARY_PATH confirmed that it's able to find the library, but as stated above, LD_LIBRARY_PATH will be set for everyone, and I cannot change that).
Is there any way I can make rpath take precedence over LD_LIBRARY_PATH, or are there any other possible solutions to my problem? Note that I'm talking about dynamic linking at runtime, I am able to control the command line at compile and link time.
Thanks.
Maybe you can use a shell wrapper which modifies the environment for just this program?
#!/bin/sh
LD_LIBRARY_PATH="/path/to/your/lib:${LD_LIBRARY_PATH}"
export LD_LIBRARY_PATH
exec /path/to/binary $#
This should overload the LD_LIBRARY_PATH before execution and then replace the wrapper with your binary via exec.
Would this help?
Linking against libstdc++.a is definitely possible, albeit tricky. Instructions here.
I am a bit skeptical of your claim that LD_LIBRARY_PATH takes precedence over the "baked in" DT_RPATH though -- at least on Linux and (I believe) Solaris, LD_LIBRARY_PATH is only looked at after DT_RPATH lookup has failed.