In OpenZFS source code, since gzip.c is located in zfs/module/zfs, is there a Zlib kernel module for ZFS? Can ZFS link with regular Zlib library to do GZIP compression?
Which platform do you care about? For Linux, you can see where this happens in the SPL in spl-zlib.c. It includes linux/zlib_compat.h, which is a wrapper around the kernel/zlib.h implementation in the Linux kernel.
Related
After How to solve "FATAL: kernel too old" when running gem5 in syscall emulation SE mode? I managed to run a statically linked hello world under certain conditions.
But if I try to run an ARM dynamically linked one against the stdlib with:
./out/common/gem5/build/ARM/gem5.opt ./gem5/gem5/configs/example/se.py -c ./a.out
it fails with:
fatal: Unable to open dynamic executable's interpreter.
How to make it find the interpreter? Hopefully without copying my cross' toolchain's interpreter on my host's root.
For x86_64 it works if I use my native compiler, and as expected strace says that it is using the native interpreter, but it does not work if I use a cross compiler.
The current FAQ says it is not possible to use dynamic executables: http://gem5.org/Frequently_Asked_Questions but I don't trust it, and then these presentations mention it:
http://www.gem5.org/wiki/images/0/0c/2015_ws_08_dynamic-linker.pdf
http://research.cs.wisc.edu/multifacet/papers/learning_gem5_tutorial.pdf
but not how to actually use it.
QEMU user mode has the -L option for that.
Tested in gem5 49f96e7b77925837aa5bc84d4c3453ab5f07408e
https://www.mail-archive.com/gem5-users#gem5.org/msg15582.html
Support for dynamic linking has been added in November 2019
At: https://gem5-review.googlesource.com/c/public/gem5/+/23066
It was working for sure at that point, but then it broke at some point and needs fixing.....
arm 32-bit https://gem5.atlassian.net/browse/GEM5-461
arm 64-bit https://gem5.atlassian.net/browse/GEM5-828
If you have a root filesystem to use, for example one generated by Buildroot you can do:
./build/ARM/gem5.opt configs/example/se.py \
--redirects /lib=/path/to/build/target/lib \
--redirects /lib64=/path/to/build/target/lib64 \
--redirects /usr/lib=/path/to/build/target/usr/lib \
--redirects /usr/lib64=/path/to/build/target/usr/lib64 \
--interp-dir /path/to/build/target \
--cmd /path/to/build/target/bin/hello
Or if you are using an Ubuntu cross compiler toolchain for example in Ubuntu 18.04:
sudo apt install gcc-aarch64-linux-gnu
aarch64-linux-gnu-gcc -o hello.out hello.c
./build/ARM/gem5.opt configs/example/se.py \
--interp-dir /usr/aarch64-linux-gnu \
--redirects /lib=/usr/aarch64-linux-gnu/lib \
--cmd hello.out
You have to add any paths that might contain dynamic libraries as a separate --redirect as well. Those are enough for C executables.
--interp-dir sets the root directory where the dynamic loader will be searched for, based on ELF metadata which says the path of the loader. For example, buildroot ELF files set that path to /lib/ld-linux-aarch64.so.1, and the loader is a file present at /path/to/build/target/lib/ld-linux-aarch64.so.1. As mentioned by Brandon, this path can be found with:
readelf -a $bin_name | grep interp
The main difficulty with syscall emulation dynamic linking, is that we want somehow:
linker file accesses to go to a magic directory to find libraries there
other file accesses from the main application to go to normal paths, e.g. to read an input file in the current working directory
and it is hard to detect if we are in the loader or not, especially because this can happen via dlopen in the middle of a program.
The --redirects option is a simple solution for that.
For example /lib=/path/to/build/target/lib makes it so that if the guest would access the C standard library /lib/libc.so.6, then gem5 sees that this is inside /lib and redirects the path to /path/to/build/target/lib/libc.so.6 instead.
The slight downside is that it becomes impossible to actually access files in the /lib directory of the host, but this is not common, so it works in most cases.
If you miss any --redirect, the dynamic linker will likely complain that the library was not found with a message of type:
hello.out: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
If that happens, you have to find the libstdc++.so.6 library in the target filesystem / toolchain and add the missing --redirect.
It later broke at https://gem5.atlassian.net/browse/GEM5-430 but was fixed again.
Downsides of dynamic linking
Once I got dynamic linking to work, I noticed that it actually has the following downsides, which might or not be considerable depending on the application:
the dynamic linker has to run some instructions, and if you have a very minimal userland test executable, and are running on a low CPU like O3, then this startup can dominate runtime, so watch out for that
ExecAll does not show symbol names for stdlib functions, you just get offsets from some random nearest symbol e.g. #__end__+274873692728. Maybe something along these lines would work: Debugging shared libraries with gdbserver but not sure
dynamically jumping to a stdlib function for the first time requires going through the dynamic linking machinery, which can create problems if you are trying to control a microbench.
I actually already hit this once: the dynamic version of a program was doing something extra that and that compounded with a gem5 bug broke my experiment, and cost me a few hours of debugging.
Interpreters like Python and Java
Python and Java are just executables, and the script to execute an argument to the executable.
So in theory, you can run them in syscall emulation mode e.g. with:
build/ARM/gem5.opt configs/example/se.py --cmd /usr/bin/python --options='hello.py arg1 arg2'
In practice however hugely complex executable like interpreters are likely to have syscalls that not yet implemented given the current state of gem5 as of November 2019, see also: When to use full system FS vs syscall emulation SE with userland programs in gem5?
Generally it is not hard to implement / ignore uneeded calls though, so give it a shot. Related threads:
Java: Running Java programs in gem5(or any language which is not C)
Python: 3.6.8 aarch64 fails with "fatal: syscall unused#278 (#278) unimplemented.", test setup
Old answer
I have been told that as of 49f96e7b77925837aa5bc84d4c3453ab5f07408e (May 2018) there is no convenient / well tested way for running dynamically linked cross arch executables in syscall emulation: https://www.mail-archive.com/gem5-users#gem5.org/msg15585.html
I suspect however that it wouldn't be very hard to patch gem5 to support it. QEMU user mode already supports that, you just have to point to the root filesystem with -L.
The cross-compiled binary should have an .interp entry if it's a dynamic executable.
Verify it with readelf:
readelf -a $bin_name | grep interp
The simulator is setup to find this section in main executable when it loads the executable into the simulated address space. If this sections exists, a c-string is set to point to that text (normally something like /lib64/ld-linux-x86-64.so.2). The simulator then calls glibc's open function with that c-string as the parameter. Effectively, this opens the dynamic linker-loader for the simulator as a normal file. The simulator then maps the file into the simulated address space in phases with mmap and mmap_fixed.
For cross compilation, this code must fail. The error message follows it directly if the simulator cannot open the file. To make this work, you need to debug the opening process and possibly the subsequent pasting of the loader into the address space. There is mechanism to set the program's entry point into the loader rather than directly into the code section of the main binary. (It's done through the auxiliary vector on the stack.) You may need to play around with that as well.
The interesting code is (as of 05/29/19) in src/base/loader/elf_object.cc.
I encountered this problem after I just cross compiled the code. You can try to add "--static" after the command.
I'm having runtime errors when running Tensorflow programs. I tracked down the problem and the source seems to be that the shared object I have compiled includes CPU features that are not supported by my processor. That causes some pointers pointing to invalid addresses. Right now there are 37 CPU feature defined in tensorflow/core/platform/cpu_info.h. My question is how to not include some of those when compiling Tensorflow's source code.
There is a ./configure option for optimization flags to the compiler (CC_OPT_FLAGS). By default this is -march=native, which will try to include instructions supported by the current architecture. You can turn off some of these manually with e.g. -mno-avx to turn off AVX.
I'm cross-compiling for VxWorks using cmake. When I run cmake the first time I have to provide informations about compiler, target OS etc..
In the cross-compile dialogue there are three target system settings I set:
Operating System
Version
Processor
(followed by compiler etc.)
While I can retrieve the first one using CMAKE_SYSTEM_NAME, i can't get the version and the processor.
Both return an empty string.
Here's an example:
MESSAGE("CMAKE_SYSTEM_PROCESSOR: ${CMAKE_SYSTEM_PROCESSOR}")
MESSAGE("CMAKE_SYSTEM_VERSION: ${CMAKE_SYSTEM_VERSION}")
Output:
CMAKE_SYSTEM_PROCESSOR:
CMAKE_SYSTEM_VERSION:
My Cmake Version is 2.8.10.2 and target OS is VxWorks (if this matters - compiler are WindRiver GNU).
How can I get the version and processor I've set in the beginning? Or is this impossible if I cross-compile to an OS that's unknown to cmake?
(Btw. Compiling works fine)
It seems this is not possible so far. I'm getting empty strings all the time.
However, there's a working solution, and i guess it's the better way:
Before:
I specified cross-compile settings (Compiler and target system, see question), then it runs over VxWorks specific parts in the CMake list (checked with if( VxWorks ) to ensure it's not executed when other systems are used).
Now (Solution):
I wrote a toolchain file and platform files for VxWorks and required processors.
Cons:
I have to write some extra files:
Toolchain file
Platform file for VxWorks
Further Platform files for each Processor (and processor type, Gnu and Diab)
Pros:
CMake list is much cleaner now
Separate Project and Target settings
Separate System and processor settings - easy to add new Processors in a very clear way but keep System settings
I write some settings in the toolchain file and CMake loads related system / processor settings
...
protoc.exe is built for windows. Every other distro, you have to compile protoc yourself to use it. We really want to drop something like protoc into our project but then have the hassle on anyone on a different OS has alot of work to do to set it up. Therefore, for this project protoc is kind of dead basically.
Is there another library like protoc that generates objects/DTOs and marshallers/unmarshallers that has a cross-platform compiler? (Java compiler would be great as we could wire it into our gradle builds).
thanks,
Dean
Avro Project is similar to Protocol Buffers (has it's own format though) and has a Java Compiler. Avro definition can be in either IDL (not unlike Protocol Buffers) and JSon. One problem is the Jar for Avro is large (it uses a lot of other Jars
You could look a compiling protoc to Java-Byte-Code. There are several projects available for this see cibyl also lists several cross compilers. The result will be slow and large but it may work.
Protocol buffer are supposed to be cross-platform compiler
There are other alternatives to Protoc buffer such as xstream, Jackson, GSON. There is detailed set of answers Here.
Hope this helps!
I haven't tried if it works, put possibly you could run protoc over JVM. There exists several compilers for running C++ on JVM.
See Compiling C++ for the JVM.
I am new to OpenCL. I have written a vector addition code in OpenCL with help from Internet. I have included one header file i.e. CL/cl.h using #include.
I am using NVIDIA graphic card and the OpenCL implementation is NVIDIA_GPU_Computing_SDK. My OpenCL header files are residing at this path /opt/NVIDIA_GPU_Computing_SDK/OpenCL/common/inc. I can run OpenCL programs through linux terminal by adding this path when compiling my code. But now I want to write CMake file for this code. CMake files are working fine for C programs, but not OpenCL programs because of this Path problem. In terminal, I used to enter $cmake ., after this $make, it will search for a Makefile which is created by cmake, now my error is after entering command make
fatal error: CL/cl.h: No such file or directory!
Now tell me how can I include this header file into CMake file?
You will need to put these lines into CMakeLists.txt:
include_directories(/opt/NVIDIA_GPU_Computing_SDK/OpenCL/common/inc)
link_directories(/opt/NVIDIA_GPU_Computing_SDK/OpenCL/common/<lib or something similar>)
add_executable(yourexe src1.c ...)
target_link_libraries(yourexe OpenCL)
But beware that this is not portable, because OpenCL SDK can be somewhere else on another machine. The proper way to do this is to use FindOpenCL.cmake module.
Maybe you can use a CMake "find" script like:
http://gitorious.org/findopencl/findopencl/blobs/master/FindOpenCL.cmake
http://code.google.com/p/opencl-book-samples/source/browse/trunk/cmake/FindOpenCL.cmake?r=14
CMake file example from OpenCL Programming Guide Book: http://code.google.com/p/opencl-book-samples/source/browse/trunk/CMakeLists.txt?r=14
I was looking for FindOpenCL.cmake macro which would work well on Windows, OSX and Linux... I couldn't find any which did work well on every platform, so I wrote new one which I use in couple of projects (webcl-validator and opencl-testsuite).
https://github.com/elhigu/cmake-findopencl
Especially Windows support is improved in this one.
In Windows it checks if 64bit or 32bit lib should be used and it also tries to find libraries from according to environment variables set by Nvidia, Intel and AMD OpenCL SDKs.
It also tries to find .lib in Cygwin, which didn't work with other scripts I tried.