For NVIDIA Collective Communications Library (NCCL) version 2, NVIDIA asks the user to first register as a developer before getting access to the installation files.
This will bring a challenge on how to install NCCL in the containers. For personal use, we can copy the installation file to the container using Dockerfile ADD command. However, this approach does not seem right for a Dockerfile to be used by others (or put in public).
Any idea?
Thanks!
I had a similar problem with oracle installation files, the only way I could think of doing this was to ask the user to manually download the files and then using Dockerfile ONBUILD command along with the Dockerfile ADD command within the dockerfile. Meaning every user will essentially have to build their own image but at least the image can be made public without infringing on NVIDIA's policies.
Something like this:
FROM example/test:latest
....
ONBUILD ADD /example/nvidia /example/nvidia
....
CMD ['./foo.sh']
Then the user would have to use their own dockerfile pulling your public image like so:
FROM myrepo/myimage:nvidia
Provided they have the NVIDIA Collective Communications Library placed in the right folder, they can just run docker build to legally have their own image with Nvidia's libraries.
Related
I am a beginner with Singularity.
What I want to achieve in the long run: I have a programming project with a long lists of dependencies, and I want to be able to give the program to other people in my company without there being bugs caused by missing dependencies, or wrong versions of dependencies.
The idea was now to use Singularity in order to easily provide a working environment.
In order to test this, I wrote a Hello World application which I now want to run in a container. I have a folder HelloWorld/ which contains the source code for a C++ Qt project. Then I wrote the following recipe file:
project.recipe
Bootstrap: docker
From: ubuntu:18.04
%setup
cp -R <some_folder>/HelloWorld ${SINGULARITY_ROOTFS}/HelloWorld
%post
apt update
apt-get install -y qt5-default
apt install -y g++
apt-get install -y build-essential
cd HelloWorld
qmake
make
echo "after build:"
ls
%runscript
echo "before execution:"
ls HelloWorld/
./HelloWorld/HelloWorld
where the echos and directory listings are for my current debugging process.
I can sucessfully build an image file using sudo singularity build --writable project.img project.recipe. (My debugging output shows me that the executable was build successfully.)
The problem is now that if I try to run it using ./project.img, or singularity run project.img, it won't find the executable.
Using my debugging output, I found out that the lines in %runscript use the folders outside of the container.
Tutorials like https://sylabs.io/guides/3.1/user-guide/build_a_container.html made it seem to me as if my recipe was the way to go, but apparently it isn't?
My questions:
Is there some way for me to access my executable? Am I calling it wrong?
Is the way I do it the way it is supposed to be done? Or would one normally do something like getting the executable outside of the container and then use the container to call that outside file? Or is there a different best practice?
If the executable is to be copied outside of the container after compilation, how do I do that? How do I access outside folders when I'm within %post?
Is this the best work process for what I want to achieve? Later on, my idea is that the big project is copied likewise in the container, dependencies are either installed or copied, then the project is compiled and finally its source being deleted. I also considered using a repository, but I can't have the project being in an open repository, and I don't want to store any passwords.
Firstly, use %files, don't use %setup. %setup is run as root and can directly modify the host server. You can very easily and accidentally break things without realizing it. You can get the same effect this way:
%files
some_folder/HelloWorld /HelloWorld
You are calling it wrong. In your %setup (and hopefully now in your %files) steps, you are copying the data to /HelloWorld. In your %runscript your are calling ./HelloWorld/HelloWorld which is the equivalent of $PWD/HelloWorld/HelloWorld. Since singularity automatically mounts in $PWD (as well as $HOME and some other directories), you are not calling what you're trying to call.
You don't copy the executable outside of the container, you just need to make sure what you're executing is where you think it is.
There is no access to the host filesystem in %post, you should have everything you need copied in via %files first.
That's a reasonable workflow. Having a local private repo for the code is probably a good idea for tracking your changes, but that's your call.
Two computers are working on the SAME repository but first computer detect the library and work well but second computer not detect it and show "Error 'PhpOffice\Phpspreadsheet\Reader\Xlsx' not found".
In vendor, the library also exist.
composer.json and composer.lock also the same on both computer.
One thing is that by git ignore, I use yii's composer mechanism at 1st computer but at sec computer(err computer), I add library manually.
If you want to use a composer package, you absolutely need to install it using composer. This ensures that the autoloader is generated properly and your class can be found through PHP.
Copying library files into vendor directory is not enough to install it. During installation Composer creates autoload script with information how to find all classes installed by Composer. If you just copy library files, Composer will not even know that it exist and will not able load any class from it.
If you cannot use Composer on server/computer A, you should install all dependencies on different computer (B) and copy the entire vendor directory into server/computer A. Autoload definitions are in vendor so it should work if you copy the whole dorectory.
I have a run configuration in PyCharm with script parameters that take in a directory. Then, I have several directories (a changing number) in my project which I would like to easily be able to run this configuration on. Is there anyway to add an option to the right-click menu of directories to run the configuration passing that directory to the configuration? Or some other method which provides similar accessibility to running the configuration on a directory?
For my specific problem, I have many log directories for TensorBoard (from TensorFlow) and I would like to selectively and easily be able to start up an instance of TensorBoard running on a given directory.
Basically if you would run it in pycharm You can write your own plugin to add some functionality to the IDE. There is a great documentation on how to create plugins in idea/pycharm:
http://www.jetbrains.org/intellij/sdk/docs/basics/getting_started.html
We have done it before and it was a successful plugin, it really sped up the development processs. :)
I use the deb file for production and the source for development.Is this the correct way to do things?
I think that the deb might have certain optimizations(pyo or pyc) for production environment.
But since I have to move my custom modules, one at a time to the production,I find it increasingly difficult.
The actual addons path is here
(1) /usr/share/pyshared/openerp/addons
But the init.d points to
(2) /usr/lib/pymodules/python2.7/openerp/addons
In some modules the __init__.py is in 1 (eg: web_rpc)
and for some its in 2 (eg: hr)
What the actual difference btw
http://nightly.openerp.com/6.1/nightly/src/
and
http://nightly.openerp.com/6.1/nightly/deb/
I haven't tried the deb files, because we use the Ubuntu all-in-one script from openerpappliance.com. It downloads the source from Launchpad and then runs the deployment scripts for you. It will also do updates after you've installed.
We're very happy with the 5.0 version, but we haven't tried the 6.1 version, yet.
you can do with 6.1 is you can give multiple addons path to the your config file in comma separated , or else you can create link in addons folder for your customized folder while you can keep cumized module where you want, just put link(shortcut) to the addon sfoderl of your. this will give you flexibility.
Thank YOu
I am starting to learn Singularity for reproducible analysis of scientific pipelines. A colleague explained that an image was used to instantiate a container. However, in reading through the documentation and tutorials, the term instance is also used and the usage of image and container seems somewhat interchangeable. So, I am not sure I precisely understand the difference between an image, container, and instance. I do get that a recipe is a text file for building one of these (I think an image?).
For example, on this page it explains:
Now we can build the definition file into an image! Simply run build
and the image will be ready to go:
$ sudo singularity build url-to-pdf-api.img Singularity
Okay, so this uses the recipe Singularity to build an image, with the intuitive extension of .img. However, the help description of the build command states:
$ singularity help build
USAGE: singularity [...] build [build
options...]
The build command
compiles a container per a recipe (definition file) or based on a URI,
location, or archive.
So this seems to indicate we are building a container?
Then, there are image and instance sub-commands.
Are all these terms used interchangeably? It seems sometimes they are and sometimes there is a difference between them.
A container is the general concept of creating a sandboxed run environment and can be used as a general term to refer to either Docker or Singularity images. However it is sometimes used to also refer to the specific files being generated. This is probably not ideal, as it can clearly cause confusion to new users.
image is generally used to to refer to the actual files created by singularity build ...
instance refers to a specific way of running singularity images. Normally, if you singularity run some_image.sif or singularity some_image.sif some_command you can't easily access its environment while it's running. However, if you instead run singularity instance start some_image.sif some_instance1 it creates a persistent service that you can access like a docker container. The singularity service/instance documentation has some good examples of how instances are used differently than the basic exec and run commands.