I have built a Singularity sandbox container using this command:
sudo singularity build --sandbox ubuntu/ library://ubuntu
Now, I would like to copy/export this container as a sif file. But I cannot find how to do this in the documentation.
Any idea ?
Ok, so by reading the doc more carefully, it's apparently not possible to keep the changes made on a sandbox into a sif file, see here
sudo singularity build ubuntu.sif ubuntu/
INFO: Starting build...
INFO: Creating SIF file...
INFO: Build complete: ubuntu.sif
See https://docs.sylabs.io/guides/3.5/user-guide/build_a_container.html#converting-containers-from-one-format-to-another
Related
Could someone please let me know how one can access files in /home within a singularity container?
I created a docker image. In this image, some packages are built and installed under /home. Some of those are also added to PYTHONPATH within the docker image. If I run the image, then a docker container is created. Within this container I can access all files under /home and use the Python modules that I added. This is a fully working docker image.
I wanted to use the packages and Python modules on a HPC system. So, I converted the docker image to a singularity image. Then, I used the singularity shell <image_name.sif> command to access the shell in the container. After that I see the prompt below.
Singularity> cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Singularity>
The host OS on the HPC system is Red Hat Linux. Since the /etc/*-release command shows Ubuntu, it seems like the /etc directory is the one inside the container. This looks reasonable. However, when I type ls /home, then I see the contents of /home on the host OS. Howe could I find the files in /home within the container?
If I type any commands to run the packages installed in /home within the container, then the singularity shell prints command not found. Also, if I run the Python interpreter, then I cannot import any modules installed within the container. Although the Python version matches the one in the container, the modules are not located. The PYTHONPATH includes paths like /home/<a_directory_name>, but the Python interpreter cannot locate the modules. Even though the docker image is fully functional, the corresponding singularity image is completely useless.
How could I use the packages and Python modules installed in /home in the singularity container?
By default Singularity automatically mounts $HOME into the container, which will shadow anything that was installed there during image creation.
To skip this, use the --no-home flag when running your singularity command. Additional options, such as mounting home to a different location, are described in the online and CLI documentation.
Currently I'm trying to build container serving VueJS application via Cloud Native Buildpacks.
I already have working Docker file that builds VueJS in production mode and then copy results to nginx image, but I would like to try to use CNB.
So I just have created empty VueJS project for test via vue create vue-tutorial and trying to do with CNB somehting like described there https://cli.vuejs.org/guide/deployment.html#heroku but using CNB.
Does anyone know working recipe how to do that with CNB?
P.S. Currently I'm trying to build that with
pack build spa --path . \ SIGINT(2) ↵ 17:22:41
--buildpack gcr.io/paketo-buildpacks/nodejs \
--buildpack gcr.io/paketo-buildpacks/nginx
but getting next error (and I'm not sure that I'm on right way):
===> DETECTING
ERROR: No buildpack groups passed detection.
ERROR: Please check that you are running against the correct path.
ERROR: failed to detect: no buildpacks participating
ERROR: failed to build: executing lifecycle: failed with status code: 100
UPD
My current dockerfile
# build stage
FROM node:lts-alpine as build-stage
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
# production stage
FROM nginx:1.19-alpine as production-stage
COPY --from=build-stage /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
We chatted about this in Slack, but I wanted to capture it here too:
pack build --buildpack heroku/nodejs --buildpack https://cnb-shim.herokuapp.com/v1/heroku-community/static yourimage
This command may do what you want. The static buildpack used in that example is not yet converted to a cloud native buildpack, but the shim may allow you to build a workable artifact. Then run your image with something like docker run -it -e PORT=5000 -p 5000:5000 yourimagename
I am working on CentOS 8 and am using Singularity 3.6.2. I have a Singularity recipe file :
BootStrap: yum
OSVersion: 8
MirrorURL: http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/
Include: yum
%files
/gpfs0/home1/group/user/path/to/some.rpm /tmp
%post
ls /tmp
echo "Hello from inside the container"
When I run :
$ sudo singularity build test.simg tmp
INFO: Starting build...
INFO: Skipping GPG Key Import
INFO: Adding owner write permission to build path: /tmp/rootfs-4db1e756-22a8-11eb-bb20-34800d2d90f0
INFO: Copying /gpfs0/home1/group/user/path/to/some.rpm to /tmp/rootfs-4db1e756-22a8-11eb-bb20-34800d2d90f0/tmp
INFO: Running post scriptlet
+ ls /tmp
qtsingleapp-RStudi-c679-6387e228-lockfile
rootfs-4db1e756-22a8-11eb-bb20-34800d2d90f0
rootfs-b10ad12c-229a-11eb-85a3-34800d2d90f0
+ echo 'Hello from inside the container'
Hello from inside the container
INFO: Creating SIF file...
According to the Singularity documentation
In the default configuration, the system default bind points are $HOME , /sys:/sys , /proc:/proc, /tmp:/tmp,
Question :
Why is the %files section putting my rpm in /tmp/rootfs-4db1e756-22a8-11eb-bb20-34800d2d90f0/tmp and not in /tmp? That seems to contradict the documentation. This is also different from the behavior observed with Singularity v2.5.1
Also, how would I access said file. The long 'hash-like' part of the path seems to change depending on the build?
I don't have an answer reconciling the documentation with where the %files section is actually putting the files, however I do have an answer for how to access the files copied. You need to use ${SINGULARITY_CONTAINER} in the %post section.
E.g.
$ cat Singularity
BootStrap: yum
OSVersion: 8
MirrorURL: http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/
Include: yum
%files
# Will need to use environmental variables to copy the code to
/gpfs0/home/group/user/path/to/some.rpm /tmp
%post
ls ${SINGULARITY_CONTAINER}/tmp
echo "Hello from inside the container"
When building yields :
$ sudo singularity build tmp.simg tmp
INFO: Starting build...
INFO: Skipping GPG Key Import
INFO: Adding owner write permission to build path: /tmp/rootfs-e2a3fbb4-242b-11eb-a267-34800d2d90f0
INFO: Copying /gpfs0/home/group/user/path/to/some.rpm to /tmp/rootfs-e2a3fbb4-242b-11eb-a267-34800d2d90f0/tmp
INFO: Running post scriptlet
+ ls /tmp/rootfs-e2a3fbb4-242b-11eb-a267-34800d2d90f0/tmp
some.rpm
+ echo 'Hello from inside the container'
Hello from inside the container
INFO: Creating SIF file...
I have a singularity container that has been made for me (to run tensorflow on comet GPU nodes) but I need to modify the keras install for my purposes.
I understand that .simg files are not editable (and that the writable .img format is deprecated), so the process of converting to an .img file, editing, and then converting back to .simg is discouraged:
sudo singularity build --writable development.img production.simg
## make changes
sudo singularity build production2.img development.simg
It seems to me the best way might be to extract the contents (say into a sandbox), edit them, and then rebuild the sandbox into an .simg image.
I know how to do the second conversion (singularity build new-sif sandbox), but how can I do the first?
I have tried the following, but the command never finishes:
sudo singularity build tf_gpu tensorflow-gpu.simg
WARNING: Authentication token file not found : Only pulls of public images will succeed
Build target already exists. Do you want to overwrite? [N/y] y
2018/10/12 08:39:54 bufio.Scanner: token too long
INFO: Starting build...
You can easily convert between a sandbox and a production build using the following:
sudo singularity build lolcow.sif docker://godlovedc/lolcow # pulls and builds an example container
sudo singularity build --sandbox lolcow_sandbox/ lolcow.sif # converts from container to a writable sandbox
sudo singularity build lolcow2 lolcow_sandbox/ # converts from sandbox to container
So, you can edit the sandbox and then rebuild accordingly.
In official docs we can see:
# docker build github.com/creack/docker-firefox
It just works fine to me. docker-firefox is a repository and has Dockerfile within root dir.
Then I want to buid redis image and exact version 2.8.10 :
# docker build github.com/docker-library/redis/tree/99c172e82ed81af441e13dd48dda2729e19493bc/2.8.10
2014/11/05 16:20:32 Error trying to use git: exit status 128 (Initialized empty Git repository in /tmp/docker-build-git067001920/.git/
error: The requested URL returned error: 403 while accessing https://github.com/docker-library/redis/tree/99c172e82ed81af441e13dd48dda2729e19493bc/2.8.10/info/refs
fatal: HTTP request failed
)
I got error above. What's the right format with build docker image from github repos?
docker build url#ref:dir
Git URLs accept context configuration in their fragment section,
separated by a colon :. The first part represents the reference that
Git will check out, this can be either a branch, a tag, or a commit
SHA. The second part represents a subdirectory inside the repository
that will be used as a build context.
For example, run this command to use a directory called docker in the
branch container:
docker build https://github.com/docker/rootfs.git#container:docker
https://docs.docker.com/engine/reference/commandline/build/
The thing you specified as repo URL is not a valid git repository. You will get error when you will try
git clone github.com/docker-library/redis/tree/99c172e82ed81af441e13dd48dda2729e19493bc/2.8.10
Valid URL for this repo is github.com/docker-library/redis. So you may want to try following:
docker build github.com/docker-library/redis
But this will not work too. To build from github, docker requires Dockerfile in repository root, howerer, this repo doesn't provide this one. So, I suggest, you only have to clone this repo and build image using local Dockerfile.
One can use the following example which sets up a Centos 7 container for testing ORC file format. Make sure to escape the # sign:
$ docker build https://github.com/apache/orc.git\#:docker/centos7 -t orc-centos7