Problems getting Singularity Compose to work - singularity-container

I wrote a small test project for Singularity Compose, consisting of a small server application, with the following YAML file:
version: "1.0"
instances:
server:
build:
context: ./server
recipe: server.recipe
ports:
- 9999:9999
When I call singularity-compose build, it successfully builds server.sif. Calling singularity-compose up also seemingly works without error, and calling singularity-compose ps results in something that looks just fine:
+ singularity-compose ps
INSTANCES NAME PID IMAGE
1 server 4176911 server.sif
However, the server application does not work, calling my test client results in it saying that there is no answer from the server.
But if I run server.sif directly without compose, everything works just fine.
Also, I tripple checked, my test application listens to port 9999, thus should be reachable from the outside.
What did I do wrong?
Edit:
I also checked whether there actually is any process listening at port 9999 by calling sudo lsof -i -P -n | grep LISTEN, this is not the case. Only when I manually start server.sif without compose it shows me the process listening.
Edit:
I went into the Singularity Compose shell and tried to start the Server application directly in there, just as a test, and it resulted in Permission denied. Not sure if that means anything.
Edit:
I now gave the application execution rights within the shell and called in there, this works. Am now trying to add execution rights in the recipe. If that works, it would be kind of strange, as the executable was build right there, and thus should already have execution rights.
Edit:
I added chmod +x in my recipe both after building Server and before executing it. Doesn't work either.
Also checked whether any bridges exist using brctl show, this is not the case.
Edit: My recipe, adjusted by the input of tsnowlan in his answer below:
Bootstrap: docker
From: ubuntu:20.04
%files
connection.cpp
connection.h
main.cpp
server.cpp
server.h
server.pro
%post
# get some basics
apt update
apt-get install -y wget
apt-get install -y software-properties-common
# get C++ compiler
apt-get install -y g++
apt-get install -y build-essential
apt-get install -y build-essential cmake
# get Qt
apt-get install -y qt5-default
# compile
qmake
make
ls
%runscript
/Server
%startscript
/Server
Again, note that the application works just fine both when compiled and startet normally and when started within a Singularity image (but without Singularity Compose).
The ls at the end of the %post block is used to verify that the Server application was build successfully.

Please share the server.recipe, as it is difficult to identify should be/is happening without it.
Without having that, my guess is that you have a %runscript in your definition file, but no %startscript. When the image is executed directly or via singularity run image.sif, the contents of %runscript determine what happens. To emulate the docker-compose style, the singularity images are started as persistent instances. In this case, the %startscript block determines what runs. If it is empty, it will just start up and sit there doing nothing. This would explain why when run by hand it works but not when using compose.

Related

Why can I access my Apache default page ONLY when I go in my container's bash?

First of all, I would like to say that I'm new to Docker and all that is around it.
I have been wanting to make a container where I have Apache, php and Firebird installed. So far, so good ; everything seems to work and I can get my default page when I type in my Internet search bar my ip address and :8080. I do so by first starting my container like this :
docker run -p 8080:80 -d apps
Where "apps" is the name of my container.
I have achieved this with my Dockerfile, which looks like this (it might be a bit messy, still learning the good practices !) :
# Download of base image - ubuntu 20.04
FROM ubuntu:20.04
# Updating/upgrading
RUN apt-get update -y && apt-get upgrade -y
# Installing apache2, php and firebird with modules
RUN DEBIAN_FRONTEND="noninteractive" apt-get install apache2 php libapache2-mod-php -y && \
apt-get install php-curl php-gd php-intl php-json php-mbstring php-xml php-zip -y && \
DEBIAN_FRONTEND="noninteractive" apt-get install firebird3.0-server -y && apt-get install firebird->
# Start up apache in foreground by default
CMD /usr/sbin/apache2 -D FOREGROUND
ENTRYPOINT service apache2 restart && /bin/bash
# Expose apache
EXPOSE 80
Now, my idea was to export this container to another computer and try the same thing. I have followed a few tutorials and got to import my container on the new machine. My problem here is that somehow, the command I previously used doesn't work ; it shows me this error :
docker: Error response from daemon: No command specified.
See 'docker run --help'.
Which is odd, because it works just fine on the other machine. I also did this command, WHICH WORKS :
docker run -i -t -p 8080:80 apps /bin/bash
This one works alright, but I don't want to have to access the bash everytime I want my Apache page to load. I would want my container to run without me having to get in my container, if that makes sense.
In my opinion, it probably comes from the fact that I only loaded the container, and not the image used to build it (maybe a bad practice? Couldn't find anything about it on google).
Here is my setup just in case ---
On the first machine (which is the one where I created the image and the container) :
Ubuntu 20.04 LTS
Apache/2.4.41
Docker 19.03.8
On the other machine which I'm trying to make my container work :
Ubuntu 18.04 LTS
Apache/2.4.29
Docker 19.03.6
Thank you for your patience and time !
apps is your docker image, if you want to give name for your container you can specify --name in the run command ie,
docker run --name container_name -p 8080:80 -d apps
You can use sudo docker save -o apps.tar apps to create a tar file of the image
then change the root permission of the tar file sudo chmod 777 apps.tar
Copy this tar file to the other system you want to try, then
sudo docker load --input apps.tar
This will load the image, then you can use the previous command to start the container
docker run -p 8080:80 -d apps
Where "apps" is the name of my container. <- This statement is incorrect and perhaps the misunderstood concept that leads you to the problem.
apps is the name of the image, not the name of the container. On the host on which you can run the container, you must have built that image from the Dockerfile that you shared using the command:
docker build -t apps .
Copy the Dockerfile on the host where you cannot run the container, built the image in-there as well and try again running the container.

Singularity container from conda environment

I want to build a container from my conda environment following this post. However, I get the following error: '/bin/sh: 1: cannot create ~/.bashrc: Directory nonexistent'. I am using a vagrant VM to build my image and would be grateful for any help.
Editing the .bashrc, aside from failing, will not be helpful as the shell loaded by singularity is explicitly --norc. You want to use the $SINGULARITY_ENVIRONMENT variable in %post to have the values available.
Something along these lines:
%post
# You may need to install some pre-reqs your host system has installed outside of conda, e.g.
# apt update && apt install -y build-essential make zlib
ENV_NAME=$(head -1 environment.yml | cut -d' ' -f2)
echo ". /opt/conda/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo "conda activate $ENV_NAME" >> $SINGULARITY_ENVIRONMENT
. /opt/conda/etc/profile.d/conda.sh
conda env create -f environment.yml -p /opt/conda/envs/$ENV_NAME
I listed a few libraries that you probably have installed in your current machine that might not be installed in the slim docker image. You can install them via apt or conda, depending on your preference. If it does happen though, it'll be specific to your environment.yml and host OS, so you'll have to iterate through until the build succeeds.

docker file to run automation test in JS files

I am trying to create a docker file to run selenium tests for a java script based project. Below is my docker file so far:
#base image
FROM selenium/standalone-chrome
#access to the project within docker container - Bundle app source
COPY ./seleniumTest/project /app
# Install Node.js
RUN sudo apt-get update
RUN sudo apt-get install --yes curl
RUN curl --silent --location https://deb.nodesource.com/setup_8.x | sudo bash -
#binding
EXPOSE 8080
#Define runtime
ENTRYPOINT /app/login.test.js
while running with $ docker run -p 4000:80 lamgadekamal/dockertest
returns: Unable to find image 'lamkam/dockertest:latest' locally docker: Error response from daemon: manifest for lamkam/dockertest:latest not found. Could not figure out why am I getting this?
I suspect that you need to build your image first, since the image cannot be found.
Run this command from the same directory where your Dockerfile is located. This will build the image.
docker build -t lamgadekamal/dockertest .
You can then verify that the image exists by running docker images
EDIT: After looking at this again, it appears that you are trying to run the wrong image. You are trying to run lamgadekamal/dockertest, but you built the image with the tag lamkam/dockertest? Seems like you have a typo. I would suggest running docker images to see exactly what is there, but in all likelihood, you need to run lamkam/dockertest.
docker run -p 4000:80 lamkam/dockertest

"docker run -dti" with a dumb terminal

updated: added the missing docker attach.
Hi am trying to run a docker container, with -dti. but I cannot access with a terminal set to dumb. is there a way to change this (it is currently set to xterm, even though my ssh client is dumb)
example:
create the container
docker run -dti --name test -v /my-folder alpine /bin/ash
docker attach test
apk --update add nodejs
cd /my-folder
npm install -g gulp
the last command always contains ascii escape chars to move the cursor.
I have tried "export TERM=dumb" inside the running container, but it does not work.
is there a way to "run" this using the dumb terminal?
I am running this from a script on another computer, via (dumb) ssh.
using the -t which sets this https://docs.docker.com/engine/reference/run/#env-environment-variables, however removing effects the command prompt (the prompt is not shown)
possible solution 1 remove the -t and keep the -i. To see if the command has completed echo out a known token (ENDENDEND). ie
docker run -di --name test -v /my-folder alpine /bin/ash
docker attach test
apk --update add nodejs;echo ENDENDEND
cd /my-folder;echo ENDENDEND
npm install -g gulp;echo ENDENDEND
not pretty, but it works (there is no ascii in the results)
Possible solution 2 use the journal, docker can log out to the linux journal, this can be gathered as commands are executed in the container. (I have yet to fully test this one out. however the log seems to be a nicer output of what happened)
update:
Yep -t is the problem.
However if you want to see the entire process when running a command, maybe this way is better:
docker run -di --name test -v/my-folder alpine /bin/ash
docker exec -it test /bin/ash
finally you need to kill the container after all jobs finished.
docker run -d means "Run container in background and print container ID"
not start the container as a daemon
I was hitting this issue on OSx running docker, i had to do 2 things to stop the terminal/ascii/ansi escape sequences.
remove the "t" option on the docker run command (from docker run -it ... to docker run -i...)
ensure to force bash or sh shells used on osx when running the command from a script file, not the default zsh
Also
the escape sequences were not always visible on the terminal
even so, they still usually caused content corruption, even with SED brought to bear
they always were shown in my editor

Automatically create docker container and launch python script

I am working on creating an automated unit testing system which will utilise docker to test individual student assignments, written in Python, against a single unit test file.
I have created a website where students can upload their assignments but I'm a little but unsure as to how to get the automation with Docker working.
The workflow looks something like this:
A student uploads an assignment for marking
This is copied to a linux host which contains docker
The file sits here while it waits to be tested
So, say I had twenty student uploading there .py files, named as their unique student numbers, could I:
Create a Docker container which runs Ubuntu and Python
Copy the student file and unit test into this container
Run the unit test
Output the results as a text file
Copy this text file back to my webserver to display the results
Could somebody point me in the right direction to get started with this automation? I'm really just after some help of the Docker side of things, not on copying the files from my webserver to the Docker host.
Thanks.
Yes, it is possible to use Docker for that.
The Dockerfile would look like this:
FROM ubuntu
MAINTAINER xxx <user#example.org>
# update ubuntu repository
RUN DEBIAN_FRONTEND=noninteractive apt-get -y update
# install ubuntu packages
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install python python-pip
# install python requirements
RUN pip install ...
# define a mount point
VOLUME /student.py
# define command for this image
CMD ["python","/student.py"]
Now, you have to build this image with docker build -t student_test ..
To start the script and grab the output you can use:
docker run --volume /path/to/s12345.py:/student.py student_test > student_results_12345.txt`.
The --volume parameter is needed, to mount a student script to the defined mount point. Also, you could start multiple containers at once.
All paths are relative to current working directory.
Checkout the following project
https://github.com/CenturyLinkLabs/buildpack-runner
Uses Heroku buildpacks to create a docker image. Crazy but a neat idea if you get it working.