Singularity Container access external file for Build process - singularity-container

I need to install Cuda, CuDNN, and NCCL in a singularity container to build the MXNET Library for GPU Support. Currently, I have this minimal Definition File:
Bootstrap: library
From: ubuntu:20.04
%environment
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
... some other stuff
%post
... some other installation stuff
apt-get install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.2.0/local_installers/cuda-repo-ubuntu2004-11-2-local_11.2.0-460.27.04-1_amd64.deb
dpkg -i cuda-repo-ubuntu2004-11-2-local_11.2.0-460.27.04-1_amd64.deb
apt-key add /var/cuda-repo-ubuntu2004-11-2-local/7fa2af80.pub
apt-get update
apt-get -y install cuda
export PATH=/usr/local/cuda-11.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64\
${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
/usr/bin/nvidia-persistenced --verbose
... clone and build mxnet
Installing Cuda is not a problem. However, in order to install CudNN, one first needs to log into an NVIDIA Account. Similarly, to install NCCL, one needs to complete a survey.
I obviously cannot do that per command line. But I can download these files beforehand. How can I access these files in %post?

You can use the %files section of the definition file. All files added there are available in %post and, if not moved, are included in the final image.

#tsnowlan gave a correct answer.
For the others wondering, how to get Cuda, CuDNN, and NCCL easily into an singularity container:
There are Docker images from NVidia, that already come with these tools:
https://hub.docker.com/r/nvidia/cuda/tags
One can easily use these images as a base image for the docker container:
https://docs.sylabs.io/guides/3.0/user-guide/build_a_container.html

Related

Installing / Building the newest version of Inkscape (v 1.2) on Linux Ubuntu 20.04

I am trying to install Inkscape 1.2beta on Linux Ubuntu 20.04. The website currently only offers an AppImage and a source tarball. Since I would like to access the newest features of Inkscape via the command line, I need to build and install the source tarball.
INSTALL.md states that I need all submodules and dependencies before install.
How do I find these dependencies to successfully build and install Inkscape?
This list should satisfy all required dependencies on Linux Ubuntu:
apt install
cmake
imagemagick
libdouble-conversion-dev
libgdl-3-dev
libagg-dev
libpotrace-dev
libboost-all-dev
libsoup2.4-dev
libgc-dev
libwpg-dev
poppler-utils
libpoppler-dev
libpoppler-glib-dev
libpoppler-private-dev
libvisio-dev libvisio-tools
libcdr-dev
libgtkmm-3.0-dev
libgspell-1-dev
libxslt-dev libxslt1-dev
libreadline6-dev
lib2geom-dev
lib2geom-dev is needed to solve error "<ieeefp.h> not found".
For building Inkscape:
Download source tarball for Inkscape v1.2 from inkscape.org and extract
cd <extracted inkscape directory>
mkdir build && cd build
cmake ..
make
make install
If you still get an error during cmake .., please comment below with the names of the missing modules in the error message.
The details on how to build Inkscape (and the dependencies) could be found in the repository itself, or Inkscape website (For completness, the steps are copied from the website here):
To obtain the latest source code, use the following command (downloads into a subdirectory of your current working directory called "inkscape" by default):
git clone --recurse-submodules https://gitlab.com/inkscape/inkscape.git
To update this code later, change into the download folder and use:
git pull --recurse-submodules && git submodule update
By default, git will download every branch and every commit. If you are on a slow machine, have limited disk space, or limited internet bandwidth, you can use shallow clone and single branch clone options to limit the amount of data it will download:
git clone --depth=1 --single-branch --recurse-submodules --shallow-submodule https://gitlab.com/inkscape/inkscape.git
Building Inkscape on Linux
Open a terminal at the root of the folder into which you downloaded the source code in the previous step.
Install build dependencies
Download and run the script to install everything required for compiling Inkscape (check script to see if your distribution is supported):
wget -v https://gitlab.com/inkscape/inkscape-ci-docker/-/raw/master/install_dependencies.sh -O install_dependencies.sh
bash install_dependencies.sh --recommended
Compile
To compile with CMake, do the following:
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=${PWD}/install_dir -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
make -j8
make install
Notes:
Using ccache is optional but speeds up compilation.
The optional -j8 argument to make tells it to run 8 jobs in parallel. Feel free to adjust this to the number of hardware threads (physical cores) available on your computer.
The recommended -DCMAKE_INSTALL_PREFIX argument allows to specify a custom isolated installation location (in the example above install_dir/ inside the build folder). It avoids installation into system locations (where it could conflict with other versions of Inkscape) and allows running multiple versions of Inkscape in parallel. It will still use all the files (including the preferences.xml) that reside in the ~/.config/inkscape directory.
Run
Run it from the build directory:
install_dir/bin/inkscape

keras failed to import pydot

I'm trying to run the Pix2Pix tutorial for Tensorflow. I'm using the official docker container for this. This is how I start my container:
docker run --gpus all -it -p 8888:8888 --rm -v $PWD:/tf -w /tmp tensorflow/tensorflow:latest-gpu-py3-jupyter
I'm not able to get pass by this cell
generator = Generator()
tf.keras.utils.plot_model(generator, show_shapes=True, dpi=64)
# output -> Failed to import pydot. You must install pydot and graphviz for `pydotprint` to work.
I have tried also installing the pydot and graphviz using pip and also apt-get. Even if this libraries are installed I get the same error.
I had same problem and follow this link
In short:
run these command on command prompt.
pip install pydot
pip install graphviz
From website, download and install graphviz software
Note: in install time, check "add to system path" option to add bin
folder to path variable otherwise you should do it manually. restart
your windows

Is there a way to use Python 3.5 instead of 3.6?

I need to install a library that is only compatible with Python 3.5. Is there a way to change the Python version in Colaboratory from 3.6 to 3.5?
The only way to vary the Python 3 version is to connect to a local runtime.
You cannot directly change the environment for the notebook.
After hours of exploration, I found a solution:
Initialize a Ngork server in the Colaboratory notebook.
connect to the Ngork server from a local terminal using SSH (or use any editor which supports SSH connections)
Install the required Python version using the terminal.
Install virtualenv.
Create a virtual environment by specifying the Python version installed.
Activate the environment.
Work in that environment from the terminal directly.
Check out Free!! GPUs on your local machine which provides to get detailed description on how to follow the steps.
There is a way to use any version of python you want 3.5 or 3.8 in this example, without having to run a kernel locally or going through an ngrok proxy.
Download the colab notebook. Open a text editor to change the kernel specification to:
"kernelspec": {
"name": "py38",
"display_name": "Python 3.8"
}
This is the same trick as the one used with Javascript, Java, and Golang.
Then upload the edited notebook to Google Drive. Open the notebook in Google Colab. It cannot find the py38 kernel, so it use normal python3 kernel.
You need to install a python 3.8, the google-colab package and the ipykernel under the name you defined above: "py38":
!wget -O mini.sh https://repo.anaconda.com/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh
!chmod +x mini.sh
!bash ./mini.sh -b -f -p /usr/local
!conda install -q -y jupyter
!conda install -q -y google-colab -c conda-forge
!python -m ipykernel install --name "py38" --user
Reload the page, and voilà, you can test the version is correct:
import sys
print("User Current Version:-", sys.version)
A working example can be found there.

Why does it take ages to install Pandas on Alpine Linux

I've noticed that installing Pandas and Numpy (it's dependency) in a Docker container using the base OS Alpine vs. CentOS or Debian takes much longer. I created a little test below to demonstrate the time difference. Aside from the few seconds Alpine takes to update and download the build dependencies to install Pandas and Numpy, why does the setup.py take around 70x more time than on Debian install?
Is there any way to speed up the install using Alpine as the base image or is there another base image of comparable size to Alpine that is better to use for packages like Pandas and Numpy?
Dockerfile.debian
FROM python:3.6.4-slim-jessie
RUN pip install pandas
Build Debian image with Pandas & Numpy:
[PandasDockerTest] time docker build -t debian-pandas -f Dockerfile.debian . --no-cache
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM python:3.6.4-slim-jessie
---> 43431c5410f3
Step 2/2 : RUN pip install pandas
---> Running in 2e4c030f8051
Collecting pandas
Downloading pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl (26.2MB)
Collecting numpy>=1.9.0 (from pandas)
Downloading numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
Collecting pytz>=2011k (from pandas)
Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
Collecting python-dateutil>=2 (from pandas)
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, pytz, six, python-dateutil, pandas
Successfully installed numpy-1.14.1 pandas-0.22.0 python-dateutil-2.6.1 pytz-2018.3 six-1.11.0
Removing intermediate container 2e4c030f8051
---> a71e1c314897
Successfully built a71e1c314897
Successfully tagged debian-pandas:latest
docker build -t debian-pandas -f Dockerfile.debian . --no-cache 0.07s user 0.06s system 0% cpu 13.605 total
Dockerfile.alpine
FROM python:3.6.4-alpine3.7
RUN apk --update add --no-cache g++
RUN pip install pandas
Build Alpine image with Pandas & Numpy:
[PandasDockerTest] time docker build -t alpine-pandas -f Dockerfile.alpine . --no-cache
Sending build context to Docker daemon 16.9kB
Step 1/3 : FROM python:3.6.4-alpine3.7
---> 4b00a94b6f26
Step 2/3 : RUN apk --update add --no-cache g++
---> Running in 4b0c32551e3f
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/17) Upgrading musl (1.1.18-r2 -> 1.1.18-r3)
(2/17) Installing libgcc (6.4.0-r5)
(3/17) Installing libstdc++ (6.4.0-r5)
(4/17) Installing binutils-libs (2.28-r3)
(5/17) Installing binutils (2.28-r3)
(6/17) Installing gmp (6.1.2-r1)
(7/17) Installing isl (0.18-r0)
(8/17) Installing libgomp (6.4.0-r5)
(9/17) Installing libatomic (6.4.0-r5)
(10/17) Installing pkgconf (1.3.10-r0)
(11/17) Installing mpfr3 (3.1.5-r1)
(12/17) Installing mpc1 (1.0.3-r1)
(13/17) Installing gcc (6.4.0-r5)
(14/17) Installing musl-dev (1.1.18-r3)
(15/17) Installing libc-dev (0.7.1-r0)
(16/17) Installing g++ (6.4.0-r5)
(17/17) Upgrading musl-utils (1.1.18-r2 -> 1.1.18-r3)
Executing busybox-1.27.2-r7.trigger
OK: 184 MiB in 50 packages
Removing intermediate container 4b0c32551e3f
---> be26c3bf4e42
Step 3/3 : RUN pip install pandas
---> Running in 36f6024e5e2d
Collecting pandas
Downloading pandas-0.22.0.tar.gz (11.3MB)
Collecting python-dateutil>=2 (from pandas)
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
Collecting pytz>=2011k (from pandas)
Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
Collecting numpy>=1.9.0 (from pandas)
Downloading numpy-1.14.1.zip (4.9MB)
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Downloading six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: pandas, numpy
Running setup.py bdist_wheel for pandas: started
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: still running...
Running setup.py bdist_wheel for pandas: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/e8/ed/46/0596b51014f3cc49259e52dff9824e1c6fe352048a2656fc92
Running setup.py bdist_wheel for numpy: started
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: still running...
Running setup.py bdist_wheel for numpy: finished with status 'done'
Stored in directory: /root/.cache/pip/wheels/9d/cd/e1/4d418b16ea662e512349ef193ed9d9ff473af715110798c984
Successfully built pandas numpy
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Successfully installed numpy-1.14.1 pandas-0.22.0 python-dateutil-2.6.1 pytz-2018.3 six-1.11.0
Removing intermediate container 36f6024e5e2d
---> a93c59e6a106
Successfully built a93c59e6a106
Successfully tagged alpine-pandas:latest
docker build -t alpine-pandas -f Dockerfile.alpine . --no-cache 0.54s user 0.33s system 0% cpu 16:08.47 total
Debian based images use only python pip to install packages with .whl format:
Downloading pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl (26.2MB)
Downloading numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
WHL format was developed as a quicker and more reliable method of installing Python software than re-building from source code every time. WHL files only have to be moved to the correct location on the target system to be installed, whereas a source distribution requires a build step before installation.
Wheel packages pandas and numpy are not supported in images based on Alpine platform. That's why when we install them using python pip during the building process, we always compile them from the source files in alpine:
Downloading pandas-0.22.0.tar.gz (11.3MB)
Downloading numpy-1.14.1.zip (4.9MB)
and we can see the following inside container during the image building:
/ # ps aux
PID USER TIME COMMAND
1 root 0:00 /bin/sh -c pip install pandas
7 root 0:04 {pip} /usr/local/bin/python /usr/local/bin/pip install pandas
21 root 0:07 /usr/local/bin/python -c import setuptools, tokenize;__file__='/tmp/pip-build-en29h0ak/pandas/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n
496 root 0:00 sh
660 root 0:00 /bin/sh -c gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DTHREAD_STACK_SIZE=0x100000 -fPIC -Ibuild/src.linux-x86_64-3.6/numpy/core/src/pri
661 root 0:00 gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DTHREAD_STACK_SIZE=0x100000 -fPIC -Ibuild/src.linux-x86_64-3.6/numpy/core/src/private -Inump
662 root 0:00 /usr/libexec/gcc/x86_64-alpine-linux-musl/6.4.0/cc1 -quiet -I build/src.linux-x86_64-3.6/numpy/core/src/private -I numpy/core/include -I build/src.linux-x86_64-3.6/numpy/core/includ
663 root 0:00 ps aux
If we modify Dockerfile a little:
FROM python:3.6.4-alpine3.7
RUN apk add --no-cache g++ wget
RUN wget https://pypi.python.org/packages/da/c6/0936bc5814b429fddb5d6252566fe73a3e40372e6ceaf87de3dec1326f28/pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
RUN pip install pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
we get the following error:
Step 4/4 : RUN pip install pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
---> Running in 0faea63e2bda
pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl is not a supported wheel on this platform.
The command '/bin/sh -c pip install pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl' returned a non-zero code: 1
Unfortunately, the only way to install pandas on an Alpine image is to wait until build finishes.
Of course if you want to use the Alpine image with pandas in CI for example, the best way to do so is to compile it once, push it to any registry and use it as a base image for your needs.
EDIT:
If you want to use the Alpine image with pandas you can pull my nickgryg/alpine-pandas docker image. It is a python image with pre-compiled pandas on the Alpine platform. It should save your time.
ANSWER: AS OF 3/9/2020, FOR PYTHON 3, IT STILL DOESN'T!
Here is a complete working Dockerfile:
FROM python:3.7-alpine
RUN echo "#testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk add --update --no-cache py3-numpy py3-pandas#testing
The build is very sensitive to the exact python and alpine version numbers - getting these wrong seems to provoke Max Levy's error so:libpython3.7m.so.1.0 (missing) - but the above does now work for me.
My updated Dockerfile is available at https://gist.github.com/jtlz2/b0f4bc07ce2ff04bc193337f2327c13b
[Earlier Update:]
ANSWER: IT DOESN'T!
In any Alpine Dockerfile you can simply do*
RUN apk add py2-numpy#community py2-scipy#community py-pandas#edge
This is because numpy, scipy and now pandas are all available prebuilt on alpine:
https://pkgs.alpinelinux.org/packages?name=*numpy
https://pkgs.alpinelinux.org/packages?name=*scipy&branch=edge
https://pkgs.alpinelinux.org/packages?name=*pandas&branch=edge
One way to avoid rebuilding every time, or using a Docker layer, is to use a prebuilt, native Alpine Linux/.apk package, e.g.
https://github.com/sgerrand/alpine-pkg-py-pandas
https://github.com/nbgallery/apks
You can build these .apks once and use them wherever in your Dockerfile you like :)
This also saves you having to bake everything else into the Docker image before the fact - i.e. the flexibility to pre-build any Docker image you like.
PS I have put a Dockerfile stub at https://gist.github.com/jtlz2/b0f4bc07ce2ff04bc193337f2327c13b that shows roughly how to build the image. These include the important steps (*):
RUN echo "#community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk update
RUN apk add --update --no-cache libgfortran
Real honest advice here, switch to Debian based image and then all your problems will be gone.
Alpine for python applications doesn't work well.
Here is an example of my dockerfile:
FROM python:3.7.6-buster
RUN pip install pandas==1.0.0
RUN pip install sklearn
RUN pip install Django==3.0.2
RUN pip install cx_Oracle==7.3.0
RUN pip install excel
RUN pip install djangorestframework==3.11.0
The python:3.7.6-buster is more appropriate in this case, in addition, you don't need any extra dependency in the OS.
Follow a usefull and recent article: https://pythonspeed.com/articles/alpine-docker-python/:
Don’t use Alpine Linux for Python images
Unless you want massively slower build times, larger images, more work, and the potential for obscure bugs, you’ll want to avoid Alpine Linux as a base image. For some recommendations on what you should use, see my article on choosing a good base image.
Just going to bring some of these answers together in one answer and add a detail I think was missed. The reason certain python libraries, particularly optimized math and data libraries, take so long to build on alpine is because the pip wheels for these libraries include binaries precompiled from c/c++ and linked against gnu-libc (glibc), a common set of c standard libraries. Debian, Fedora, CentOS all (typically) use glibc, but alpine, in order to stay lightweight, uses musl-libc instead. c/c++ binaries build on a glibc system will not work on a system without glibc and the same goes for musl.
Pip looks first for a wheel with the correct binaries, if it can't find one, it tries to compile the binaries from the c/c++ source and links them against musl. In many cases, this won't even work unless you have the python headers from python3-dev or build tools like make.
Now the silver lining, as others have mentioned, there are apk packages with the proper binaries provided by the community, using these will save you the (sometimes lengthy) process of building the binaries.
You can, in fact, install from a pure python .whl on alpine, but, at the time of this writing, manylinux did not support binary distributions for alpine due to the musl/gnu issue.
Update Oct 2022
Newer versions of python/pip support musl via the package musllinux which, I assume, is a musl impl for manylinux. Still no official 'musl' support for CUDA though.
ATTENTION
Look at the #jtlz2 answer with the latest update
OUTDATED
So, py3-pandas & py3-numpy packages moved to the testing alpine repository, so, you can download it by adding these lines in to the your Dockerfile:
RUN echo "http://dl-8.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories \
&& apk update \
&& apk add py3-numpy py3-pandas
Hope it helps someone!
Alpine packages links:
- py3-pandas
- py3-numpy
Alpine repositories docks info.
In this case the alpine not be the best solution change alpine for slim:
FROM python:3.8.3-alpine
Change to that:
FROM python:3.8.3-slim
In my case it was resolved with this small change.
This worked for me:
FROM python:3.8-alpine
RUN echo "#testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk add --update --no-cache py3-numpy py3-pandas#testing
ENV PYTHONPATH=/usr/lib/python3.8/site-packages
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
EXPOSE 5003
ENTRYPOINT [ "python" ]
CMD [ "app.py" ]
Most of the code here is from the answer of jtlz2 from this same thread and Faylixe from another thread.
Turns out the lighter version of pandas is found in the Alpine repository py3-numpy but it doesn't get installed in the same file path from where Python reads the imports by default. Therefore you need to add the ENV. Also be mindful about the alpine version.
I have solved the installation with some additional changes:
Requirements
Migrate from python3.8-alpine to python3.10-alpine:
docker pull python:3.10-alpine
Important!
I had to migrate because when I was installing py3-pandas, it installed the package as python3.10, not in the required version
that I was using python3.8).
To figure out where the libraries of a package were installed, you can check that with the following command:
apk info -L py3-pandas
Not install backports.zoneinfo package since python3.9 (I had to add a condition in the requirements.txt to install the package with versions lower than 3.9).
backports.zoneinfo==0.2.1;python_version<"3.9"
Installation
After the previous changes, I proceed to install panda performing the following:
Add 3 additional repositories to /etc/apk/repositories (the repositories can vary based on the version of your distribution), reference here:
for x in $(echo "main community testing"); \
do echo "https://dl-cdn.alpinelinux.org/alpine/edge/${x}" >> /etc/apk/repositories; \
done
Validate the content of the file /etc/apk/repositories:
$ cat /etc/apk/repositories
https://dl-cdn.alpinelinux.org/alpine/v3.16/main
https://dl-cdn.alpinelinux.org/alpine/v3.16/community
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community
https://dl-cdn.alpinelinux.org/alpine/edge/testing
Perform to install pandas (pynum is installed automatically as a dependency of pandas):
sudo apk update && sudo apk add py3-pandas
Set the environment variable PYTHONPATH:
export PYTHONPATH=/usr/lib/python3.10/site-packages/
Validate the packages can be imported (on my case I tested it with django):
python manage.py shell
import pandas as pd
import numpy as np
technologies = ['Spark','Pandas','Java','Python', 'PHP']
fee = [25000,20000,15000,15000,18000]
duration = ['5o Days','35 Days',np.nan,'30 Days', '30 Days']
discount = [2000,1000,800,500,800]
columns=['Courses','Fee','Duration','Discount']
df = pd.DataFrame(list(zip(technologies,fee,duration,discount)), columns=columns)
print(df)
pandas is considered a community supported package, so the answers pointing to edge/testing are not going to work as Alpine does not officially support pandas as a core package (it still works, it's just not supported by the core Alpine developers).
Try this Dockerfile:
FROM python:3.8-alpine
RUN echo "#community http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories \
&& apk add py3-pandas#community
ENV PYTHONPATH="/usr/lib/python3.8/site-packages"
This works for the vanilla Alpine image too, using FROM alpine:3.12.
Update: thanks to #cegprakash for raising the question about how to work with this setup when you also have a requirements.txt file that must be satisfied inside the container.
I added one line to the Dockerfile snippet to export the PYTHONPATH variable into the container runtime. If you do this, it won't matter whether pandas or numpy are included in the requirements file or not (provided they are pegged to the same version that was installed via apk).
The reason this is needed is that apk installs the py3-pands#community package under /usr/lib, but that location is not on the default PYTHONPATH that pip checks before installing new packages. If we don't include this step to add it, pip and python will not find the package and pip will try to download and install it under /usr/local which is what we're trying to avoid.
And given that we really want to make sure that pip doesn't try to install pandas, I would suggest to not include pandas or numpy in the requirements.txt file if you've already installed them with apk using the above method. It's just a little extra insurance that things will go as intended.
The following Dockerfile worked for me to install pandas, among other dependencies as listed below.
python:3.10-alpine Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.10-alpine as base
RUN apk add --update --no-cache --virtual .tmp-build-deps \
gcc g++ libc-dev linux-headers postgresql-dev build-base \
&& apk add libffi-dev
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir --upgrade -r requirements.txt
pyproject.toml dependencies
python = "^3.10"
Django = "^3.2.9"
djangorestframework = "^3.12.4"
PyYAML = ">=5.3.0,<6.0.0"
Markdown = "^3.3.6"
uritemplate = "^4.1.1"
install = "^1.3.5"
drf-spectacular = "^0.21.0"
django-extensions = "^3.1.5"
django-filter = "^21.1"
django-cors-headers = "^3.10.1"
httpx = "^0.22.0"
channels = "^3.0.4"
daphne = "^3.0.2"
whitenoise = "^6.2.0"
djoser = "^2.1.0"
channels-redis = "^3.4.0"
pika = "^1.2.1"
backoff = "^2.1.2"
psycopg2-binary = "^2.9.3"
pandas = "^1.5.0"
alpine takes lot of time to install pandas and the image size is also huge. I tried the python:3.8-slim-buster version of python base image. Image build was very fast and size of image was less than half in comparison to alpine python docker image
https://github.com/dguyhasnoname/k8s-cluster-checker/blob/master/Dockerfile

cuda install error on Ubuntu 17.04

abigail#abilina:~/Downloads$ sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
Selecting previously unselected package cuda-repo-ubuntu1604.
(Reading database ... 205999 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu1604_8.0.61-1_amd64.deb ...
Unpacking cuda-repo-ubuntu1604 (8.0.61-1) ...
Setting up cuda-repo-ubuntu1604 (8.0.61-1) ...
Warning: The postinst maintainerscript of the package cuda-repo-ubuntu1604
Warning: seems to use apt-key (provided by apt) without depending on gnupg or gnupg2.
Warning: This will BREAK in the future and should be fixed by the package maintainer(s).
Note: Check first if apt-key functionality is needed at all - it probably isn't!
Warning: apt-key should not be used in scripts (called from postinst maintainerscript of the package cuda-repo-ubuntu1604)
OK
abigail#abilina:~/Downloads$ sudo apt-get install cuda
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
cuda : Depends: cuda-8-0 (>= 8.0.61) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
My ubuntu version is 17.04. Does this mean my Linux currently can't install CUDA? I want to install TensorFlow with GPU support.
Per suggestion:
abigail#abilina:~/Downloads$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
https://launchpad.net/ubuntu/zesty/amd64/nvidia-cuda-toolkit
above is the correct one.
apt-get install nvidia-cuda-toolkit
but remember the cuda installed through apt were installed in different location. manually create ln -s at /usr/local/cuda for include,lib64,and bin
I have successfully installed CUDA 8.0 + the latest patch from NVIDIA on Ubuntu 17.04:
Download the .run file from https://developer.nvidia.com/cuda-downloads, choosing Ubuntu 16.04 (Base Installer)
You will not be able to install it by just running, because it is looking for a file called InstallUtils.pm which is not present in Ubuntu 17.04, but curiously, is present in the .run file - so: unpack the .run file using ./cuda*.run --tar mxvf
copy InstallUtils.pm (should be in the /bin path) to /usr/lib/x86_64-linux-gnu/perl-base
Run the installer (You may want to say no to the driver install step to keep the one you install through apt - I'm using 381.22, because 375.26, which is provided by the .run file does not support my 1080ti)
gcc 6 is incompatible with CUDA, but this is easily remedied for compiling the sample files: just add export EXTRA_NVCCFLAGS="-Xcompiler -std=c++98" to your bashrc file, and comment out
from one of the headers (I think it was host_config.h, but you'll see it once you try to compile) - comment out these lines:
#if __GNUC__ > 5 || (__GNUC__ == 5 && __GNUC_MINOR__ > 3)
#error -- unsupported GNU version! gcc versions later than 5.3 are not supported!
This is all from memory, so hopefully it's accurate enough.
I managed to find this solution thanks to these useful posts:
https://devtalk.nvidia.com/default/topic/983777/can-t-locate-installutils-pm-in-inc/
https://devtalk.nvidia.com/default/topic/949770/cuda-8-0rc-supporting-gcc6-/
For ubuntu 17.04, I had to use cuda 9.0 (deb version)
https://developer.nvidia.com/cuda-release-candidate-download
I couldn't get it to work otherwise. Cuda 8.0 needs gcc 5.3.1 but cuda 9.0 is compatible with gcc 6.3.0 which is installed on ubuntu 17.04 automatically.
More precisely, this is what I did:
On Ubuntu 17.04, install CUDA 9.0 — you can currently download the beta version 
https://developer.nvidia.com/cuda-release-candidate-download
I downloaded the .deb file and haven’t had any problems — follow the steps they recommend when you download cuda 9.0 
sudo dpkg -i cuda-repo-ubuntu1704-9-0-local-rc_9.0.103-1_amd64.deb
sudo apt-key add /var/cuda-repo-9.0-local-rc/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda
Then follow the post installation steps from the nvidia instructions (i.e., setting PATH and LD_LIBRARY_PATH) 
 http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions)
export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64 ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Cuda 9 is compatible with gcc 6.3.0 (which comes with 17.04). I used arch=sm_52 and sometimes for my make files have to go ‘make clean’.
Installing Cuda 9.0 was the simplest solution in my case.
Alternatively, if you'd prefer cuda 8, you can download the deb file and then use the command
dpkg-deb -x cuda_8.*.deb /usr/local/cuda-8.0
to extract the contents from the deb file and have them placed in the desired directory.
Source: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#advanced-setup)