How can I install Tensorflow GPU with latest version of CUDA and cuDNN - tensorflow

Installing TensorFlow for object detection is annoying sometimes, especially when wired errors happen after starting one's own object detection project by finetuning pre-trained model.
How to install latest Tensorflow GPU support and latest CUDA/CUDNN without any error?

To install tensorflow-gpu object detection api on Ubuntu 16.04
upgrade system
sudo apt-get update
sudo apt-get upgrade
install basic packages
sudo apt-get install vim curl python-dev gnupg-curl python-tk git
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo -H python get-pip.py
install tensorflow-gpu (tensorflow version=1.14)
sudo -H pip install tensorflow-gpu
install CUDA (Final CUDA version = 10.1)
// Please turn off your secure boot from BIOS
sudo apt-get install gnupg-curl
// Here we install version 10.0 to avoid other issues. Later we can upgrade it to version 10.1
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt-get update
install cuDNN (cuDNN version = 7.6.3.30-1, compatible to CUDA 10.1)
sudo apt-get install --no-install-recommends cuda-10-0
// restart your computer here
nvidia-smi.
sudo apt-get install --no-install-recommends libcudnn7=7.6.3.30-1+cuda10.0 libcudnn7-dev=7.6.3.30-1+cuda10.0
sudo apt-get install -y --no-install-recommends libnvinfer5=5.1.5-1+cuda10.0 libnvinfer-dev=5.1.5-1+cuda10.0
sudo apt-get update
// this upgrade command will upgrade your CUDA to version 10.1
sudo apt-get upgrade
sudo apt-get autoremove
install Tensorflow object detection api
sudo -H pip install Cython
sudo -H pip install contextlib2
sudo -H pip install pillow
sudo -H pip install lxml
sudo -H pip install jupyter
sudo -H pip install matplotlib
mkdir tensorflow
cd tensorflow
git clone https://github.com/tensorflow/models
// install protocbuf version 3.0.0
wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
unzip protobuf.zip
sudo cp ./bin/protoc /bin/
sudo cp -r ./include/google /usr/local/include/
cd tensorflow/models/research
protoc object_detection/protos/*.proto --python_out=.
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make
cp -r pycocotools <path_to_tensorflow>/models/research/
Export Environment
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64
export PYTHONPATH=$PYTHONPATH:~/tensorflow/models/research:~/tensorflow/models/research/object_detection/slim
Other
If we install CUDA 10.1 it may have a compatibility issue that libxxxx.so.10.0 is not found when you start training your project.
To solve them:
a. sudo ln -s /usr/lib/x86_64-linux-gnu/libcublas.so.10.1 /usr/local/cuda-10.1/lib64/libcublas.so.10.0
b. sudo ln -s /usr/local/cuda-10.1/lib64/libcudart.so.10.1 /usr/local/cuda-10.1/lib64/libcudart.so.10.0
c. sudo ln -s /usr/local/cuda-10.1/lib64/libcufft.so.10 /usr/local/cuda-10.1/lib64/libcufft.so.10.0
d. sudo ln -s /usr/local/cuda-10.1/lib64/libcurand.so.10 /usr/local/cuda-10.1/lib64/libcurand.so.10.0
e. sudo ln -s /usr/local/cuda-10.1/lib64/libcusolver.so.10 /usr/local/cuda-10.1/lib64/libcusolver.so.10.0
f. sudo ln -s /usr/local/cuda-10.1/lib64/libcusparse.so.10 /usr/local/cuda-10.1/lib64/libcusparse.so.10.0
All Set

Related

unable to install singularity in Centos or Ubuntu

I need help on how to install singularity in Centos or Ubuntu successfully. I followed https://docs.sylabs.io/guides/3.0/user-guide/installation.html#install-on-linux but basically package "singularity" is unavailable. Any thoughts? Thanks!
Ubuntu:
sudo apt-get install -y singularity-container
E: Package 'singularity-container' has no installation candidate
Centos7:
sudo yum update -y && \
> sudo yum install -y epel-release && \
> sudo yum update -y && \
> sudo yum install -y singularity-runtime singularity
No package singularity-runtime available.
No package singularity available.

Installing version 1.15 of tensorflow-serving-api to Centos 8

I am trying to install Tensorflow-serving to my Centos 8 machine. Installing with Docker image is not an option for Centos. So I try to install with pip. These are the commands for installing tensorflow-model-server:
pip3 install tensorflow-serving-api==1.15
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install tensorflow-model-server
The problem is I need version 1.15.0 and I couldn't find how to modify links to install the 1.15 version. Any help for modifying links, ideas for installing "tensorflow/serving" to Centos 8 will be appreciated by me :)
I found the links:
wget 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.15.0/t/tensorflow-model-server/tensorflow-model-server_1.15.0_all.deb'
dpkg -i tensorflow-model-server_1.15.0_all.deb
pip3 install tensorflow-serving-api==1.15
With these commands, it works :)

Raspberry Pi 4 OBS Virtual camera

I wrote this bash file, but not working. I think RPI just 32bit and the package what I found 64bit. Do you have any idea, how to find 32bit version of the package or how can I solve the problem?
I used this repos:
https://github.com/umlaeute/v4l2loopback
https://github.com/CatxFish/obs-v4l2sink
This code was the base
https://obsproject.com/forum/threads/obs-raspberry-pi-build-instructions.115739/
#!/bin/bash
#########################################################
# delete unnessery programs from pi
#########################################################
sudo apt-get -y remove --purge libreoffice*
sudo apt-get -y remove --purge scratch2 minecraft-pi wolfram-engine sonic-pi dillo squeak-vm squeak-plugins-scratch geany
sudo apt-get clean
sudo apt-get -y autoremove
sudo apt-get update
sudo apt-get dist-upgrade
#########################################################
#install obs with virtual camera
#########################################################
#make swap file
sudo dd if=/dev/zero of=/var/swap2 bs=1024 count=1000000
sudo chmod 600 /var/swap2
sudo mkswap /var/swap2
sudo swapon /var/swap2
sudo swapon
#install obs
sudo apt install qtbase5-dev
mkdir -p ~/install
cd ~/install
git clone --recursive https://github.com/obsproject/obs-studio.git
sudo apt-get --allow-releaseinfo-change update
sudo DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
sudo apt-get -y install build-essential checkinstall cmake libmbedtls-dev libasound2-dev libavcodec-dev libavdevice-dev libavfilter-dev libavformat-dev libavutil-dev libcurl4-openssl-dev libfontconfig1-dev libfreetype6-dev libgl1-mesa-dev libjack-jackd2-dev libjansson-dev libluajit-5.1-dev libpulse-dev libqt5x11extras5-dev libspeexdsp-dev libswresample-dev libswscale-dev libudev-dev libv4l-dev libvlc-dev libx11-dev libx11-xcb1 libx11-xcb-dev libxcb-xinput0 libxcb-xinput-dev libxcb-randr0 libxcb-randr0-dev libxcb-xfixes0 libxcb-xfixes0-dev libx264-dev libxcb-shm0-dev libxcb-xinerama0-dev libxcomposite-dev libxinerama-dev pkg-config python3-dev qtbase5-dev libqt5svg5-dev swig
sudo wget http://ftp.uk.debian.org/debian/pool/non-free/f/fdk-aac/libfdk-aac1_0.1.4-2+b1_armhf.deb
sudo wget http://ftp.uk.debian.org/debian/pool/non-free/f/fdk-aac/libfdk-aac-dev_0.1.4-2+b1_armhf.deb
sudo dpkg -i libfdk-aac1_0.1.4-2+b1_armhf.deb
sudo dpkg -i libfdk-aac-dev_0.1.4-2+b1_armhf.deb
sudo git clone --recursive https://github.com/obsproject/obs-studio.git
cd obs-studio
sudo mkdir build && cd build
sudo cmake -DUNIX_STRUCTURE=1 -DCMAKE_INSTALL_PREFIX=/usr ..
sudo make -j4
sudo make install
#install virtual cam
cd ~/install
sudo apt-get install -y v4l-utils v4l2loopback-utils
sudo wget http://raspbian.raspberrypi.org/raspbian/pool/main/v/v4l2loopback/v4l2loopback-dkms_0.12.5-1_all.deb
sudo apt-get install -y ~/install/v4l2loopback-dkms_0.12.5-1_all.deb
#install virtual cam obs plugin
cd ~/install
sudo wget https://github.com/CatxFish/obs-v4l2sink/releases/download/0.1.0/obs-v4l2sink.deb
sudo apt-get install -y ~/install/obs-v4l2sink.deb
mkdir -p ~/.config/obs-studio/plugins/v4l2sink/bin/64bit/
#run OBS
MESA_GL_VERSION_OVERRIDE=3.3 obs #only need to run this command from now on

How to install Tensorflow federated directly from GitHub or local download?

I want to have access to features from TensorFlow federated (tff.python.research) which aren't present with the pip3 install method.
I'm working on a remote server that does not have bazel, thus I cannot build from source. Are there other ways to get and install the latest working version of TFF from its GitHub REPO?
(https://github.com/tensorflow/federated)
To install the latest Tensorflow 2.0 federated, you may follow the steps below.
Install TensorFlow Federated using pip
Install the Python development environment
On Ubuntu:
$ sudo apt update
$ sudo apt install python3-dev python3-pip # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
On macOS:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
$ brew update
$ brew install python # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
Create a virtual environment
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment, run deactivate.
Install the TensorFlow Federated pip package.
(venv) $ pip install --upgrade tensorflow_federated
(Optional) Test Tensorflow Federated.
(venv) $ python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"
Build the TensorFlow Federated pip package
Install the Python development environment.
On Ubuntu:
$ sudo apt update
$ sudo apt install python3-dev python3-pip # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
On macOS:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
$ brew update
$ brew install python # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
Install Bazel
Install Bazel, the build tool used to compile Tensorflow Federated.
Clone the Tensorflow Federated repository.
$ git clone https://github.com/tensorflow/federated.git
$ cd "federated"
Create a virtual environment.
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment, run deactivate.
Install Tensorflow Federated dependencies.
(venv) $ pip install --requirement "requirements.txt"
(Optional) Test Tensorflow Federated.
(venv) $ bazel test //tensorflow_federated/...
Create a new project.
$ mkdir "/tmp/project"
$ cd "/tmp/project"
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment run deactivate.
Install the pip package.
(venv) $ pip install --upgrade "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"
Test Tensorflow Federated.
(venv) $ python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"
Reference: https://www.tensorflow.org/federated/install

How to install Scrapy on Amazon Linux AMI

I want to use Scrapy which is a crawler framework, and install it to my Server.
This is my Server spec.
========================================================
NAME="Amazon Linux AMI"
VERSION="2015.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.09"
PRETTY_NAME="Amazon Linux AMI 2015.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.09
========================================================
I have read a lot of web pages about this and followed the steps.
Still I always get an error.
I've used the command sudo pip install scrapy and I get this log.
This is red logs.
-> Failed building wheel for lxml
-> Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-lT29Ha/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WTCmji-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-lT29Ha/lxml
I've spent 2 days on the Scrapy install. Please help me.
I just did this yesterday on AWS. I used a nano instance so needed to add some swap first.
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile swap swap sw 0 0" >> /etc/fstab
sudo swapon /swapfile
Then install the prerequisites and scrapy:
sudo su
yum update -y
yum install python-pip -y
yum install python-devel -y
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
CFLAGS="-O0" pip install lxml
pip install scrapy
scrapy -v