How to install Scrapy on Amazon Linux AMI - scrapy

I want to use Scrapy which is a crawler framework, and install it to my Server.
This is my Server spec.
========================================================
NAME="Amazon Linux AMI"
VERSION="2015.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.09"
PRETTY_NAME="Amazon Linux AMI 2015.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.09
========================================================
I have read a lot of web pages about this and followed the steps.
Still I always get an error.
I've used the command sudo pip install scrapy and I get this log.
This is red logs.
-> Failed building wheel for lxml
-> Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-lT29Ha/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WTCmji-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-lT29Ha/lxml
I've spent 2 days on the Scrapy install. Please help me.

I just did this yesterday on AWS. I used a nano instance so needed to add some swap first.
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile swap swap sw 0 0" >> /etc/fstab
sudo swapon /swapfile
Then install the prerequisites and scrapy:
sudo su
yum update -y
yum install python-pip -y
yum install python-devel -y
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
CFLAGS="-O0" pip install lxml
pip install scrapy
scrapy -v

Related

unable to install singularity in Centos or Ubuntu

I need help on how to install singularity in Centos or Ubuntu successfully. I followed https://docs.sylabs.io/guides/3.0/user-guide/installation.html#install-on-linux but basically package "singularity" is unavailable. Any thoughts? Thanks!
Ubuntu:
sudo apt-get install -y singularity-container
E: Package 'singularity-container' has no installation candidate
Centos7:
sudo yum update -y && \
> sudo yum install -y epel-release && \
> sudo yum update -y && \
> sudo yum install -y singularity-runtime singularity
No package singularity-runtime available.
No package singularity available.

Installing version 1.15 of tensorflow-serving-api to Centos 8

I am trying to install Tensorflow-serving to my Centos 8 machine. Installing with Docker image is not an option for Centos. So I try to install with pip. These are the commands for installing tensorflow-model-server:
pip3 install tensorflow-serving-api==1.15
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install tensorflow-model-server
The problem is I need version 1.15.0 and I couldn't find how to modify links to install the 1.15 version. Any help for modifying links, ideas for installing "tensorflow/serving" to Centos 8 will be appreciated by me :)
I found the links:
wget 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.15.0/t/tensorflow-model-server/tensorflow-model-server_1.15.0_all.deb'
dpkg -i tensorflow-model-server_1.15.0_all.deb
pip3 install tensorflow-serving-api==1.15
With these commands, it works :)

Is Scrapy compatible with Python 3.8 on ubuntu?

While i am try to install scrapy on ubuntu mechine using
**pip install scrapy**
it's not going to workout.
can anyone suggest us?
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
In the documentation, it was mentioned that to install scrapy in a venv on ubuntu based os, we need to install these dependencies sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

How can I install Tensorflow GPU with latest version of CUDA and cuDNN

Installing TensorFlow for object detection is annoying sometimes, especially when wired errors happen after starting one's own object detection project by finetuning pre-trained model.
How to install latest Tensorflow GPU support and latest CUDA/CUDNN without any error?
To install tensorflow-gpu object detection api on Ubuntu 16.04
upgrade system
sudo apt-get update
sudo apt-get upgrade
install basic packages
sudo apt-get install vim curl python-dev gnupg-curl python-tk git
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo -H python get-pip.py
install tensorflow-gpu (tensorflow version=1.14)
sudo -H pip install tensorflow-gpu
install CUDA (Final CUDA version = 10.1)
// Please turn off your secure boot from BIOS
sudo apt-get install gnupg-curl
// Here we install version 10.0 to avoid other issues. Later we can upgrade it to version 10.1
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt-get update
install cuDNN (cuDNN version = 7.6.3.30-1, compatible to CUDA 10.1)
sudo apt-get install --no-install-recommends cuda-10-0
// restart your computer here
nvidia-smi.
sudo apt-get install --no-install-recommends libcudnn7=7.6.3.30-1+cuda10.0 libcudnn7-dev=7.6.3.30-1+cuda10.0
sudo apt-get install -y --no-install-recommends libnvinfer5=5.1.5-1+cuda10.0 libnvinfer-dev=5.1.5-1+cuda10.0
sudo apt-get update
// this upgrade command will upgrade your CUDA to version 10.1
sudo apt-get upgrade
sudo apt-get autoremove
install Tensorflow object detection api
sudo -H pip install Cython
sudo -H pip install contextlib2
sudo -H pip install pillow
sudo -H pip install lxml
sudo -H pip install jupyter
sudo -H pip install matplotlib
mkdir tensorflow
cd tensorflow
git clone https://github.com/tensorflow/models
// install protocbuf version 3.0.0
wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
unzip protobuf.zip
sudo cp ./bin/protoc /bin/
sudo cp -r ./include/google /usr/local/include/
cd tensorflow/models/research
protoc object_detection/protos/*.proto --python_out=.
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make
cp -r pycocotools <path_to_tensorflow>/models/research/
Export Environment
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64
export PYTHONPATH=$PYTHONPATH:~/tensorflow/models/research:~/tensorflow/models/research/object_detection/slim
Other
If we install CUDA 10.1 it may have a compatibility issue that libxxxx.so.10.0 is not found when you start training your project.
To solve them:
a. sudo ln -s /usr/lib/x86_64-linux-gnu/libcublas.so.10.1 /usr/local/cuda-10.1/lib64/libcublas.so.10.0
b. sudo ln -s /usr/local/cuda-10.1/lib64/libcudart.so.10.1 /usr/local/cuda-10.1/lib64/libcudart.so.10.0
c. sudo ln -s /usr/local/cuda-10.1/lib64/libcufft.so.10 /usr/local/cuda-10.1/lib64/libcufft.so.10.0
d. sudo ln -s /usr/local/cuda-10.1/lib64/libcurand.so.10 /usr/local/cuda-10.1/lib64/libcurand.so.10.0
e. sudo ln -s /usr/local/cuda-10.1/lib64/libcusolver.so.10 /usr/local/cuda-10.1/lib64/libcusolver.so.10.0
f. sudo ln -s /usr/local/cuda-10.1/lib64/libcusparse.so.10 /usr/local/cuda-10.1/lib64/libcusparse.so.10.0
All Set

Error with apr-util while installing httpd 2.4.1 on centos

I am trying to install httpd-2.4.1 on my CentOS machine (it has 2.2.15 currently). I did the following...
sudo yum install gcc
sudo yum install perl
sudo yum install libbz2 libbz2-devel
sudo yum install bzip2-devel
sudo yum install curl-devel
sudo yum install libjpeg-devel
sudo yum install libpng-devel
sudo yum install libX11-devel gd-devel
sudo yum install libc-client-devel
sudo yum install libmcrypt-devel
sudo yum install bzip2
sudo yum install pcre-devel
sudo yum install openssl-devel
sudo yum install libxml2-devel
sudo yum install libmcrypt
#apache
wget http://mirror.nus.edu.sg/apache//apr/apr-util-1.4.1.tar.gz
wget http://mirror.nus.edu.sg/apache//apr/apr-1.4.6.tar.gz
extract and move to /srclib in httpd-2.4.1 directory
sudo ./configure --prefix=/usr/local/apache --enable-so --enable-mods-shared=most --with-included-apr --with-included-apr-util --enable-load-all-modules
sudo make
sudo make install
And here I get the error:
/usr/bin/ld: /usr/local/apache/lib/libexpat.a(xmlparse.o): relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
/usr/local/apache/lib/libexpat.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
libtool: install: error: relink `libaprutil-1.la' with the above command before installing it
How do I relink libaprutil-1.la with the above command? (i am guessing it refers to -fPIC?)
You didn't include --with-included-apr-util while running ./configure
./configure --with-included-apr --with-included-apr-util