I want to use Scrapy which is a crawler framework, and install it to my Server.
This is my Server spec.
========================================================
NAME="Amazon Linux AMI"
VERSION="2015.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.09"
PRETTY_NAME="Amazon Linux AMI 2015.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.09
========================================================
I have read a lot of web pages about this and followed the steps.
Still I always get an error.
I've used the command sudo pip install scrapy and I get this log.
This is red logs.
-> Failed building wheel for lxml
-> Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-lT29Ha/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WTCmji-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-lT29Ha/lxml
I've spent 2 days on the Scrapy install. Please help me.
I just did this yesterday on AWS. I used a nano instance so needed to add some swap first.
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo echo "/swapfile swap swap sw 0 0" >> /etc/fstab
sudo swapon /swapfile
Then install the prerequisites and scrapy:
sudo su
yum update -y
yum install python-pip -y
yum install python-devel -y
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
CFLAGS="-O0" pip install lxml
pip install scrapy
scrapy -v
Related
I need help on how to install singularity in Centos or Ubuntu successfully. I followed https://docs.sylabs.io/guides/3.0/user-guide/installation.html#install-on-linux but basically package "singularity" is unavailable. Any thoughts? Thanks!
Ubuntu:
sudo apt-get install -y singularity-container
E: Package 'singularity-container' has no installation candidate
Centos7:
sudo yum update -y && \
> sudo yum install -y epel-release && \
> sudo yum update -y && \
> sudo yum install -y singularity-runtime singularity
No package singularity-runtime available.
No package singularity available.
I am trying to install Tensorflow-serving to my Centos 8 machine. Installing with Docker image is not an option for Centos. So I try to install with pip. These are the commands for installing tensorflow-model-server:
pip3 install tensorflow-serving-api==1.15
echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install tensorflow-model-server
The problem is I need version 1.15.0 and I couldn't find how to modify links to install the 1.15 version. Any help for modifying links, ideas for installing "tensorflow/serving" to Centos 8 will be appreciated by me :)
I found the links:
wget 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.15.0/t/tensorflow-model-server/tensorflow-model-server_1.15.0_all.deb'
dpkg -i tensorflow-model-server_1.15.0_all.deb
pip3 install tensorflow-serving-api==1.15
With these commands, it works :)
While i am try to install scrapy on ubuntu mechine using
**pip install scrapy**
it's not going to workout.
can anyone suggest us?
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
In the documentation, it was mentioned that to install scrapy in a venv on ubuntu based os, we need to install these dependencies sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
Installing TensorFlow for object detection is annoying sometimes, especially when wired errors happen after starting one's own object detection project by finetuning pre-trained model.
How to install latest Tensorflow GPU support and latest CUDA/CUDNN without any error?
To install tensorflow-gpu object detection api on Ubuntu 16.04
upgrade system
sudo apt-get update
sudo apt-get upgrade
install basic packages
sudo apt-get install vim curl python-dev gnupg-curl python-tk git
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo -H python get-pip.py
install tensorflow-gpu (tensorflow version=1.14)
sudo -H pip install tensorflow-gpu
install CUDA (Final CUDA version = 10.1)
// Please turn off your secure boot from BIOS
sudo apt-get install gnupg-curl
// Here we install version 10.0 to avoid other issues. Later we can upgrade it to version 10.1
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt-get update
install cuDNN (cuDNN version = 7.6.3.30-1, compatible to CUDA 10.1)
sudo apt-get install --no-install-recommends cuda-10-0
// restart your computer here
nvidia-smi.
sudo apt-get install --no-install-recommends libcudnn7=7.6.3.30-1+cuda10.0 libcudnn7-dev=7.6.3.30-1+cuda10.0
sudo apt-get install -y --no-install-recommends libnvinfer5=5.1.5-1+cuda10.0 libnvinfer-dev=5.1.5-1+cuda10.0
sudo apt-get update
// this upgrade command will upgrade your CUDA to version 10.1
sudo apt-get upgrade
sudo apt-get autoremove
install Tensorflow object detection api
sudo -H pip install Cython
sudo -H pip install contextlib2
sudo -H pip install pillow
sudo -H pip install lxml
sudo -H pip install jupyter
sudo -H pip install matplotlib
mkdir tensorflow
cd tensorflow
git clone https://github.com/tensorflow/models
// install protocbuf version 3.0.0
wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
unzip protobuf.zip
sudo cp ./bin/protoc /bin/
sudo cp -r ./include/google /usr/local/include/
cd tensorflow/models/research
protoc object_detection/protos/*.proto --python_out=.
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make
cp -r pycocotools <path_to_tensorflow>/models/research/
Export Environment
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64
export PYTHONPATH=$PYTHONPATH:~/tensorflow/models/research:~/tensorflow/models/research/object_detection/slim
Other
If we install CUDA 10.1 it may have a compatibility issue that libxxxx.so.10.0 is not found when you start training your project.
To solve them:
a. sudo ln -s /usr/lib/x86_64-linux-gnu/libcublas.so.10.1 /usr/local/cuda-10.1/lib64/libcublas.so.10.0
b. sudo ln -s /usr/local/cuda-10.1/lib64/libcudart.so.10.1 /usr/local/cuda-10.1/lib64/libcudart.so.10.0
c. sudo ln -s /usr/local/cuda-10.1/lib64/libcufft.so.10 /usr/local/cuda-10.1/lib64/libcufft.so.10.0
d. sudo ln -s /usr/local/cuda-10.1/lib64/libcurand.so.10 /usr/local/cuda-10.1/lib64/libcurand.so.10.0
e. sudo ln -s /usr/local/cuda-10.1/lib64/libcusolver.so.10 /usr/local/cuda-10.1/lib64/libcusolver.so.10.0
f. sudo ln -s /usr/local/cuda-10.1/lib64/libcusparse.so.10 /usr/local/cuda-10.1/lib64/libcusparse.so.10.0
All Set
I am trying to install httpd-2.4.1 on my CentOS machine (it has 2.2.15 currently). I did the following...
sudo yum install gcc
sudo yum install perl
sudo yum install libbz2 libbz2-devel
sudo yum install bzip2-devel
sudo yum install curl-devel
sudo yum install libjpeg-devel
sudo yum install libpng-devel
sudo yum install libX11-devel gd-devel
sudo yum install libc-client-devel
sudo yum install libmcrypt-devel
sudo yum install bzip2
sudo yum install pcre-devel
sudo yum install openssl-devel
sudo yum install libxml2-devel
sudo yum install libmcrypt
#apache
wget http://mirror.nus.edu.sg/apache//apr/apr-util-1.4.1.tar.gz
wget http://mirror.nus.edu.sg/apache//apr/apr-1.4.6.tar.gz
extract and move to /srclib in httpd-2.4.1 directory
sudo ./configure --prefix=/usr/local/apache --enable-so --enable-mods-shared=most --with-included-apr --with-included-apr-util --enable-load-all-modules
sudo make
sudo make install
And here I get the error:
/usr/bin/ld: /usr/local/apache/lib/libexpat.a(xmlparse.o): relocation R_X86_64_32S against `.text' can not be used when making a shared object; recompile with -fPIC
/usr/local/apache/lib/libexpat.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
libtool: install: error: relink `libaprutil-1.la' with the above command before installing it
How do I relink libaprutil-1.la with the above command? (i am guessing it refers to -fPIC?)
You didn't include --with-included-apr-util while running ./configure
./configure --with-included-apr --with-included-apr-util