I want to try out the develop branch of spaCy in order to test the features of v3. I built it successfully, but get the following message when trying to download a model:
'No compatible models found for v3.0.0 of spaCy'
What can I do? How are contributors supposed to get models for a non-released version?
This is the Dockerfile I used:
FROM python:3.8
RUN apt-get update && apt-get install -y \
git \
make \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
RUN git clone https://github.com/explosion/spaCy /spaCy
WORKDIR /spaCy
RUN git checkout develop
ENV PYTHONPATH=.
RUN pip install -r requirements.txt
RUN python setup.py build_ext --inplace
RUN python setup.py install
RUN python -m spacy download en_core_web_sm
Related
I want to have access to features from TensorFlow federated (tff.python.research) which aren't present with the pip3 install method.
I'm working on a remote server that does not have bazel, thus I cannot build from source. Are there other ways to get and install the latest working version of TFF from its GitHub REPO?
(https://github.com/tensorflow/federated)
To install the latest Tensorflow 2.0 federated, you may follow the steps below.
Install TensorFlow Federated using pip
Install the Python development environment
On Ubuntu:
$ sudo apt update
$ sudo apt install python3-dev python3-pip # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
On macOS:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
$ brew update
$ brew install python # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
Create a virtual environment
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment, run deactivate.
Install the TensorFlow Federated pip package.
(venv) $ pip install --upgrade tensorflow_federated
(Optional) Test Tensorflow Federated.
(venv) $ python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"
Build the TensorFlow Federated pip package
Install the Python development environment.
On Ubuntu:
$ sudo apt update
$ sudo apt install python3-dev python3-pip # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
On macOS:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
$ brew update
$ brew install python # Python 3
$ sudo pip3 install --upgrade virtualenv # system-wide install
Install Bazel
Install Bazel, the build tool used to compile Tensorflow Federated.
Clone the Tensorflow Federated repository.
$ git clone https://github.com/tensorflow/federated.git
$ cd "federated"
Create a virtual environment.
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment, run deactivate.
Install Tensorflow Federated dependencies.
(venv) $ pip install --requirement "requirements.txt"
(Optional) Test Tensorflow Federated.
(venv) $ bazel test //tensorflow_federated/...
Create a new project.
$ mkdir "/tmp/project"
$ cd "/tmp/project"
$ virtualenv --python python3 "venv"
$ source "venv/bin/activate"
(venv) $ pip install --upgrade pip
Note: To exit the virtual environment run deactivate.
Install the pip package.
(venv) $ pip install --upgrade "/tmp/tensorflow_federated/tensorflow_federated-"*".whl"
Test Tensorflow Federated.
(venv) $ python -c "import tensorflow_federated as tff; print(tff.federated_computation(lambda: 'Hello World')())"
Reference: https://www.tensorflow.org/federated/install
I am attempting to install a numpy dependancy inside a docker container. (My code heavily uses it). On building the container the numpy library simply does not install and the build fails. This is on OS raspbian-buster/stretch. This does however work when building the container on MAC OS.
I suspect some kind of python related issue, but can not for the life of me figure out how to make it work.
I should point out that removing the pip install numpy from the requirements file and using it in its own RUN statement in the dockerfile does not solve the issue.
The Dockerfile:
FROM python:3.6
ENV PYTHONUNBUFFERED 1
ENV APP /app
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN mkdir $APP
WORKDIR $APP
ADD requirements.txt .
RUN pip install -r requirements.txt
COPY . .
The requirements.txt contains all the project requirements, amounf which is numpy.
Step 6/15 : RUN pip install numpy==1.14.3
---> Running in 266a2132b078
Collecting numpy==1.14.3
Downloading https://files.pythonhosted.org/packages/b0/2b/497c2bb7c660b2606d4a96e2035e92554429e139c6c71cdff67af66b58d2/numpy-1.14.3.zip (4.9MB)
Building wheels for collected packages: numpy
Building wheel for numpy (setup.py): started
Building wheel for numpy (setup.py): still running...
Building wheel for numpy (setup.py): still running...
EDIT:
So after the comment by skybunk and the suggestion to head to official docs, some more debugging on my part, the solution wound up being pretty simple. Thanks skybunk to you go all the glory. Yay.
Solution:
Use alpine and install python install package dependencies, upgrade pip before doing a pip install requirements.
This is my edited Dockerfile - working obviously...
FROM python:3.6-alpine3.7
RUN apk add --no-cache --update \
python3 python3-dev gcc \
gfortran musl-dev \
libffi-dev openssl-dev
RUN pip install --upgrade pip
ENV PYTHONUNBUFFERED 1
ENV APP /app
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN mkdir $APP
WORKDIR $APP
ADD requirements.txt .
RUN pip install -r requirements.txt
COPY . .
To use Numpy on python3 here, we first head over to the official documentation to find what dependencies are required to build Numpy.
Mainly these 5 packages + their dependencies must be installed:
Python3 - 70 mb
Python3-dev - 25 mb
gfortran - 20 mb
gcc - 70 mb
musl-dev -10 mb (used for tracking unexpected behaviour/debugging)
An POC setup would look something like this -
Dockerfile:
FROM gliderlabs/alpine
ADD repositories.txt /etc/apk/repositories
RUN apk add --no-cache --update \
python3 python3-dev gcc \
gfortran musl-dev
ADD requirements-pip.txt .
RUN pip3 install --upgrade pip setuptools && \
pip3 install -r requirements-pip.txt
ADD . /app
WORKDIR /app
ENV PYTHONPATH=/app/
ENTRYPOINT python3 testscript.py
repositories.txt
http://dl-5.alpinelinux.org/alpine/v3.4/main
requirements-pip.txt
numpy
testscript.py
import numpy as np
def random_array(a, b):
return np.random.random((a, b))
a = random_array(2,2)
b = random_array(2,2)
print(np.dot(a,b))
To run this - clone alpine, build it using "docker build -t gliderlabs/alpine ."
Build and Run your Dockerfile
docker build -t minidocker .
docker run minidocker
Output should be something like this-
[[ 0.03573961 0.45351115]
[ 0.28302967 0.62914049]]
Here's the git link, if you want to test it out
From the error logs, it does not seem that it is from numpy. but you can install numpy before the requirment.txt and verify if it's working.
FROM python:3.6
RUN pip install numpy==1.14.3
Build
docker build -t numpy .
Run and Test
docker run numpy bash -c "echo import numpy as np > test.py ; python test.py"
So you will see no error on import.
or You can try numpy as an alpine package
FROM python:3-alpine3.9
RUN apk add --no-cache py3-numpy
Or better to post the requirement.txt.
I had lot of trouble with this issue using FROM python:3.9-buster and pandas.
My requirements.txt had the python-dev-tools, numpy and pandas, along with other packages.
I always got something like this when attempting to build:
preluded by:
and by:
Following hints by Adiii in this thread, I did some debug and found out that this actually works and builds a perfectly running container:
RUN pip3 install NumPy==1.18.0
RUN pip3 install python-dev-tools
RUN pip3 install pandas
RUN pip3 install -r requirements.txt
So, giving a specific RUN layer to the pip3 installing pandas solved the problem!
Another method is to install from the 'slim' distribution of python (based on debian):
FROM python:slim
CMD pip install numpy
123Mb
This results in a smaller image than that of alpine:
FROM python:3-alpine3.9
RUN apk add --no-cache py3-numpy
187MB
Plus it gives better support for other whl libraries for slim is based on a glibc library (against which all the wheels are built) while apline uses musl (incompatible with the wheels), so all packages will have to be either apk added or compiled from sources.
I am working through the Tensorflow serving_basic example at:
https://tensorflow.github.io/serving/serving_basic
Setup
Following: https://tensorflow.github.io/serving/setup#prerequisites
Within a docker container based off of ubuntu:latest, I have installed:
bazel:
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key
sudo apt-get update && sudo apt-get install bazel
sudo apt-get upgrade bazel
grpcio:
pip install grpcio
all packages:
sudo apt-get update && sudo apt-get install -y build-essential curl libcurl3-dev git libfreetype6-dev libpng12-dev libzmq3-dev pkg-config python-dev python-numpy python-pip software-properties-common swig zip zlib1g-dev
tensorflow serving:
git clone --recurse-submodules https://github.com/tensorflow/serving
cd serving
cd tensorflow
./configure
cd ..
I've built the source with bazel and all tests ran successfully:
bazel build tensorflow_serving/...
bazel test tensorflow_serving/...
I can successfully export the mnist model with:
bazel-bin/tensorflow_serving/example/mnist_export /tmp/mnist_model
And I can serve the exported model with:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_name=mnist --model_base_path=/tmp/mnist_model/
The problem
When I test the server and try to connect a client to the model server with:
bazel-bin/tensorflow_serving/example/mnist_client --num_tests=1000 --server=localhost:9000
I see this output:
root#dc3ea7993fa9:~/serving# bazel-bin/tensorflow_serving/example/mnist_client --num_tests=2 --server=localhost:9000
Extracting /tmp/train-images-idx3-ubyte.gz
Extracting /tmp/train-labels-idx1-ubyte.gz
Extracting /tmp/t10k-images-idx3-ubyte.gz
Extracting /tmp/t10k-labels-idx1-ubyte.gz
AbortionError(code=StatusCode.NOT_FOUND, details="FeedInputs: unable to find feed output images")
AbortionError(code=StatusCode.NOT_FOUND, details="FeedInputs: unable to find feed output images")
Inference error rate is: 100.0%
The "--use_saved_model" model flag is set to default "true"; use the --use_saved_model=false when starting the server. This should work:
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --use_saved_model=false --port=9000 --model_name=mnist --model_base_path=/tmp/mnist_model/
I mentioned this on the tensorflow github, and the solution was to remove the original model that had been created. If you're running into this, run
rm -rf /tmp/mnist_model
and rebuild it
I installed the latest Tensorflow 0.5.0 from source via git clone.
and want to update to Tensorflow 0.6.0
git pull
./configure
bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
but the Tensorflow library in the directory /usr/lib/python2.7/site-packages still has the version 0.5.0
the version in the result of "pip show tensorflow" also is 0.5.0
To install the TensorFlow library from source, you need to build a PIP package and install it. The steps are as follows:
$ git pull
$ ./configure
$ bazel build -c opt //tensorflow/tools/pip_package:build_pip_package
# ...or, with GPU support
$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
# The name of the .whl file will depend on your platform.
$ pip install /tmp/tensorflow_pkg/tensorflow-0.6.0-cp27-none-linux_x86_64.whl
git pull doesn't work for me since some local files are modified by the last build so with a slight modification I update like this:
git fetch --all
git reset --hard origin/master
./configure
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
sudo pip install /tmp/tensorflow_pkg/tensorflow-0.8.0-py2-none-any.whl
Tested to work as of today. The Installation from Source instructions in tensorflow docs are misleading in the sense they only include the real pip wheel installation commands for Mac and the example-trainer build command exists instead in Linux instructions.
To show the version:
python -c "import tensorflow; print(tensorflow.__version__);"
And if it is not the latest, you have uninstall it via pip uninstall:
sudo pip uninstall tensorflow
and subsequently install it:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl
sudo pip install $TF_BINARY_URL
Before trying to update tensorflow try updating pip
pip install --upgrade pip
If you are upgrading from a previous installation of TensorFlow < 0.7.1, you should uninstall the previous TensorFlow and protobuf using,
pip uninstall
first to make sure you get a clean installation of the updated protobuf dependency.
Uninstall the TensorFlow on your system, and check out Download and Setup to reinstall again.
If you are using pip install, go check the available version over https://storage.googleapis.com/tensorflow, search keywords with linux/cpu/tensorflow to see the availabilities.
Then, set the path for download and execute in sudo.
$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.0.0-py2-none-any.whl
$ sudo pip install --upgrade $TF_BINARY_URL
For more detail, follow this link in here
If you get the error saying not a supported wheel on this platform. You might be updating tensorflow for python3. For that you will need pip3
Try installing pip3
sudo apt-get -y install python3-pip
Then, set the path for download if you haven't already set the path
$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.1.0rc0-cp35-cp35m-linux_x86_64.whl
$ pip3 install --ignore-installed --upgrade $TF_BINARY_URL
updating tensorflow install with sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl.
I find it in the below issue,mohamed-ali's comment. https://github.com/tensorflow/tensorflow/issues/1105
Users sometimes need to know how to install a newer version of Pandas than their OS package manager offers. Pandas requires NumPy, and works best with SciPy, Matplotlib and IPython.
How can I install the latest versions of NumPy/Scipy/Matplotlib/IPython/Pandas?
Using Ubuntu, here is how to install the entire NumPy/Scipy/Matplotlib/IPython/Pandas
stack from Github in a virtualenv using Python2.7:
Note: The instructions below install the latest development version of each package. If you wish to install the latest tagged version, then after git clone, inspect the tags available with
git tag
and select the version you wish to install with
git checkout tag-name
Install virtualenv and virtualenvwrapper:
sudo apt-get install python-virtualenv
sudo pip install virtualenvwrapper
# edit ~/.bashrc to include
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
# edit ~/.profile to include
export WORKON_HOME=$HOME/.virtualenvs
# You may have to log out then log back in to make the change effective
Make a virtualenv
mkvirtualenv --system-site-packages dev
workon dev
# If you want to make this virtual environment your default Python,
# edit ~/.bashrc to include
workon dev
Add site-packages to sys.path:
add2virtualenv $USER/.virtualenvs/dev/lib/python2.7/site-packages
Install Cython
pip install -U Cython
Install git
sudo apt-get install git
Install NumPy
cd ~/src
git clone https://github.com/numpy/numpy.git
sudo apt-get install python-dev build-essential
sudo apt-get install libatlas-base-dev libatlas3gf-base
# ensure clean build
# this is not necessary the first time, but useful when upgrading
cd ~/src/numpy
/bin/rm -rf ~/src/numpy/build
cdsitepackages && /bin/rm -rf numpy numpy-*-py2.7.egg-info
cd ~/src/numpy
python setup.py build --fcompiler=gnu95
python setup.py install
Install SciPy
cd ~/src
git clone https://github.com/scipy/scipy.git
# ensure clean build
cd ~/src/scipy
/bin/rm -rf ~/src/scipy/build
cdsitepackages && /bin/rm -rf scipy scipy-*-py2.7.egg-info
cd ~/src/scipy
git clean -xdf
python setup.py install
Install Matplotlib dependencies
pip install -U pyparsing
pip install -U six
pip install -U python-dateutil
pip install -U pytz
sudo apt-get install libzmq-dev
pip install -U tornado pygments pyzmq
pip install -U nose
sudo apt-get install python-qt4 python-qt4-doc python-pyside python-cairo python-wxgtk2.8 python-gtk2 dvipng
apt-cache depends python-matplotlib | awk '/Depends:/{print $2}' | xargs dpkg --get-selections
sudo apt-get build-dep python-matplotlib
Install Matplotlib
cd ~/src/
git clone https://github.com/matplotlib/matplotlib
# ensure clean build
cd ~/src/matplotlib
/bin/rm -rf ~/src/matplotlib/build
cdsitepackages && /bin/rm -rf matplotlib* mpl_toolkits
# compile and install
cd ~/src/matplotlib
python setup.py build
python setup.py install
Install IPython
cd ~/src
git clone https://github.com/ipython/ipython.git
# ensure clean build
cd ~/src/ipython
/bin/rm -rf ~/src/ipython/build
cdsitepackages && /bin/rm -rf ipython-*-py2.7.egg
cd ~/src/ipython
python setupegg.py install
Install Pandas
cd ~/src
git clone https://github.com/pydata/pandas.git
cd ~/src/pandas
# update
git fetch origin
git rebase --interactive origin/master
# ensure clean build and install
/bin/rm -rf ~/src/pandas/{build,dist} && cdsitepackages && /bin/rm -rf pandas* && cd ~/src/pandas && python setup.py build_ext --inplace && python setup.py install
Updating:
The advantage of
the git approach is that it provides a way to always keep these packages
up-to-date:
cd ~/src/package-name
git fetch origin
git rebase --interactive origin/master
Follow the instructions above to ensure a clean build, and then rebuild and
reinstall the package.
Shorthand for using pip with GitHub directly
The above steps to clone and install packages can be automated to an extent with pip. For example, we can also install NumPy like this:
pip install git+git://github.com/numpy/numpy.git
The updating would then be just
pip install numpy --upgrade --force-reinstall
--force-reinstall flag may be needed because pip checks the version from PyPI and doesn't update if the current version isn't smaller.
Via the Anaconda distribution:
Download and install
wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
chmod +x miniconda.sh
./miniconda.sh -b
export PATH=/home/travis/miniconda/bin:$PATH
conda update conda --yes
Install just the packages in the title in their own environment:
conda create --name myenv --yes python=3.4 pandas matplotlib ipython-notebook
source activate myenv
Note: I believe anaconda supports Python versions 2.6, 2.7, 3.3, and 3.4.