Theano fails due to NumPy Fortran mixup under Ubuntu - numpy

I installed Theano on my machine, but the nosetests break with a Numpy/Fortran related error message. For me it looks like Numpy was compiled with a different Fortran version than Theano. I already reinstalled Theano (sudo pip uninstall theano + sudo pip install --upgrade --no-deps theano) and Numpy / Scipy (apt-get install --reinstall python-numpy python-scipy), but this did not help.
What steps would you recommend?
Complete error message:
ImportError: ('/home/Nick/.theano/compiledir_Linux-2.6.35-31-generic-x86_64-with-Ubuntu-10.10-maverick--2.6.6/tmpIhWJaI/0c99c52c82f7ddc775109a06ca04b360.so: undefined symbol: _gfortran_st_write_done'
My research:
The Installing SciPy / BuildingGeneral page about the undefined symbol: _gfortran_st_write_done' error:
If you see an error message
ImportError: /usr/lib/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done
when building SciPy, it means that NumPy picked up the wrong Fortran compiler during build (e.g. ifort).
Recompile NumPy using:
python setup.py build --fcompiler=gnu95
or whichever is appropriate (see python setup.py build --help-fcompiler).
But:
Nick#some-serv2:/usr/local/lib/python2.6/dist-packages/numpy$ python setup.py build --help-fcompiler
This is the wrong setup.py file to run
Used software versions:
scipy 0.10.1 (scipy.test() works)
NumPy 1.6.2 (numpy.test() works)
theano 0.5.0 (several tests fails with undefined symbol: _gfortran_st_write_done')
python 2.6.6
Ubuntu 10.10
[UPDATE]
So I removed numpy and scipy from my system with apt-get remove and using find -name XXX -delete of what was left.
Than I installed numpy and scipy from the github sources with sudo python setpy.py install.
Afterwards I entered again sudo pip uninstall theano and sudo pip install --upgrade --no-deps theano.
Error persists :/
I also tried the apt-get source ... + apt-get build-dep ... approach, but for my old Ubuntu (10.10) it installs too old version of numpy and scipy for theano: ValueError: numpy >= 1.4 is required (detected 1.3.0 from /usr/local/lib/python2.6/dist-packages/numpy/__init__.pyc)

I had the same problem, and after reviewing the source code, user212658's answer seemed like it would work (I have not tried it). I then looked for a way to deploy user212658's hack without modifying the source code.
Put these lines in your theanorc file:
[blas]
ldflags = -lblas -lgfortran
This worked for me.

Have you tried to recompile NumPy from the sources?
I'm not familiar with the Ubuntu package system, so I can't check what's in your dist-packages/numpy. With a clean archive of the NumPy sources, you should have a setup.py at the same level as the directories numpy, tools and benchmarks (among others). I'm pretty sure that's the one you want to use for a python setup.py build.
[EDIT]
Now that you have recompiled numpy with the proper --fcompiler option, perhaps could you try to do the same with Theano, that is, compiling directly from sources without relying on a apt-get or even pip. You should have a better control on the build process that way, which will make debugging/trying to find a solution easier.

I had the same problem. The solution I found is to add a hack in theano/gof/cmodule.py to link against gfortran whenever 'blas' is in the libs. That fixed it.
class GCC_compiler(object):
...
#staticmethod
def compile_str(module_name, src_code, location=None,
include_dirs=None, lib_dirs=None, libs=None,
preargs=None):
...
cmd.extend(['-l%s' % l for l in libs])
if 'blas' in libs:
cmd.append('-lgfortran')

A better fix is to remove atlas and install openblas. openblas is faster then atlas. Also, openblas don't request gfortran and is the one numpy was linked with. So it will work out of the box.

Related

Pandas incompatible with numpy

I am using anaconda 3. When I try to import pandas I receive the following message:
ImportError: this version of pandas is incompatible with numpy < 1.15.4
your numpy version is 1.15.3.
Please upgrade numpy to >= 1.15.4 to use this pandas version
Printing numpy.__path__ gives me the following
['C:\Users\andrei\AppData\Roaming\Python\Python37\site-packages\numpy']
In conda list, my numpy version is 1.19.1. I checked the above directory to find that it has only numpy 1.15.3 inside and nothing else. Spyder is using this path instead of the anaconda's path to numpy for some arcane reason.
Looks like you have somehow installed several versions of NumPy. Try to remove them all by running several times conda remove numpy and pip uninstall numpy. If you have two versions, the corresponding uninstall command needs to be run twice. After these, install a fresh version of NumPy conda install numpy
You can verify if you still have a version of NumPy installed
conda list | grep numpy
pip list | grep numpy
Note that these commands show only one version number even if you have several copies installed.
You can use conda to upgrade to upgrade your numpy. Run this command in the terminal:
conda update numpy
You need to remove this directory
C:\Users\andrei\AppData\Roaming\Python\
to fix this problem. It seems at some point you used pip to install numpy and that's interfering with the packages installed by conda (which is reporting the right version, as you said).
Furthermore, please be aware that pip and conda packages are binary incompatible, so you should avoid as much as possible to mix them.

`save_model` requires h5py error occurss after installing h5py and cython packages too

I need to save my new sequential model but when I use the model.save(filename),it shows error like save_model requires hp5y.I tried installing h5py in conda by 'conda install -c anaconda h5py'command. And I also installed cython,but then the error exists .what should I do?
one should include the package in pycharm project interpreter after being downloaded in the conda environment.

numpy installation into pypy3 virtual env : `undefined symbol: cblas_sgemm`

I am trying to install numpy into a pypy3 virtualenv, but I'm stuck with that error (at importing) :
venv_pypy/site-packages/numpy-1.16.0.dev0+1d38e41-py3.5-linux-x86_64.egg/numpy/core/_multiarray_umath.pypy3-60-x86_64-linux-gnu.so: undefined symbol: cblas_sgemm
I am on an up to date archlinux, numpy works fine with CPython, but I have a project using pandas (which depends on numpy) that I need to test on pypy.
I first tried the recommended method (pip install numpy in the venv) but didn't work. (install is fine, but still the same error at execution).
I then, I tried what is suggested https://stackoverflow.com/a/14391693/1745291 (linked from Numpy multiarray.so: undefined symbol: cblas_sgemm ), since I didn't installed ATLAS (aur package on arch I don't want to install), to try building with OpenBias. But still not working (same error, and the method could be outdated since it's from 2013)
...And finally, I tried a build with no accelerations (at least, that is claimed) following : https://docs.scipy.org/doc/numpy-1.15.0/user/building.html#disabling-atlas-and-other-accelerated-libraries
...But still the same result...
What am i doing wrong ?
You can try uninstall it from pip and install from apt (if you are using ubuntu etc.)
This approach solved my problem
pip3 uninstall numpy
sudo apt-get install python3-numpy

Is it possible to install scipy under pypy?

I created a virtualenv with pypy and tried to install scipy, but installation ended with the following error:
distutils.errors.DistutilsError: Setup script exited with error: Command "cc -O2 -fPIC -Wimplicit -O2 -fPIC -Wimplicit -O2 -fPIC -Wimplicit -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DNO_ATLAS_INFO=1 -DHAVE_CBLAS -Ibuild/src.linux-x86_64-3.2/numpy/core/src/private -Inumpy/core/include -Ibuild/src.linux-x86_64-3.2/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/luke/Programowanie/Python/connect4/venv-pypy/include -Ibuild/src.linux-x86_64-3.2/numpy/core/src/private -Ibuild/src.linux-x86_64-3.2/numpy/core/src/private -Ibuild/src.linux-x86_64-3.2/numpy/core/src/private -c numpy/core/src/multiarray/scalarapi.c -o build/temp.linux-x86_64-3.2/numpy/core/src/multiarray/scalarapi.o" failed with exit status 1
There is a specific numpy version suited for pypy, but the only thing I found about scipy is an old call for donations. Has anything changed since then?
Yes, it is possible, starting from Scipy 1.1.0. New enough PyPy (>= 6.0.0) and Numpy are however required, numpy>= 1.14.3, and preferably numpy>=1.15.0 when it's released. Installation can be done via the usual
pypy3 -mpip install numpy
pypy3 -mpip install scipy
assuming you have BLAS/LAPACK installed, so first you should make sure you are able to install numpy/scipy on normal Python from sources.
Nobody probably has tried whether this works with Pypy on Windows, so you are on your own there. It also probably won't work with NumPyPy --- you need the vanilla Numpy.
The other answers say "No" since that was the situation before May 2018.
No. SciPy has bits and pieces talking to cpython's C API, and pypy does not support that.
EDIT: As of May 2018, this is incorrect, and scipy is pypy-compatible. See the answer by #pv. for details.
Seems that the link in #piv.s great answer does not contain what is needed anymore.
For installing BLAS/LAPACK and avoiding installation errors such as NotFoundError: no lapack/blas resources found on Ubuntu first run:
sudo apt-get install libatlas-base-dev
Then using the pip that corresponds to your pypy (not the system's):
pip install numpy
pip install scipy
or the one mentioned by piv. will do the trick.
No, but miraculously, matplotlib happens to work with pypy, both in jupyter notebook and as a standalone version (through wx widgets).
So it isn't that impossible.
As of this writing, numpy and scipy are installable with PyPy through pip, but there's a catch: if you're going to run that within a virtualenv (which you probably should), remember to create the virtualenv with "--always-copy", otherwise virtualenv will create a symlink for the "include" dir, making it read-only for the user, which will break the installation for pybind11 (which is a dependency for scipy).

Matplotlib install failure on Mac OSX 10.8 Mountain Lion

I tried to install matplotlib on my MacBook Air, but it always gives me this error message:
Processing matplotlib-1.1.1_notests.tar.gz
Running matplotlib-1.1.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3jFpXK/matplotlib-1.1.1/egg-dist-tmp-jC7QY3
basedirlist is: []
============================================================================
BUILDING MATPLOTLIB
matplotlib: 1.1.1
python: 2.7.2 (default, Jun 20 2012, 16:23:33) [GCC 4.2.1
Compatible Apple Clang 4.0
(tags/Apple/clang-418.0.60)]
platform: darwin
REQUIRED DEPENDENCIES
numpy: 1.6.1
freetype2: found, but unknown version (no pkg-config)
* WARNING: Could not find 'freetype2' headers in any
* of '.', './freetype2'.
OPTIONAL BACKEND DEPENDENCIES
libpng: found, but unknown version (no pkg-config)
* Could not find 'libpng' headers in any of '.'
Tkinter: Tkinter: version not identified, Tk: 8.5, Tcl: 8.5
Gtk+: no
* Building for Gtk+ requires pygtk; you must be able
* to "import gtk" in your build/install environment
Mac OS X native: yes
Qt: no
Qt4: no
PySide: no
Cairo: no
OPTIONAL DATE/TIMEZONE DEPENDENCIES
datetime: present, version unknown
dateutil: 1.5
pytz: matplotlib will provide
adding pytz
OPTIONAL USETEX DEPENDENCIES
dvipng: 1.14
ghostscript: 9.05
latex: 3.1415926
[Edit setup.cfg to suppress the above messages]
pymods ['pylab']
packages ['matplotlib', 'matplotlib.backends', 'matplotlib.backends.qt4_editor', 'matplotlib.projections', 'matplotlib.testing', 'matplotlib.testing.jpl_units', 'matplotlib.tests', 'mpl_toolkits', 'mpl_toolkits.mplot3d', 'mpl_toolkits.axes_grid', 'mpl_toolkits.axes_grid1', 'mpl_toolkits.axisartist', 'matplotlib.sphinxext', 'matplotlib.tri', 'matplotlib.delaunay', 'pytz']
warning: no files found matching 'KNOWN_BUGS'
warning: no files found matching 'INTERACTIVE'
warning: no files found matching 'MANIFEST'
warning: no files found matching '__init__.py'
warning: no files found matching 'examples/data/*'
warning: no files found matching 'lib/mpl_toolkits'
warning: no files found matching 'LICENSE*' under directory 'license'
clang: warning: argument unused during compilation: '-mno-fused-madd'
In file included from src/ft2font.cpp:3:
src/ft2font.h:16:10: fatal error: 'ft2build.h' file not found
**#include <ft2build.h>
^
1 error generated.
error: Setup script exited with error: command 'clang' failed with exit status 1**
I tried to install freetype and libpng using homebrew but it doesn't work. How can I get ft2build.h?
The following worked for matplotlib installation after installing python according to instructions from thegreenroom. Those instructions didn't work for me after I installed Python. I followed the instructions from Scipy.org to install numpy and scipy. Then I did (adapted from above answer):
brew install freetype
brew install libpng
However I got the same error message whether I installed with pip install matplotlib or trying to install from source, doing
python setup.py build
python setup.py install
in the matplotlib directory I cloned via git clone https://github.com/matplotlib/matplotlib.git.
The error persisted until I ran
brew link freetype
Then from the cloned matplotlib directory I ran
python setup.py build
python setup.py install
And the installation succeeded.
This may help folks looking for a non-homebrew solution.
My goal: use pip install to build matplotlib for a non-system python 2.7.3 build.
Using latest X-Code and X-Code command line tools as of Feb 2013, no matter what gymnastics I tried, I always received C++ ostream related template errors when compiling ft2build with gcc.
I was able to get a pip install to work with the following env vars:
export CC=clang
export CXX=clang++
export LDFLAGS="-L/usr/X11/lib"
export CFLAGS="-I/usr/X11/include -I/usr/X11/include/freetype2"
I simply forced clang and added my xquartz paths. No extra pkg-config or libpng builds, no sudo-ed symlinks.
I think the other answers are on the right track, but I encountered this same problem and can attest that:
brew install pkg-config
brew install freetype
pip install matplotlib
would yield the same result. Typically on an Ubuntu box my next response would have been
sudo apt-get install libfreetype-dev
or some variation of that to install the header. However, I could find no such homebrew package. Furthermore, I was able to locate the header file in question in a pretty normal location on my system:
zoidberg:~ matt$ locate ft2build.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/X11/include/ft2build.h
/usr/X11/include/ft2build.h
So I suspect there's either a problem with my system paths or with the homebrew packaged matplotlib. Since I'm lazy, I just tried installing the matplotlib package head from github:
pip install git+git://github.com/matplotlib/matplotlib.git#egg=matplotlib-dev
and it worked for me.
Here's the brew + pip recipe I used from a cold start. If you already have python and gfortran and such, jump in at the point where you need. The crucial steps appear to be brew install freetype and brew install libpng prior to doing pip install matplotlib
$ ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"
$ brew doctor
$ brew install python
$ export PATH=/usr/local/bin:/usr/local/share/python:$PATH
$ easy_install pip
$ brew install gfortran
$ pip install numpy
$ pip install scipy
$ brew install pkg-config
$ brew install freetype
$ brew install libpng
$ pip install matplotlib
$ python
>>> import numpy
>>> import scipy
>>> import matplotlib
Note the 'no pkg-config' notices. You should have pkg-config on your search path, and it presumably needs to be the homebrew version so that it knows where the homebrew versions of the libraries are.
You need freetype:
brew install freetype
See the following:
http://comments.gmane.org/gmane.comp.python.matplotlib.general/31394
I also use mac air with OS X ver 10.8.2.
Using following commands can get rid of this failure:
brew install freetype
brew install libpng
pip install matplotlib
That's all. There may be some warning in the installing process, but that does not affect.
Use following python code to test:
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
which is referred from the url: http://matplotlib.org/users/pyplot_tutorial.html.
I have found installing these pacakges via homebrew to be the most reliable method
# if you haven't installed python via brew already:
brew install python
# sets up python as default python instead of system python
brew link python
# Add more brew formulae so we can install our py libs with brew
brew tap samueljohn/python
brew tap homebrew/science
# install numpy,scipy,matplotlib and dependencies ( gfortran, etc.. )
brew install numpy
brew install scipy
brew insatll matplotlib
I am using MacOs 10.8 too, and I encountered the same problem regarding not finding ft2build.h header when I tried to install matplotlib, what I l did to resolve the reported issue is :
1) replacing "ft2build.h" in the following call within the file "setupext.py" :
return self._check_for_pkg_config('freetype2',
'ft2build.h',
min_version='2.4',
version=version)
by the complete path to the ft2build.h header, in my case :
return self._check_for_pkg_config('freetype2',
'/usr/local/Cellar/freetype/2.5.2/include/freetype2/ft2build.h',
min_version='2.4',
version=version)
I think, but have not tested, that the problem is solved in an upstream commit of matplotlib. Here is my reasons why:
Discussion:
https://github.com/vbraun/sage/commit/5d17ca989eb58559af8f43b43e368c378c1bf6bb
Fix:
https://github.com/vbraun/sage/blob/5d17ca989eb58559af8f43b43e368c378c1bf6bb/build/pkgs/matplotlib/patches/pkg-config.patch
I fixed the problem with Cyris's answer thought.
I also had a similar issue, and fixed it in a reasonably straightforward way using homebrew.
You don't have to link the freetype libraries for this reason. The main problem is, after you install freetype using homebrew, you are told to add
-I/usr/local/opt/freetype/include
to your CPPFLAGS. However, this alone is not sufficient, and you also have to add the the freetype2 subfolder as well.
So in order to install matplotlib, do this:
brew install freetype
brew install libpng
LDFLAGS="-L/usr/local/opt/freetype/lib -L/usr/local/opt/libpng/lib" CPPFLAGS="-I/usr/local/opt/freetype/include -I/usr/local/opt/libpng/include -I/usr/local/opt/freetype/include/freetype2" pip install matplotlib