# Install RAPIDS
!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!bash rapidsai-csp-utils/colab/rapids-colab.sh stable
import sys, os, shutil
sys.path.append('/usr/local/lib/python3.7/site-packages/')
os.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'
os.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'
os.environ["CONDA_PREFIX"] = "/usr/local"
for so in ['cudf', 'rmm', 'nccl', 'cuml', 'cugraph', 'xgboost', 'cuspatial']:
fn = 'lib'+so+'.so'
source_fn = '/usr/local/lib/'+fn
dest_fn = '/usr/lib/'+fn
if os.path.exists(source_fn):
print(f'Copying {source_fn} to {dest_fn}')
shutil.copyfile(source_fn, dest_fn)
# fix for BlazingSQL import issue
# ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /usr/local/lib/python3.7/site-packages/../../libblazingsql-engine.so)
if not os.path.exists('/usr/lib64'):
os.makedirs('/usr/lib64')
for so_file in os.listdir('/usr/local/lib'):
if 'libstdc' in so_file:
shutil.copyfile('/usr/local/lib/'+so_file, '/usr/lib64/'+so_file)
shutil.copyfile('/usr/local/lib/'+so_file, '/usr/lib/x86_64-linux-gnu/'+so_file)
Im successfully able to install RAPIDS using the above script but I simply cant get ride of the following error:
Exception: Version mismatch: this is the 'cffi' package version 1.14.5, located in '/usr/local/lib/python3.7/dist-packages/cffi/api.py'. When we import the top-level '_cffi_backend' extension module, we get version 1.14.3, located in '/usr/local/lib/python3.7/site-packages/_cffi_backend.cpython-37m-x86_64-linux-gnu.so'. The two versions should be equal; check your installation.
I have tried everything from here and enter link description here, upgraded, downgraded, uninstalled, installed but nothing works. Any help would be greatly appreciated.
Related
I am working on some OpenAI's API integrations. I am facing an issue while installing the tiktoken (a fast BPE tokeniser for use with OpenAI's models).
When I try to run on my macbook machine:
pip3 install tiktoken -v
It throws the following error (Using pip 22.3.1, error truncated because very long):
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
Perhaps try: xcode-select --install
*********************************************************************************
error: command '/usr/bin/clang' failed with exit code 1
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /Users/gianlucatrentin/gptenv/bin/python3 -u -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/lxml_b78f1cdebea441d2b840e6957fabe83a/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-wheel-sd9x1ng7
cwd: /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/lxml_b78f1cdebea441d2b840e6957fabe83a/
Building wheel for lxml (setup.py) ... error
ERROR: Failed building wheel for lxml
Running setup.py clean for lxml
Running command python setup.py clean
Building lxml version 4.9.2.
Building without Cython.
Building against libxml2 2.9.13 and libxslt 1.1.35
running clean
removing 'build/temp.macosx-10.9-universal2-cpython-310' (and everything under it)
removing 'build/lib.macosx-10.9-universal2-cpython-310' (and everything under it)
'build/bdist.macosx-10.9-universal2' does not exist -- can't clean it
'build/scripts-3.10' does not exist -- can't clean it
removing 'build'
...
× Running setup.py install for pycryptodomex did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /Users/gianlucatrentin/gptenv/bin/python3 -u -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/pycryptodomex_7192f9e9e3b34b15b1960598d9d7eb02/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' install --record /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-record-n_aypwf1/install-record.txt --single-version-externally-managed --compile --install-headers /Users/gianlucatrentin/gptenv/include/site/python3.10/pycryptodomex
cwd: /private/var/folders/sc/gl1552412gb0lg7_x33zrgj00000gn/T/pip-install-l7tec_bv/pycryptodomex_7192f9e9e3b34b15b1960598d9d7eb02/
Running setup.py install for pycryptodomex ... error
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> pycryptodomex
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
I try to upgrade all the mentione libraries but couldn't make it work.
Do you have suggestions on how to solve this installing issue?
I have always used following commands to install Rapids on Colab (from https://colab.research.google.com/drive/1rY7Ln6rEE1pOlfSHCYOVaqt8OvDO35J0#forceEdit=true&offline=true&sandboxMode=true)
!git clone https://github.com/rapidsai/rapidsai-csp-utils.git
!python rapidsai-csp-utils/colab/env-check.py
!bash rapidsai-csp-utils/colab/update_gcc.sh
import os
os._exit(00)
import condacolab
condacolab.install()
import condacolab
condacolab.check()
# Installing RAPIDS is now 'python rapidsai-csp-utils/colab/install_rapids.py <release> <packages>'
# The <release> options are 'stable' and 'nightly'. Leaving it blank or adding any other words will default to stable.
!python rapidsai-csp-utils/colab/install_rapids.py stable
import os
os.environ['NUMBAPRO_NVVM'] = '/usr/local/cuda/nvvm/lib64/libnvvm.so'
os.environ['NUMBAPRO_LIBDEVICE'] = '/usr/local/cuda/nvvm/libdevice/'
os.environ['CONDA_PREFIX'] = '/usr/local'
it always worked, but lately I get
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
CondaHTTPError: HTTP 403 FORBIDDEN for url <https://conda.anaconda.org/rapidsai/linux-64/ucx-1.11.2 gef2bbcf-cuda11.2_0.tar.bz2>
Elapsed: 00:00.358595
I have retried several times but it doesnt work, how can I solve it?
There are some answers to this question in a Python environment, but the solutions did not work for my RStudio environment. Here is my code:
library(keras)
library(tensorflow)
use_condaenv("tf")
train_dir = "C:/training_images/"
train_datagen <- image_data_generator(rescale = 1/255)
validation_datagen <- image_data_generator(rescale = 1/255)
train_generator <- flow_images_from_directory(
train_dir,
train_datagen,
target_size = c(150, 150),
batch_size = 20,
class_mode = "binary"
)
batch <- generator_next(train_generator)
The code works until the last "batch" line where it explodes like this:
Error in py_iter_next(it, completed) :
ImportError: Could not import PIL.Image. The use of `load_img` requires PIL.
Detailed traceback:
File "C:\Users\mory3\ANACON~1\envs\tf\lib\site-packages\keras_preprocessing\image\iterator.py", line 104, in __next__
return self.next(*args, **kwargs)
File "C:\Users\mory3\ANACON~1\envs\tf\lib\site-packages\keras_preprocessing\image\iterator.py", line 116, in next
return self._get_batches_of_transformed_samples(index_array)
File "C:\Users\mory3\ANACON~1\envs\tf\lib\site-packages\keras_preprocessing\image\iterator.py", line 230, in _get_batches_of_transformed_samples
interpolation=self.interpolation)
File "C:\Users\mory3\ANACON~1\envs\tf\lib\site-packages\keras_preprocessing\image\utils.py", line 108, in load_img
raise ImportError('Could not import PIL.Image. '
R version 3.6.1
Conda version 4.7
Python version 3.7
I had this same problem
After a few hours of looking, I came up with a solution that worked for me.
I used this code for solving the PIL problem. I tried using anaconda prompt but this code worked in r for me...
reticulate::py_install("pillow",env=tf)
I came up with this error next...
loaded runtime CuDNN library: 7.4.2 but source was compiled with: 7.6.0.
Make sure you have the correct cudnn version installed. For me it was CUDA 10 with 7.6.0 cudnn with 10. The output of the error will tell you which one to use.
Make sure you have cleaned any extra path variables that are in your environmental variables from installing previous versions.
I'm using windows 10
gpu = GeForce GTX 1060 with Max-Q Design
R - 3.6.1
tensorflow = 1.13
python = 3.7
anaconda = Anaconda3–2019.03-Windows-x86_64.exe
I ended up uninstalling Anaconda altogether, which made troubleshooting the remaining errors in the Python connection to R much simpler.
I had same problem with "Deep Learning with R" CNN example on Win7. I solved it like this:
I added Anaconda3 paths to PATH. In my case it was Windows so paths were like that:
C:\Anaconda3\Scripts;C:\Anaconda3\Library\bin By default there were no paths to conda in $PATH.
installed pillow (it contains PIL) to python with:
pip install pillow
configured r-reticulate.
This answer Could not import PIL.Image even if Pillow already installed? helped me. I had pillow already but conda environment wasn't configured properly so pillow wasn't visible.
Also install Nvidia CUDA if you don't have it - you need it too for tensorflow.
Here is my code:
import os
import pandas as pd
def load_hdf(filename):
"""
Load the first key of an HDF file
"""
hdf = pd.HDFStore(filename,mode = 'r')
keys = hdf.keys()
if not keys:
hdf.close()
return pd.DataFrame()
data_df = hdf.get(keys[0])
hdf.close()
return data_df
And when I do:
load_hdf(os.path.join(PATH, 'crm.hd5'))
I have this error:
HDFStore requires PyTables, "No module named 'tables'" problem importing
When I try:
pip install tables
I have the error:
Using Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
* USE_PKGCONFIG: False
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and library can be found by setting the ``HDF5_DIR`` environment variable or by using the ``--hdf5`` command-line option.
...
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/2s/sn3gzfwd6_37v0ggqd0n8qy00000gn/T/pip-install-1mx6wjd3/tables/
I already have Pytables, hdf5 in my Anaconda. I have Python 3.7.
I also had pytables installed and could not find a solution. What worked for me is to install the release candidate of HDF5 2.8.0rc1 (as seen here). Seems that the HDF5 version that panda installs is not fully compatible.
So try:
pip install h5py==2.8.0rc1
Hope it helps.
I am unable to import matplotlib._png:
import matplotlib._png as _png ImportError:
/home/james/opt/python/virtualenvs/work/lib/python2.7/site-packages/matplotlib-1.3.x-py2.7-linux-x86_64.egg/matplotlib/_png.so:
undefined symbol: png_set_longjmp_fn
This error prevents me from running import pylab (sincce this ultimately imports matplotlib._png).
I installed matplotlib from source, and made sure to add the path with local installations (/home/james/local) to basedir in setupext.py before running python setup.py install.
REQUIRED DEPENDENCIES AND EXTENSIONS
numpy: yes [version 1.7.1]
dateutil: yes [using dateutil version 2.1]
tornado: yes [using tornado version 3.0.1]
pyparsing: yes [using pyparsing version 1.5.7]
pycxx: yes [Couldn't import. Using local copy.]
libagg: yes [pkg-config information for 'libagg' could not
be found Using local copy.]
freetype: yes [version 16.0.10]
png: yes [version 1.2.10]
My research so far:
As can be seen above, matplotlib seems to find version 1.2.10 even though the version that I have under /home/james/local is 1.6.2:
$ find . -iname '*libpng*'
./libpng16.so.16.1.0
./libpng16.so
./libpng16.so.16
./libpng16.a
./libpng.a
./libpng.so
./libpng16.la
./pkgconfig/libpng.pc
./pkgconfig/libpng16.pc
./libpng.la
More specifically, I modified the following line in setupext.py with:
return basedir_map.get(sys.platform, ['/home/james/local', '/usr/local', '/usr'])
but matplotlib seems to have found the system version:
$ locate libpng
/usr/lib/libpng.so
/usr/lib/libpng.so.3
/usr/lib/libpng.so.3.10.0
/usr/lib/libpng12.a
/usr/lib/libpng12.so
/usr/lib/libpng12.so.0
/usr/lib/libpng12.so.0.10.0
Could this be the problem? Why am I unable to import matplotlib._png?
Update:
Looking at setupext.py, it looks like python setup install queries pkg-config through the SetupPackage method _check_for_pkg_config to determine the version of libpng I have installed. It turns out that pkg-config is returning the system installation:
$ pkg-config --libs libpng
-lpng12
even though I have updated basedir in matplotlib's setupext.py, and LD_LIBRARY_PATH to make them point to the the more recent version of libpng that I have locally installed.
Any ideas on how to have pkg-config return the right version?
It's a pkg-config issue; matplotlib's installation is (unfortunately, or perhaps not) relying too much on pkg-config's output.
Assuming you have build libpng the normal way, there should be a pkgconfig subdirectory in your /home/james/local/lib, which contains libpng.pc (and libpng16.pc). When setupext.py runs pkg-config, the latter should of course try and pick up the correct .pc file for libpng. For that, use the PKG_CONFIG_PATH variable and point it to the pkgconfig subdirectory:
$ export PKG_CONFIG_PATH=/home/james/local/lib/pkgconfig
Then, install matplotlib again, and see that it now finds the correct libpng version:
$ python setup.py build
basedirlist is: ['/usr/local', '/usr']
============================================================================
BUILDING MATPLOTLIB
matplotlib: 1.1.0
python: 2.7.4 (default, Apr 8 2013, 16:36:47) [GCC 4.4.5]
platform: linux2
REQUIRED DEPENDENCIES
numpy: 1.7.0
freetype2: 12.0.6
OPTIONAL BACKEND DEPENDENCIES
libpng: 1.6.1
Tkinter: Tkinter: 81008, Tk: 8.4, Tcl: 8.4
(For me, with a different PKG_CONFIG_PATH of course. Yes, I may want to upgrade some dependencies.)
Note that I didn't even alter basedirlist; it's just at its default.
In case pkg-config fails to now pick up some other package, just add more directories to PKG_CONFIG_PATH with colons in between. But I guess this should be enough.
Try
export LD_LIBRARY_PATH=/home/james/local/lib
and then execute Matplotlib... that would point matplotlib to your local version.