import cupy error: .../nvrtc.cpython-39-x86_64-linux-gnu.so: undefined symbol: nvrtcGetNVVMSize, version libnvrtc.so.11.2 - cupy

I'm using GTX2080 on Ubuntu 22.04
here's the output of nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0
so I installed cupy using:
pip install cupy-cuda117
However, when I run import cupy it returns an import error:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
File ~/anaconda3/lib/python3.9/site-packages/cupy/__init__.py:18, in <module>
17 try:
---> 18 from cupy import _core # NOQA
19 except ImportError as e:
File ~/anaconda3/lib/python3.9/site-packages/cupy/_core/__init__.py:1, in <module>
----> 1 from cupy._core import core # NOQA
2 from cupy._core import fusion # NOQA
File cupy/_core/core.pyx:1, in init cupy._core.core()
File ~/anaconda3/lib/python3.9/site-packages/cupy/cuda/__init__.py:8, in <module>
7 from cupy._environment import get_hipcc_path # NOQA
----> 8 from cupy.cuda import compiler # NOQA
9 from cupy.cuda import device # NOQA
File ~/anaconda3/lib/python3.9/site-packages/cupy/cuda/compiler.py:18, in <module>
17 from cupy_backends.cuda.api import runtime
---> 18 from cupy_backends.cuda.libs import nvrtc
19 from cupy import _util
ImportError: /home/sam/anaconda3/lib/python3.9/site-packages/cupy_backends/cuda/libs/nvrtc.cpython-39-x86_64-linux-gnu.so: undefined symbol: nvrtcGetNVVMSize, version libnvrtc.so.11.2
The above exception was the direct cause of the following exception:
ImportError Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import cupy
File ~/anaconda3/lib/python3.9/site-packages/cupy/__init__.py:20, in <module>
18 from cupy import _core # NOQA
19 except ImportError as e:
---> 20 raise ImportError(f'''
21 ================================================================
22 {_environment._diagnose_import_error()}
23
24 Original error:
25 {type(e).__name__}: {e}
26 ================================================================
27 ''') from e
30 from cupy import cuda # NOQA
31 # Do not make `cupy.cupyx` available because it is confusing.
ImportError:
================================================================
Failed to import CuPy.
If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.
On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm.
On Windows, try setting CUDA_PATH environment variable.
Check the Installation Guide for details:
https://docs.cupy.dev/en/latest/install.html
Original error:
ImportError: /home/sam/anaconda3/lib/python3.9/site-packages/cupy_backends/cuda/libs/nvrtc.cpython-39-x86_64-linux-gnu.so: undefined symbol: nvrtcGetNVVMSize, version libnvrtc.so.11.2
================================================================
it says "On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm." I've edit the .bashrc file to include the path of cuda:
export PATH="/usr/local/cuda-11.7/bin:$PATH"
export LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64\
${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
but still no luck, same import error. I tried reboot but same result... Thanks for any help!

Related

Error when "import tensorflow as tf" in colab

When I "import tensorflow as tf" in google colab I am getting the following error. Any idea how to fix it ?
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-37-07e0f991eb97> in <module>()
----> 1 import tensorflow as tf
2
3 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/pywrap_tf_session.py in <module>()
17 # pylint: disable=invalid-import-order,g-bad-import-order, wildcard-import, unused-import
18 from tensorflow.python import pywrap_tensorflow
---> 19 from tensorflow.python.client._pywrap_tf_session import *
20 from tensorflow.python.client._pywrap_tf_session import _TF_SetTarget
21 from tensorflow.python.client._pywrap_tf_session import _TF_SetConfig
ImportError: SystemError: <built-in method __contains__ of dict object at 0x7f79368db0a0> returned a result with an error set
---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
This issue has been fixed now in 2.8.0 stable version. If it still persists ,please install Numpy 1.21.5 manually and Try again.
I've post a similar error on tensorflow github issue.
Tensorflow 2.8.0 | Error Message: ImportError: SystemError.
My solution (since I'm using conda environment on Windows 10), is to downgrade the Tensorflow version to 2.7.0. Hope help helps.

CuPy on AMD GPU causing an ImportError

I'm trying to use/test the (experimental) AMD GPU support of CuPy (8.1.0). In short: I'm having an ImportError.
I followed the instructions for installing ROCm on a Ubuntu focal machine (with a gfx906/Radeon VII (Vega 20) card).
From the doc I set the mentioned variables (with adjusted HCC_AMDGPU_TARGET), the ROCM_HOME and export PATH=$PATH:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin.
pip3 install -v --no-cache-dir cupy succeeded without any complaints. Just a few compiler notes and warnings because of the -v option
Now when I try to import cupy it's throwing an ImportError.
In [1]: import cupy
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/cupy/__init__.py in <module>
19 message='can\'t resolve package from __spec__')
---> 20 from cupy import core # NOQA
21 except ImportError as e:
/usr/local/lib/python3.8/dist-packages/cupy/core/__init__.py in <module>
----> 1 from cupy.core import core # NOQA
2 from cupy.core import internal # NOQA
3
cupy/core/core.pyx in init cupy.core.core()
cupy/core/_routines_manipulation.pyx in init cupy.core._routines_manipulation()
cupy/core/_routines_indexing.pyx in init cupy.core._routines_indexing()
cupy/core/_routines_math.pyx in init cupy.core._routines_math()
cupy/core/_reduction.pyx in init cupy.core._reduction()
cupy/core/_cub_reduction.pyx in init cupy.core._cub_reduction()
ModuleNotFoundError: No module named 'cupy.cuda.cub'
The above exception was the direct cause of the following exception:
ImportError Traceback (most recent call last)
<ipython-input-1-329ec5cf1bc8> in <module>
----> 1 import cupy
/usr/local/lib/python3.8/dist-packages/cupy/__init__.py in <module>
39 original error: {}'''.format(_exc_info[1])) # NOQA
40
---> 41 raise ImportError(_msg) from e
42
43
ImportError: CuPy is not correctly installed.
If you are using wheel distribution (cupy-cudaXX), make sure that the version of CuPy you installed matches with the version of CUDA on your host.
Also, confirm that only one CuPy package is installed:
$ pip freeze
If you are building CuPy from source, please check your environment, uninstall CuPy and reinstall it with:
$ pip install cupy --no-cache-dir -vvvv
Check the Installation Guide for details:
https://docs.cupy.dev/en/latest/install.html
original error: No module named 'cupy.cuda.cub'
What should I do next? What should I look for in the compilation process? Thanks in advance
EDIT On request here the complete stdout and stderr of the installation process and a few io's of the console: https://seafile.cloud.uni-hannover.de/d/68862cd1057e47d180aa/
To summarize the discussion in the comments so far:
For CuPy v8.x (the latest stable releases): Up to v8.1 the offending cupy.cuda.cub module is not built in ROCm/HIP environments, which will hopefully be fixed in v8.2 (see ticket).
For CuPy v9.x (the master branch): It should just work as long as rocPRIM and hipCUB are correctly installed. It's likely that the OP's local environment is not correctly set up, leading to cupy.cuda.cub module not built and thus ImportError.

Not able to import tensorflow_datasets module in jupyter notebook

I am trying tensorflow course from Udacity which uses google colab to write/run the code. But I want to run the code on my local machine and hence have created a new environment to run the code , but am unable to import tensorflow_dataset into the tensorflow environment .
I have tried searching for the module from the anaconda navigator to install it in tensorflow environment but anaconda navigator does not provide the tensorflow datasets module.
From cmd (command prompt) I tried to execute the below commands
1.activate tensorflow
2. pip install tensorflow_datasets
It says the module is installed but when I tried to import it from jupyter notebook I get error that there is no module with name tensorflow_datasets
1.activate tensorflow
2. pip install tensorflow_datasets
import tensorflow_datasets as tfds
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-5-46a8a2031c9c> in <module>()
----> 1 import tensorflow_datasets as tfds
ImportError: No module named 'tensorflow_datasets'
And if I am trying to import from tensorflow env
1.jupyter notebook
2.import tensorflow_datasets as tdfs
ImportError Traceback (most recent call last)
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py in <module>()
57
---> 58 from tensorflow.python.pywrap_tensorflow_internal import *
59 from tensorflow.python.pywrap_tensorflow_internal import __version__
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py in <module>()
27 return _mod
---> 28 _pywrap_tensorflow_internal = swig_import_helper()
29 del swig_import_helper
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py in swig_import_helper()
23 try:
---> 24 _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
25 finally:
~\Miniconda2\envs\tensorflow\lib\imp.py in load_module(name, file, filename, details)
242 else:
--> 243 return load_dynamic(name, filename, file)
244 elif type_ == PKG_DIRECTORY:
~\Miniconda2\envs\tensorflow\lib\imp.py in load_dynamic(name, path, file)
342 name=name, loader=loader, origin=path)
--> 343 return _load(spec)
344
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
<ipython-input-1-46a8a2031c9c> in <module>()
----> 1 import tensorflow_datasets as tfds
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow_datasets\__init__.py in <module>()
44 # needs to happen before anything else, since the imports below will try to
45 # import tensorflow, too.
---> 46 from tensorflow_datasets.core import tf_compat
47 tf_compat.ensure_tf_install()
48
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow_datasets\core\__init__.py in <module>()
16 """API to define datasets."""
17
---> 18 from tensorflow_datasets.core.dataset_builder import BeamBasedBuilder
19 from tensorflow_datasets.core.dataset_builder import BuilderConfig
20 from tensorflow_datasets.core.dataset_builder import DatasetBuilder
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow_datasets\core\dataset_builder.py in <module>()
27 from absl import logging
28 import six
---> 29 import tensorflow as tf
30
31 from tensorflow_datasets.core import api_utils
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\__init__.py in <module>()
26
27 # pylint: disable=g-bad-import-order
---> 28 from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
29 from tensorflow.python.tools import module_util as _module_util
30
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\__init__.py in <module>()
47 import numpy as np
48
---> 49 from tensorflow.python import pywrap_tensorflow
50
51 # Protocol buffers
~\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py in <module>()
72 for some common reasons and solutions. Include the entire stack trace
73 above this error message when asking for help.""" % traceback.format_exc()
---> 74 raise ImportError(msg)
75
76 # pylint: enable=wildcard-import,g-import-not-at-top,unused-import,line-too-long
ImportError: Traceback (most recent call last):
File "C:\Users\HOME\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\HOME\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\HOME\Miniconda2\envs\tensorflow\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\HOME\Miniconda2\envs\tensorflow\lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\HOME\Miniconda2\envs\tensorflow\lib\imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.```
tensorflow_datasets is a separate module. You can install it using pip :
pip install tensorflow_datasets
Then you can import it as:
import tensorflow_datasets as tfds
to use tensorflow-datasets in anaconda you have to use the below command
conda install -c anaconda tensorflow-datasets
i tried using pip install and all but it was not working fine.but the above command worked like a charm.
using pip install will only install tensorflow-datasets in system python installation, which cant be used inside anaconda environment
To use tensorflow datasets, open the terminal/command prompt and enter :
pip install tensorflow_datasets
And if you are using Anaconda, you need to open up your environment and then manually import it as it doesn't gets installed.
If you are having anaconda prompt, then as the answers already suggest installing tensorflow-datasets should work without issues. You can find the related Pypi package here. This installation should work in command prompt (with pip /python installed) as well.
pip install tensorflow-datasets
If you are directly installing via conda, it is best to set up conda-forge as all the packages are shared in a single channel.Importantly the corresponding builds for AMD,Linux and MacOS are also up-to-date. For installing , here are the steps:
Making sure conda version is >=4.9:
conda --version
conda update conda
Adding conda-forge as the highest priority channel:
conda config --add channels conda-forge
Activate Strict priority (optional for Conda>=5.0):
conda config --set channel_priority strict
Install the tensorflow-datasets package from conda-forge:
conda install tensorflow-datasets==4.3.0
4.3.0 is the latest version as per Pypi.
If you are using Anaconda, go to Anaconda Navigator -> Environments -> Choose your environment. After that simply find which package would you like to install and click it. After you accept it, it uploads automatically. It is really easy. (It took half an hour to find a correct answer for me, but after finding in Anaconda it is way easier than using Terminal) Do not forget to restart Anaconda after installing new packages.

Related to installing of tensorflow in python 3.7

As I am using the command prompt to run python when installing tensorflow it is saying requirment satisfied but when importing giving an error
ImportError
Traceback (most recent call last)
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py in swig_import_helper()
17 try:
---> 18 fp, pathname, description = imp.find_module('_pywrap_tensorflow_internal', [dirname(__file__)])
19 except ImportError:
c:\users\chirag\appdata\local\programs\python\python37\lib\imp.py in find_module(name, path)
295 else:
--> 296 raise ImportError(_ERR_MSG.format(name), name=name)
297
ImportError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
ModuleNotFoundError Traceback (most recent call last)
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow.py in <module>
57
---> 58 from tensorflow.python.pywrap_tensorflow_internal import *
59 from tensorflow.python.pywrap_tensorflow_internal import __version__
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py in <module>
27 return _mod
---> 28 _pywrap_tensorflow_internal = swig_import_helper()
29 del swig_import_helper
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py in swig_import_helper()
19 except ImportError:
---> 20 import _pywrap_tensorflow_internal
21 return _pywrap_tensorflow_internal
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
<ipython-input-1-d6579f534729> in <module>
----> 1 import tensorflow
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\__init__.py in <module>
22
23 # pylint: disable=g-bad-import-order
---> 24 from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
25
26 try:
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\__init__.py in <module>
47 import numpy as np
48
---> 49 from tensorflow.python import pywrap_tensorflow
50
51 from tensorflow.python.tools import component_api_helper
c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow.py in <module>
72 for some common reasons and solutions. Include the entire stack trace
73 above this error message when asking for help.""" % traceback.format_exc()
---> 74 raise ImportError(msg)
75
76 # pylint: enable=wildcard-import,g-import-not-at-top,unused-import,line-too-long
ImportError: Traceback (most recent call last):
File "c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
fp, pathname, description = imp.find_module('_pywrap_tensorflow_internal', [dirname(__file__)])
File "c:\users\chirag\appdata\local\programs\python\python37\lib\imp.py", line 296, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "c:\users\chirag\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
import _pywrap_tensorflow_internal
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors for some common reasons and solutions.
Include the entire stack trace above this error message when asking for help.
The TensorFlow build from source on Windows guide mentions it only supports Python 3.5 and Python 3.6.
Last time I installed tensorflow on Windows, if I recall correctly I used Python 3.6.4 64-bit. (I had some errors with Python 3.6.5 but that was last summer so there's a chance slightly newer versions of 3.6 might work).
Docs mention:
Requires Python 3.4, 3.5, or 3.6
Because I wanted to install tensorflow-gpu I also had to be mindful of the CUDA Toolkit and CuDNN versions too (some combinations didn't work).
Unfortunately I don't remember the CUDA/CuDNN versions 100% but documentation mentions CUDA 9.0 (e.g. Download cuDNN v7.0.5 (Dec 5, 2017), for CUDA 9.0
Looking at the nVidia CUDA Archive that's from Sept. 2017 so I would go with a CuDNN version compatible with CUDA 9.0 from around that time from their CuDNN Archive(e.g. 7.0.5 for Windows 10).
Not 100% guaranteed if this will work, but something like this:
Uninstall Python 3.7 (unless you can find a simple way to install multiple version of Python and switch as needed, but doing quick searches reveal it's a bit of a headache on Windows)
Install Python 3.6.4 64 bit
Install pip (if it's not already installed) (download get-pip.py and run it)
Install virtualenv(pip install virtualenv, virtualenv tf-gpu,.\tf-gpu\Scripts\activate) (the point of the virtual environment is to keep a local tensorflow setup as opposed to a global module which might cause headaches with other projects using different TensorFlow dependency versions (numpy,protobuf,six,etc.)
Install CUDA 9.0
Install CuDNN (7.0.5 or wee bit higher, but for CUDA 9.0 for sure)
Reboot, double check Windows Environment Variables for CUDA/CUDNN paths and verify the installs
Finally install TensorFlow with GPU support in the virtual environment:pip install tensorflow-gpu (will take a bit of time: might as well hydrate, etc.)
Hopefully the installation completed successfully in which case verify it: quick test: python -c 'import tensorflow;print("TensorFlow version:",tensorflow.__version__)'...if this doesn't throw errors and prints the version open run the HelloWorld example (which should also print some CUDA info) once the session runs.
If you don't want tensorflow-gpu or don't have a tensorflow compatible GPU you can skip the CUDA/CuDNN part. I still recommend making a virtual environment (e.g. tf-cpu) then use pip install tensorflow instead of pip install tensorflow-gpu within the activated virtual environment.

ImportError: No module named 'twisted.persisted'

I haved install scrapy on Python 3.5.2, but when I exeute scrapy -v on command-line, it occurs:
>> scrapy -v
Traceback (most recent call last):
File "/usr/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/usr/local/python3.5.2/lib/python3.5/site-packages/scrapy/__init__.py", line 27, in <module>
from . import _monkeypatches
File "/usr/local/python3.5.2/lib/python3.5/site-packages/scrapy/_monkeypatches.py", line 20, in <module>
import twisted.persisted.styles # NOQA
ImportError: No module named 'twisted.persisted'
So I enter ipython3 to ensure whether it's installed porperly:
In [2]: import twisted
In [3]: twisted.version
Out[3]: Version('twisted', 15, 2, 1)
But when I import scrapy, the same as command-line:
In [4]: import scrapy
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-4-51c73a18167b> in <module>()
----> 1 import scrapy
/usr/local/python3.5.2/lib/python3.5/site-packages/scrapy/__init__.py in <module>()
25
26 # Apply monkey patches to fix issues in external libraries
---> 27 from . import _monkeypatches
28 del _monkeypatches
29
/usr/local/python3.5.2/lib/python3.5/site-packages/scrapy/_monkeypatches.py in <module>()
18 # Undo what Twisted's perspective broker adds to pickle register
19 # to prevent bugs like Twisted#7989 while serializing requests
---> 20 import twisted.persisted.styles # NOQA
21 # Remove only entries with twisted serializers for non-twisted types.
22 for k, v in frozenset(copyreg.dispatch_table.items()):
ImportError: No module named 'twisted.persisted'
I'm so confused, I just want to run a spaider, So help me !
Your Twisted version is outdated, if you look at requirements-py3.txt file:
Twisted >= 15.5.0
lxml>=3.2.4
pyOpenSSL>=0.13.1
cssselect>=0.9
queuelib>=1.1.1
w3lib>=1.17.0
service_identity
Where your's is, 15.2.
Currently the latest version 17.1.0 and there are no downsides of running it; Try:
pip3 install twisted --upgrade