I am trying to build tensorflow from source on a remote server (with no superuser privileges) because I got this error when I simply installed with pip:
Loaded runtime CuDNN library: 7.1.2 but source was compiled with: 7.4.2. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
I completed all the steps listed here successfully, but I still get the same error as above, despite setting CudNN version as 7.1.2 before building.
Is there any way I can see the configurations to verify that they have been set properly?
A file is generated after running ./configure with the name .tf_configure.bazelrc you can inspect that file.
Related
While training with Tensorflow 2 Object Detection API in Google Collab, I get the following errors:
2022-03-07 15:27:51.475298: E tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-03-07 15:27:51.477142: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNKNOWN: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
I have tried updating my CuDNN library in Google collab, using this StackOverflow answer:
Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. when using google colab.
However, this does not solve the issue.
I am using Tensorflow 2.7.0.
The following is a workaround for the problem, as posted on the official Github
This may not work always and may not be safe, but as a workaround for the time being, you may try running !apt install --allow-change-held-packages libcudnn8=8.1.0.77-1+cuda11.2
I get this error when I run the model.fit_generator code to train images using the CNN model. I don't understand the error, and what should I do? Can anyone help me?
this is the full error description
`Loaded runtime CuDNN library: 8.0.5, but the source was compiled with: 8.1.0. CuDNN library needs to have a matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, ensure the library loaded at runtime is compatible with the version specified during compile configuration.
I had the same error "tensorflow/stream_executor/cuda/cuda_dnn.cc:362] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0."
I solved it by downgrading the TensorFlow version, here it says that you use a new version of TensorFlow that is not compatible with the google colab CuDNN version. I used TensorFlow 2.4.0 plus all the dependence required on version 2.4.0.
Here it says which version of TensorFlow to use for cudnn compatibility, https://www.tensorflow.org/install/source
You should always have version of libraries installed that is matching the version dependency you want to use is compiled with.
You can download the version you need from nvidia website or use conda for package management. It will handle all dependencies for you.
You can miniconda and type conda install -c anaconda tensorflow-gpu to get it sorted for you. If you need a specific version of python, you can create environment with it.
My solution:
After confirming that my cuda and cudnn versions are compatible with tensorflow, I first thought that the system did not synchronize after the installation was completed. After several restarts, it was found that it was not and could not be the problem, so I started to check all the cuda in the system. For the software that depends on cudnn, matlab was uninstalled during the period but it was useless. Later, I thought that pytorch is also related to cuda and cudnn. I checked the version of pytorch and found that I was using torch 1.8, and the cuda it was adapted to was 11.1 , The corresponding cudnn is 8.0.5, now the case is solved. Finally upgraded pytorch and solved it.
I have faced the same issue. It seems like if TensorFlow versions requires specific cuDNN version.
Check the link for required versions.
https://www.tensorflow.org/install/source#gpu
Thanks for This answer.
My solution:
After confirming that my cuda and cudnn versions are compatible with
tensorflow, I first thought that the system ...
It helps me a lot,but I use different way to solve this problem.
I found that pytorch 1.8 is compatible with cudnn 8.1.0. So, instead of upgrade pytorch version, I overwrite the cudnn 8.0.5 dll library with cudnn 8.1.0 in directory D:\Program Files\Python37\Lib\site-packages\torch\lib. You can find this location with Everything, which is always helpful.
I am working on a Python script (I use Python 3.7.3) that uses tensorflow-gpu (1.14.0) and used PyInstaller 3.5 to convert this script to an executable. I am using CUDA 10.0 and cuDNN 7.6.1 and my graphics card is a NVIDIA GeForce GTX 960M. I recently deinstalled CUDA to test if the executable of the Python script still runs and surprisingly it still runs via GPU, which does not work when I now run the Python script directly.
My question is, can this executable be run on systems without the CUDA toolkit but with a CUDA-capable graphics card?
According to this documentation PyInstaller will make and store a private copy of all of the dependent external libraries which Python code relies on when building a single file executable.
Therefore it is safe to assume that your executable runs irrespective of the installation status of the CUDA toolkit because it has a full private copy of the necessary CUDA libraries internally which it uses when the executable is run.
According to the GitHub issues in the official repository (here and here for example) CUDA libraries are usually dynamically loaded at run-time and not at link-time, so they are typically not included in the final exe file (or folder) with the result that the exe file won't work on a machine without CUDA installed. The solution (please refer to the linked issues too) is to put the DLLs necessary to run the exe in its dist folder (if generated without the --onefile option) or install the CUDA runtime on the target machine.
The behaviour that you're experimenting maybe it's due to the specific version of TF, that loads the libraries in a different fashion with respect to what described above, but it's not the expected behaviour nowadays.
I am trying to use tensorflow gpu version to train and test my deep learning model. But here comes the problem. When I train my model in one python file things go on well. Tensorflow-gpu can be used properly. Then I save my model as a pretrained on as grapg.pb format and try to reuse it in another python file.
Then I got the following error messages.
E tensorflow/stream_executor/cuda/cuda_dnn.cc:363] Loaded runtime CuDNN
library: 7.1.4 but source was compiled with: 7.2.1. CuDNN library major
and minor version needs to match or have higher minor version in case of
CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN
library. If building from sources, make sure the library loaded at runtime
is compatible with the version specified during compile configuration.
I checked my cudnn version, in fact it is version 7.4.2. I also checked my environment path settings /cuda/v9.0/bin, cuda/v9.0/lib/x64, /cuda/v9.0/include are in there.
So why this happens? And how can I solve this?
--
cuda:v9.0
cudnn:7.4.2 (I think, I copy those cudnn files manually)
windows 10
python: 3.5
If you have multiple CuDNN installed thorough various ways like anaconda module and windows installation, you need to remove the older version in order for your code to detect the latest version and reinstall tensorflow-gpu.
You can follow this guide for installation based on OS.
I installed CUDA 9.0 because without it, Tensorflow gives the error:
ImportError: Could not find 'cudart64_90.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable.
I have the path variable set to the bin directory of CUDA 9.0 installation, where the required DLL file is present. I tried setting it to its parent directory too. But it still gives me the same error.
I found the solution. And it was the good old advice - "Have you tried turning it off and on again?"
I restarted the computer, Tensorflow found cudart64_90.dll, but now it could not find cudnn64_7.dll. I'm providing the steps ahead to get rid of the issues I encountered.
If you've installed Tensorflow GPU version, you're likely to run into the problem mentioned in the post. Especially if you've not installed NVDIA development toolkits before. Follow these steps:
1. Install CUDA
Get it from here. Install only the version mentioned in Tensorflow's ImportError.
ImportError: Could not find 'cudart64_90.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Download and install CUDA 9.0 from this URL: https://developer.nvidia.com/cuda-toolkit
It explicitly tells you the version number. Initially, I installed CUDA 9.1 instead of 9.0, it didn't work. The installation on Windows is straight forward. Run the .exe, uncheck NVIDIA Geforce and other packages if you already have them installed.
2. Include CUDA path in PATH variable
Point it to the bin directory of your tensorflow installation.
Check here if you don't know how to set the PATH variable. Now try importing Tensorflow, if it still doesn't work, try rebooting the system.
Now you will likely run into the error:
ImportError: Could not find 'cudnn64_7.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Note that installing cuDNN is a separate step from installing CUDA, and this DLL is often found in a different directory from the CUDA DLLs. You may install the necessary DLL by downloading cuDNN 7 from this URL: https://developer.nvidia.com/cudnn
3. Install cuDNN
Once again, only install the version mentioned in the error.To get the installer, you need to have an NVIDIA developer account. If you don't have it, sign up and it will direct you to the link to download cuDNN. Select the version compatible with your CUDA version (it's in the package name). Download the zip archive. Extract it somewhere on your disk.
4. Include cuDNN path in PATH variable
Similar to step two. This time, point it to the bin directory in your extracted archive of cuDNN. Now import Tensorflow. Restart system if required.
It should now work.
When I reached to step 3, I copied and pasted the files inside the archive
as in here
Copy <installpath>\cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin.
Copy <installpath>\cuda\ include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include.
Copy <installpath>\cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\lib\x64.
I had the same problem for a couple hours as well. Just restarted my computer and that fixed the issue you were having, so give that a try.
Always check the cuda version, in this case you have to install cuda version 9.0 this will create the cudart64_90.dll file in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin folder.
This will surely work.