I have installed Python 3.5, tensorflow-gpu 1.5, CUDA v.9, cuDNN v.7 and NVIDIA AVX 5200. Although, I have installed tensorflow, bit cannot be used in Python and returns the following exception:
2019-03-31 22:27:45.285194: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
2019-03-31 22:27:46.132472: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: NVS 5200M major: 2 minor: 1 memoryClockRate(GHz): 1.344
pciBusID: 0000:01:00.0
totalMemory: 1.00GiB freeMemory: 823.00MiB
2019-03-31 22:27:46.132585: I C:\tf_jenkins\workspace\rel- win\M\windows-gpu\PY\35 \tensorflow\core\common_runtime\gpu\gpu_device.cc:1168] Ignoring visible gpu device (device: 0, name: NVS 5200M, pci bus id: 0000:01:00.0, compute capability: 2.1) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0.
And when checking it as:
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
I get:
name: "/device:CPU:0"
device_type: "CPU"
How do I solve this problem?
Related
My main program in Keras and Tensorflow hits a seg fault
python fine-tune.py --train_dir /root/data/train_dump/ --val_dir
/root/data/val_dump/ --nb_epoch 100 --batch_size 32
Using TensorFlow backend.
WARNING:root:Keras version 2.2.2 detected. Last version known to be fully compatible of Keras is 2.1.3 .
WARNING:root:TensorFlow version 1.10.0 detected. Last version known to be fully compatible is 1.5.0 .
Found 2976 images belonging to 4 classes.
Found 1100 images belonging to 4 classes.
2018-09-06 23:28:06.891710: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-09-06 23:28:08.030896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: TITAN V major: 7 minor: 0 memoryClockRate(GHz): 1.455
pciBusID: 0000:08:00.0
totalMemory: 11.78GiB freeMemory: 11.36GiB
2018-09-06 23:28:08.030988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-06 23:28:08.688955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-06 23:28:08.689049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-09-06 23:28:08.689071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-09-06 23:28:08.689614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10974 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:08:00.0, compute capability: 7.0)
Epoch 1/100
2018-09-06 23:29:31.853354: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2018-09-06 23:29:31.853522: E tensorflow/stream_executor/cuda/cuda_dnn.cc:360] Possibly insufficient driver version: 390.30.0
Segmentation fault (core dumped)
Yet tensorflow appears to be okay via this tf.session() test:
In [2]: import tensorflow as tf
...:
...: with tf.device('/gpu:0'):
...: a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
...: b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
...: c = tf.matmul(a, b)
...: with tf.Session() as sess:
...: print(sess.run(c))
...:
2018-09-06 23:37:31.603090: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-09-06 23:37:32.765804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: TITAN V major: 7 minor: 0 memoryClockRate(GHz): 1.455
pciBusID: 0000:08:00.0
totalMemory: 11.78GiB freeMemory: 11.36GiB
2018-09-06 23:37:32.765866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-06 23:37:33.358184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-06 23:37:33.358276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-09-06 23:37:33.358295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-09-06 23:37:33.358793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10974 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:08:00.0, compute capability: 7.0)
[[22. 28.]
[49. 64.]]
Thinking maybe the warnings had something to do with it, I did give that a try.
Installing collected packages: keras
Found existing installation: Keras 2.2.2
Uninstalling Keras-2.2.2:
Successfully uninstalled Keras-2.2.2
Successfully installed keras-2.1.3
(fastai) root#607b0f29-ad6b-482c-aead-aeae0a84fe2f:~# python fine-tune.py --train_dir /root/data/train_dump/ --val_dir /root/data/val_dump/ --nb_epoch 100 --batch_size 32
Using TensorFlow backend.
Found 2976 images belonging to 4 classes.
Found 1100 images belonging to 4 classes.
2018-09-07 00:30:10.722085: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-09-07 00:30:11.880320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: TITAN V major: 7 minor: 0 memoryClockRate(GHz): 1.455
pciBusID: 0000:08:00.0
totalMemory: 11.78GiB freeMemory: 11.36GiB
2018-09-07 00:30:11.880388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN V, pci bus id: 0000:08:00.0, compute capability: 7.0)
Epoch 1/100
2018-09-07 00:30:51.858613: E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2018-09-07 00:30:51.858965: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module 390.30 Wed Jan 31 22:08:49 PST 2018
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)
"""
2018-09-07 00:30:51.859055: E tensorflow/stream_executor/cuda/cuda_dnn.cc:393] possibly insufficient driver version: 390.30.0
2018-09-07 00:30:51.859099: F tensorflow/core/kernels/conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)
Aborted (core dumped)
I am using tensorflow-gpu. I want to use GTX1070, but tensorflow-gpu uses my CPU. I don't know　what to do.
I use CUDA 9.0 and CUDNN 7.1.4. My tensorflow-gpu version is 1.9.
After running this command on the official website
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
2018-07-30 10:53:43.369025: I
T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141]
Your CPU supports instructions that this TensorFlow binary was not
compiled to use: AVX2 2018-07-30 10:53:43.829922: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392]
Found device 0 with properties: name: GeForce GTX 1070 major: 6 minor:
1 memoryClockRate(GHz): 1.683 pciBusID: 0000:01:00.0 totalMemory:
8.00GiB freeMemory: 6.63GiB 2018-07-30 10:53:43.919043: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392]
Found device 1 with properties: name: GeForce GTX 1050 major: 6 minor:
1 memoryClockRate(GHz): 1.455 pciBusID: 0000:05:00.0 totalMemory:
2.00GiB freeMemory: 1.60GiB 2018-07-30 10:53:43.926001: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1456]
Ignoring visible gpu device (device: 1, name: GeForce GTX 1050, pci
bus id: 0000:05:00.0, compute capability: 6.1) with Cuda
multiprocessor count: 5. The minimum required count is 8. You can
adjust this requirement with the env var
TF_MIN_GPU_MULTIPROCESSOR_COUNT. 2018-07-30 10:53:43.934810: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1471]
Adding visible gpu devices: 0 2018-07-30 10:53:44.761551: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:952]
Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-30 10:53:44.765678: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:958]
0 1 2018-07-30 10:53:44.768363: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971]
0: N N 2018-07-30 10:53:44.771773: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971]
1: N N 2018-07-30 10:53:44.774913: I
T:\src\github\tensorflow\tensorflow\coenter code
herere\common_runtime\gpu\gpu_device.cc:1084] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:0 with 6395 MB
memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus
id: 0000enter code here:01:00.0, compute capability: 6.1)
As I can see from the log excerpt of your tensorflow engine - it uses GPU device 0
(/job:localhost/replica:0/task:0/device:GPU:0 with 6395 MB memory) -> physical
GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute
capability: 6.1)
But refuses to use your GeForce GTX 1050. This is possible due to the Environment variable TF_MIN_GPU_MULTIPROCESSOR_COUNT which is seems set to 8.
Try to set it to value of 5 as advised in your log earlier:
set TF_MIN_GPU_MULTIPROCESSOR_COUNT=5
If you want to be sure which device is used - initialize the session with
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
You can read more on Using GPUs tensorflow docs page
Im trying since a while to install tensorflow-gpu and had a lot of trouble with CUDA. First the Visual Studio integration in the CUDA setup always gave me an error, but if i leave out the Visual studio integration during CUDA installation, the installation is working.
Is the Studio integration mandatory for using tensorflow-gpu?
So then i installed all of the 3 patches for CUDA 9.0 and placed the Cudnn files in the folder.
Next, i went to my environtment variables and added this path (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0)
So when i open a command prompt and import tensorflow, in all tutorials i saw about this topic there where some lines about stuff being loaded successfully? I dont have these.
So im running this little code
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
And am getting this as answer
"C:\Program Files\Python35\python.exe" C:/Users/Felix/Documents/Uni/Semesterarbeit/doesitwork.py
2018-06-21 13:41:41.187933: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-21 13:41:41.748188: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:21:00.0
totalMemory: 8.00GiB freeMemory: 6.63GiB
2018-06-21 13:41:41.748527: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-21 13:43:44.853239: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-21 13:43:44.853436: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2018-06-21 13:43:44.853564: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2018-06-21 13:43:44.853860: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1)
Device mapping:
2018-06-21 13:43:45.164653: I T:\src\github\tensorflow\tensorflow\core\common_runtime\direct_session.cc:284] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1
Process finished with exit code 0
Is this what its supposed to look like?
Looking forward to an answer
Cheers,
Felix
Take a look at the last line in your log.
'job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1'
I'm trying to use tensorflow GPU on my ubuntu 16.04 machine. I've successfuly installed CUDA toolkit (8.0.61) and cuDNN (6.0.21). The problem is, I can't use tensorflow gpu even after this installtion process.
While importing tensorflow is not showing any import lines for tensorflow GPU.its not showing any lines while importing tensorflow, while other my other ubuntu machine it shows some lines while importing tensorflow.
Nvidia driver version and usage
There are no lines when importing TensorFlow. Only starting a sessions gives you some output:
>>> import tensorflow as tf
>>> tf.Session()
2017-09-17 20:06:20.174697: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
2017-09-17 20:06:20.343531: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-09-17 20:06:20.344062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Found device 0 with properties:
name: GeForce GTX 960 major: 5 minor: 2 memoryClockRate(GHz): 1.329
pciBusID: 0000:01:00.0
totalMemory: 3.94GiB freeMemory: 3.61GiB
2017-09-17 20:06:20.344084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:01:00.0, compute capability: 5.2)
<tensorflow.python.client.session.Session object at 0x7f387d89f210>
After consulting my crystal ball, I suggest you to further check the environment variable TF_CPP_MIN_LOG_LEVEL.
I just installed two Nvidia K2200 GPU's, CUDA software, and CuDNN software on my Windows 10 computer. I went to check if everything is working well by following this Stack Overflow answer but I got a big message with a bunch of warnings. I am not sure how to interpret it. Does the message mean that something and my TensorFlow/Keras code won't work?
Here is the messsage:
2017-08-09 09:03:52.984209: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.984358: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.985302: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.986429: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.987150: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.990185: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.990775: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:52.991261: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-08-09 09:03:53.310243: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:940] Found device 0 with properties:
name: Quadro K2200
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:04:00.0
Total memory: 4.00GiB
Free memory: 3.35GiB
2017-08-09 09:03:53.405531: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\stream_executor\cuda\cuda_driver.cc:523] A non-primary context 000001B8981C7F00 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-08-09 09:03:53.406260: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:940] Found device 1 with properties:
name: Quadro K2200
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 3.35GiB
2017-08-09 09:03:53.409719: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:832] Peer access not supported between device ordinals 0 and 1
2017-08-09 09:03:53.411660: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:832] Peer access not supported between device ordinals 1 and 0
2017-08-09 09:03:53.412396: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:961] DMA: 0 1
2017-08-09 09:03:53.413047: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0: Y N
2017-08-09 09:03:53.413445: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 1: N Y
2017-08-09 09:03:53.414996: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro K2200, pci bus id: 0000:04:00.0)
2017-08-09 09:03:53.415559: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Quadro K2200, pci bus id: 0000:01:00.0)
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 15789200439240454107
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 3280486400
locality {
bus_id: 1
}
incarnation: 685299155373543396
physical_device_desc: "device: 0, name: Quadro K2200, pci bus id: 0000:04:00.0"
, name: "/gpu:1"
device_type: "GPU"
memory_limit: 3280486400
locality {
bus_id: 1
}
incarnation: 16323028758437337139
physical_device_desc: "device: 1, name: Quadro K2200, pci bus id: 0000:01:00.0"
]
You can try adding a load (e.g. train some model) and check "nvidia-smi" from the terminal while it's working - it should show your GPU utilization.