This command is giving GPU with 4614 MB Memory.
But the RTX2060 has 6GB Memory. Why is it showing only just over 4GB?
>>> tf.test.is_built_with_cuda()
True
>>> tf.test.is_gpu_available(cuda_only=False,min_cuda_compute_capability=None)
2019-10-29 17:02:40.062465: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-10-29 17:02:40.072455: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2019-10-29 17:02:40.105489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
2019-10-29 17:02:40.111138: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-10-29 17:02:40.117217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-10-29 17:02:44.865862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-29 17:02:44.870341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-10-29 17:02:44.872351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-10-29 17:02:44.876727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 4614 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
True
Tensorflow reports the memory it's able to use on the device. So if you have anything else using some GPU memory, the memory reported by TensorFlow will be lower than the total available memory.
Also, some OS only allows a certain fraction of the memory to be used by applications to ensure it will still be able to use the device (cf this issue on TensorFlow's GitHub: https://github.com/tensorflow/tensorflow/issues/22623)
Related
tensorflow sticks to Creating CudaSolver handles for stream without moving forward for multiple hours when training the deep neural network.
tensorflow version:1.12
cuda version: 9.0
cudnn version: 7.3
2019-06-13 09:45:17.087403: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-06-13 09:45:18.568961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:5a:00.0
totalMemory: 15.89GiB freeMemory: 15.60GiB
2019-06-13 09:45:18.569067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-06-13 09:45:19.636474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-06-13 09:45:19.636549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-06-13 09:45:19.636578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-06-13 09:45:19.636958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15117 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:5a:00.0, compute capability: 6.0)
2019-06-13 09:45:21.246683: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x56140303bcf0
It is that complex deep neural network with a large of dataset would take a long period of time to train. After running a few hours, the model starts to update.
my laptop has two different graphic cards; one of them is Intel other one is Nvdia GeForce GTX 1060 with Max-Q Design.
I want to train my model using Nvidia GPU however while i start the code i see that Intel GPU is using instead of Nvidia. (By checking resource monitor)
How can i push my code to evaluate operations on Nvidia?
Thanks in advance.
Code output is on below;
2019-05-16 22:00:08.022034: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-05-16 22:00:09.109537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1060 with Max-Q Design major: 6 minor: 1 memoryClockRate(GHz): 1.3415
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2019-05-16 22:00:09.115599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-16 22:00:09.732522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-16 22:00:09.735119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-05-16 22:00:09.736802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-05-16 22:00:09.738843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4714 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-05-16 22:00:10.947582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-16 22:00:10.950781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-16 22:00:10.953529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-05-16 22:00:10.955159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-05-16 22:00:10.957471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4714 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-05-16 22:00:11.016401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-16 22:00:11.020139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-16 22:00:11.023906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-05-16 22:00:11.026780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-05-16 22:00:11.030436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4714 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\user\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-05-16 22:00:12.013549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-05-16 22:00:12.016873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-16 22:00:12.021027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-05-16 22:00:12.023952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-05-16 22:00:12.026487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4714 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
Processing...
row 0 of 1696
2019-05-16 22:00:12.326057: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library cublas64_90.dll locally
I am using tensorflow-gpu. I want to use GTX1070, but tensorflow-gpu uses my CPU. I don't know　what to do.
I use CUDA 9.0 and CUDNN 7.1.4. My tensorflow-gpu version is 1.9.
After running this command on the official website
>>> import tensorflow as tf
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
2018-07-30 10:53:43.369025: I
T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141]
Your CPU supports instructions that this TensorFlow binary was not
compiled to use: AVX2 2018-07-30 10:53:43.829922: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392]
Found device 0 with properties: name: GeForce GTX 1070 major: 6 minor:
1 memoryClockRate(GHz): 1.683 pciBusID: 0000:01:00.0 totalMemory:
8.00GiB freeMemory: 6.63GiB 2018-07-30 10:53:43.919043: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392]
Found device 1 with properties: name: GeForce GTX 1050 major: 6 minor:
1 memoryClockRate(GHz): 1.455 pciBusID: 0000:05:00.0 totalMemory:
2.00GiB freeMemory: 1.60GiB 2018-07-30 10:53:43.926001: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1456]
Ignoring visible gpu device (device: 1, name: GeForce GTX 1050, pci
bus id: 0000:05:00.0, compute capability: 6.1) with Cuda
multiprocessor count: 5. The minimum required count is 8. You can
adjust this requirement with the env var
TF_MIN_GPU_MULTIPROCESSOR_COUNT. 2018-07-30 10:53:43.934810: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1471]
Adding visible gpu devices: 0 2018-07-30 10:53:44.761551: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:952]
Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-30 10:53:44.765678: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:958]
0 1 2018-07-30 10:53:44.768363: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971]
0: N N 2018-07-30 10:53:44.771773: I
T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971]
1: N N 2018-07-30 10:53:44.774913: I
T:\src\github\tensorflow\tensorflow\coenter code
herere\common_runtime\gpu\gpu_device.cc:1084] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:0 with 6395 MB
memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus
id: 0000enter code here:01:00.0, compute capability: 6.1)
As I can see from the log excerpt of your tensorflow engine - it uses GPU device 0
(/job:localhost/replica:0/task:0/device:GPU:0 with 6395 MB memory) -> physical
GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute
capability: 6.1)
But refuses to use your GeForce GTX 1050. This is possible due to the Environment variable TF_MIN_GPU_MULTIPROCESSOR_COUNT which is seems set to 8.
Try to set it to value of 5 as advised in your log earlier:
set TF_MIN_GPU_MULTIPROCESSOR_COUNT=5
If you want to be sure which device is used - initialize the session with
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
You can read more on Using GPUs tensorflow docs page
Im trying since a while to install tensorflow-gpu and had a lot of trouble with CUDA. First the Visual Studio integration in the CUDA setup always gave me an error, but if i leave out the Visual studio integration during CUDA installation, the installation is working.
Is the Studio integration mandatory for using tensorflow-gpu?
So then i installed all of the 3 patches for CUDA 9.0 and placed the Cudnn files in the folder.
Next, i went to my environtment variables and added this path (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0)
So when i open a command prompt and import tensorflow, in all tutorials i saw about this topic there where some lines about stuff being loaded successfully? I dont have these.
So im running this little code
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
And am getting this as answer
"C:\Program Files\Python35\python.exe" C:/Users/Felix/Documents/Uni/Semesterarbeit/doesitwork.py
2018-06-21 13:41:41.187933: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-21 13:41:41.748188: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:21:00.0
totalMemory: 8.00GiB freeMemory: 6.63GiB
2018-06-21 13:41:41.748527: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-21 13:43:44.853239: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-21 13:43:44.853436: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2018-06-21 13:43:44.853564: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2018-06-21 13:43:44.853860: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1)
Device mapping:
2018-06-21 13:43:45.164653: I T:\src\github\tensorflow\tensorflow\core\common_runtime\direct_session.cc:284] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1
Process finished with exit code 0
Is this what its supposed to look like?
Looking forward to an answer
Cheers,
Felix
Take a look at the last line in your log.
'job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1070, pci bus id: 0000:21:00.0, compute capability: 6.1'
When running a tensorflow program leveraging GPU devices, I got the following message. What does the statement of Ignoring visible gpu
device (device: 0, name: Quadro 5000, pci bus id: 0000:05:00.0) with Cuda
compute capability 2.0. mean? What is to be ignored?
2017-12-04 16:06:17.784599: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with
properties:
name: Quadro 5000
major: 2 minor: 0 memoryClockRate (GHz) 1.026
pciBusID 0000:05:00.0
Total memory: 2.50GiB
Free memory: 2.22GiB
2017-12-04 16:06:17.784636: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-12-04 16:06:17.784645: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2017-12-04 16:06:17.784658: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:950] Ignoring visible gpu
device (device: 0, name: Quadro 5000, pci bus id: 0000:05:00.0) with Cuda
compute capability 2.0. The minimum required Cuda capability is 3.0.
TensorFlow prints this message when you have a GPU installed in the local machine but its compute capability is too low for TensorFlow to use that GPU for accelerated computation. As the error message states:
The minimum required Cuda capability is 3.0.
This means that you must use a CUDA-capable GPU with compute capability 3.0 or greater to use TensorFlow for GPU.