Non-OK-status: status: Internal: initialization error - tensorflow

environment:tensorflow 2.3.0 Anaconda with CUDA Toolkit 10.1 CUDNN 7.6.0 (Windows 10)
I run my code with multiprocessing it return a issue
Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: initialization error
but the problem goes away when I run the code using "else" methods:
This is the related code:
if runInProcess:
p = multiprocessing.Process(target=Train, args=(yml, train_subs, loadData, channDic, eventTypeDic, getData, folderIndex, modelPath))
p.start()
p.join()
else:
Train(yml, train_subs, loadData, channDic, eventTypeDic, getData, folderIndex, modelPath)
My GPU : RTX2080Ti 12G
I already run the nvcc -V and it returns :
meya#admin:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
This is the full result :
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2022-05-30 09:54:42.815682: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-05-30 09:54:42.815732: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2022-05-30 09:54:42.815769: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-05-30 09:54:42.815804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-05-30 09:54:42.815838: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2022-05-30 09:54:42.815872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2022-05-30 09:54:42.815907: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2022-05-30 09:54:42.818666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2022-05-30 09:54:42.818744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2022-05-30 09:54:43.491362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-05-30 09:54:43.491412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2022-05-30 09:54:43.491419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2022-05-30 09:54:43.492693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9374 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:af:00.0, compute capability: 7.5)
../model/1S_FB1-100_S100_chanSeq.yml
Will Return
folder Index: 1
--------------------- Fetching 1/1 Data from suject 1 ---------------------
Extracting EDF parameters from
/data0/meya/MI/RaiseLower/Action/session_miltiprocess/s01/evt.bdf...
BDF file detected
Setting channel info structure...
Creating raw.info structure...
/data0/meya/code/code_test/MI/interSub/../../MI/loadData.py:204: RuntimeWarning: Omitted
136 annotation(s) that were outside data range.
annotationData = mne.io.read_raw_bdf(eventFile)
mne version > 0.20
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 4 - 1e+02 Hz
FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 4.00
- Lower transition bandwidth: 2.00 Hz (-6 dB cutoff frequency: 3.00 Hz)
- Upper passband edge: 100.00 Hz
- Upper transition bandwidth: 25.00 Hz (-6 dB cutoff frequency: 112.50 Hz)
- Filter length: 1651 samples (1.651 sec)
----------------------Build Model----------------------
Model Type: EEGNet
2022-05-30 09:55:53.385452: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: initialization error
What are the issues and how to fix it?

Related

Trying to use Tensorflow with RTX 3090 Errors

I'm attempting to use Tensorflow with a Rtx 3090 GPU, however I've been experiencing a variety of issues for several days. I tried the remedies suggested here and in other places, but they didn't work. Either a kernel error occurs, or the program proceeds with the CPU without seeing the GPU. Could you please assist me?
2021󈚰󈚺 13:21:07.654550: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021󈚰󈚺 13:21:09.144192: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance‑critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021󈚰󈚺 13:21:09.149726: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021󈚰󈚺 13:21:09.172491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021󈚰󈚺 13:21:09.173145: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021󈚰󈚺 13:21:09.201143: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021󈚰󈚺 13:21:09.201496: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021󈚰󈚺 13:21:09.218490: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021󈚰󈚺 13:21:09.222724: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021󈚰󈚺 13:21:09.253841: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021󈚰󈚺 13:21:09.272022: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021󈚰󈚺 13:21:09.272867: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021󈚰󈚺 13:21:09.273229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021󈚰󈚺 13:21:09.715332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021󈚰󈚺 13:21:09.715688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021󈚰󈚺 13:21:09.715891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021󈚰󈚺 13:21:09.716223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/device:GPU:0 with 18786 MB memory) ‑> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:08:00.0, compute capability: 8.6)
2021󈚰󈚺 13:21:10.046619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021󈚰󈚺 13:21:10.047281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021󈚰󈚺 13:21:10.047754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 82 deviceMemorySize: 24.00GiB deviceMemoryBandwidth: 871.81GiB/s
2021󈚰󈚺 13:21:10.048414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021󈚰󈚺 13:21:10.048707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021󈚰󈚺 13:21:10.049027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021󈚰󈚺 13:21:10.049227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021󈚰󈚺 13:21:10.049491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18786 MB memory) ‑> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:08:00.0, compute capability: 8.6)
2021󈚰󈚺 13:21:10.928282: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021󈚰󈚺 13:21:25.315947: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
These are just informational messages as they are prefixed with I, if it is the error message they would be prefixed with E or W for warnings are as shown below:
2020-12-30 21:30:27.549172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cupti64_101.dll
2020-12-30 21:30:27.599977: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory.
2021-12-30 21:30:27.704083: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1307] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
You can surpass these warnings using below code:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
You can also check executing this code:
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

Slow loading time - EfficientDet D2

I am loading a Tensorflow 2 version of EfficientDet D2 (http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d2_coco17_tpu-32.tar.gz) using a Jetson AGX Xavier.
I run the following script:
#!/usr/bin/python3
import tensorflow as tf
import time
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as viz_utils
PATH_TO_SAVED_MODEL = "./efficientdet_d2_coco17_tpu-32/saved_model/"
print('Loading model...')
start_time = time.time()
# Load saved model and build the detection function
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)
end_time = time.time()
elapsed_time = end_time - start_time
print('Done! Took {} seconds'.format(elapsed_time))
However, the performance results is a loading time of more than 13 minutes.
This is the output after the command has been executed:
./test.py
2021-07-04 10:58:58.074413: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-04 10:59:05.375568: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
Loading model...
2021-07-04 11:00:54.337115: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-07-04 11:00:54.342226: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-07-04 11:00:54.347726: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.347959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-07-04 11:00:54.348037: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-04 11:00:54.353788: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-04 11:00:54.354040: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-07-04 11:00:54.358471: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-04 11:00:54.359514: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-04 11:00:54.364904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-04 11:00:54.369140: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-04 11:00:54.369861: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-04 11:00:54.370262: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.370843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.371060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-04 11:00:54.375404: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.375623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-07-04 11:00:54.375714: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-04 11:00:54.375823: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-04 11:00:54.375908: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-07-04 11:00:54.376011: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-04 11:00:54.376090: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-04 11:00:54.376167: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-04 11:00:54.376287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-04 11:00:54.376369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-04 11:00:54.376673: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.376972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:00:54.377093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-07-04 11:05:01.847060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-04 11:05:01.847174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-07-04 11:05:01.847226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-07-04 11:05:01.847710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:05:01.848589: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:05:01.848911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:908] ARM64 does not support NUMA - returning NUMA node zero
2021-07-04 11:05:01.849096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 19271 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-07-04 11:05:01.850298: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Done! Took 793.8719098567963 seconds
With the computing power of the Xavier I would expect much better performance? Anybody knows what the cause to this could be?
Thanks for any help or input!
The time it takes is not just to load the model, but to initialize the device. Maybe the problem is in the driver.
To prove it, try to init a smaller model, or toy example like a+b=c. I expect it will take similar time.
Also the computing power has nothing to do with loading the model. Loading of the model depends more on memory management of the driver and TF. The actual building of the model in the memory may be done on the CPU, even when using GPU or other accelerator (just guessing).
My experience with CUDA and TF is 5 minutes initialization time with one version of CUDA, TF and GPU driver. And less than 30 sec with another version of CUDA and TF, on the same hardware (8x1080ti GPUs).

tensorflow Lite Segmentation fault

System information
OS Platform and Distribution :CentOS Linux release 7.7.1908
-TensorFlow version:2.3.0
I try to convert the tensorflow offical image caption model to TFLite model
And Now I have successfully convert the model using tf.lite.TFLiteConverter.from_concrete_functions
as following:
#tf.function
def evaluate(img_tensor_val):
temp_input = tf.expand_dims(img_tensor_val, 0)
img_tensor_val = image_features_extract_model(temp_input)
img_tensor_val = tf.reshape(img_tensor_val, (img_tensor_val.shape[0], -1, img_tensor_val.shape[3]))
hidden = decoder.reset_states(batch_size=1)
features = encoder(img_tensor_val)
dec_input = tf.expand_dims([tokenizer.word_index['<start>']], 0)
result = []
for i in range(max_length):
predictions, hidden, attention_weights = decoder(dec_input, features, hidden)
print(predictions.shape)
# result.append(predictions)
predicted_id = tf.random.categorical(predictions, 1)[0][0]
#
#
result.append(predicted_id)
#
#
# if predicted_id == 3:
# return result
# # result.append(tf.gather(tokenizer.index_word, predicted_id))
# #
# # if tf.gather(tokenizer.index_word, predicted_id) == '<end>':
# # return result
#
dec_input = tf.expand_dims([predicted_id], 0)
return result
export_dir = "./"
tflite_enc_input = ''
ckpt.f = evaluate
to_save = evaluate.get_concrete_function(tf.TensorSpec(shape=(299, 299, 3),dtype=tf.dtypes.float32))
converter = tf.lite.TFLiteConverter.from_concrete_functions([to_save])
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,
tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
And I Visualize the converted_model.tflite by Netorn:
But when I invoke the interpreter the problem came:
LOG:
2020-10-03 12:11:24.049222: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-03 12:11:30.184705: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-03 12:11:30.213363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:af:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2020-10-03 12:11:30.213414: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-03 12:11:30.219666: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-03 12:11:30.223018: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-03 12:11:30.224419: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-03 12:11:30.227861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-03 12:11:30.230195: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-03 12:11:30.236320: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-03 12:11:30.239374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-03 12:11:30.239829: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-03 12:11:30.248265: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2600000000 Hz
2020-10-03 12:11:30.249524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615faa7fa90 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-03 12:11:30.249552: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-03 12:11:30.381691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615faaec0c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-03 12:11:30.381734: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0
2020-10-03 12:11:30.383860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:af:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2020-10-03 12:11:30.383900: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-03 12:11:30.383930: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-03 12:11:30.383944: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-03 12:11:30.383959: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-03 12:11:30.383973: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-03 12:11:30.383987: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-03 12:11:30.384002: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-03 12:11:30.387738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-03 12:11:30.387786: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-03 12:11:31.156790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-03 12:11:31.156840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-10-03 12:11:31.156853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-10-03 12:11:31.160006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30098 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:af:00.0, compute capability: 7.0)
**(299, 299, 3)**
INFO: Created TensorFlow Lite delegate for select TF ops.
2020-10-03 12:11:31.760387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:af:00.0 name: Tesla V100-SXM2-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.72GiB deviceMemoryBandwidth: 836.37GiB/s
2020-10-03 12:11:31.760470: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-03 12:11:31.760523: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-03 12:11:31.760551: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-03 12:11:31.760577: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-03 12:11:31.760601: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-03 12:11:31.760625: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-03 12:11:31.760647: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-03 12:11:31.763282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-03 12:11:31.763329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-03 12:11:31.763346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-10-03 12:11:31.763360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-10-03 12:11:31.766083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30098 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:af:00.0, compute capability: 7.0)
**INFO: TfLiteFlexDelegate delegate: 51 nodes delegated out of 2014 nodes with 51 partitions.**
**Segmentation fault (core dumped)**
The invoke of Interpreter
def load_image(image_path):
img = tf.io.read_file(image_path)
img = tf.image.decode_jpeg(img, channels=3)
img = tf.image.resize(img, (299,299))
img = tf.keras.applications.inception_v3.preprocess_input(img)
return img, image_path
image = load_image('./test.jpg')[0]
print(image.shape)
interpreter = tf.lite.Interpreter(model_path='./converted_model.tflite')
input_details = interpreter.get_input_details()
interpreter.allocate_tensors()
interpreter.set_tensor(input_details[0]['index'], image)
interpreter.invoke()
raw_prediction = interpreter.tensor(
interpreter.get_output_details()[0]['index'])()
print(raw_prediction)
Please tell me what 's the problem of the program?What's the meaning of 'Segmentation fault (core dumped)' ?
After I change the GPU I use to CPU, the program is now correct.But I don't know the Why the theraw_prediction=291 .But In the evaluation function, the return is a list[].How can this be?

no kernel image is available for execution on the device Fatal Python error: Aborted

I want to run yolov4 code in this repo: https://github.com/hunglc007/tensorflow-yolov4-tflite
And I installed python 3.7 and all requirements and cuda and cudnn.
By the log, the cudnn and cuda is installed well, but there is error of "no kernel image is available for execution on the device" what is this error? is it related in cuda or cudnn version error?
Python: 3.7.9, CUDA: 10.1, Tensorflow:2.3.0rc0, Tensorflow-GPU:not installed, CUDNN:7.5.0, OS: Windows10(x64)
py -3.7 save_model.py --weights ./data/yolov4.weights --output ./checkpoints/yolov4-416-tflite --input_size 416 --model yolov4 --framework tflite
2020-09-03 11:02:05.897607: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-09-03 11:02:09.504648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-09-03 11:02:09.997508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 1.2415GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
2020-09-03 11:02:10.017273: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-09-03 11:02:10.036505: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-09-03 11:02:10.059534: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-09-03 11:02:10.074749: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-09-03 11:02:10.094710: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-09-03 11:02:10.115167: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-09-03 11:02:10.140633: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-09-03 11:02:10.148636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-03 11:02:10.155846: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-09-03 11:02:10.188413: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x295adc030a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-03 11:02:10.199421: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-03 11:02:10.207675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 1.2415GHz coreCount: 3 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 13.41GiB/s
2020-09-03 11:02:10.222939: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-09-03 11:02:10.231890: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-09-03 11:02:10.241896: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-09-03 11:02:10.250393: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-09-03 11:02:10.260177: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-09-03 11:02:10.268644: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-09-03 11:02:10.278132: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-09-03 11:02:10.286635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-03 11:02:10.380510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-03 11:02:10.388703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-09-03 11:02:10.394562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-09-03 11:02:10.402323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1464 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)
2020-09-03 11:02:10.429701: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x295ae120140 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-03 11:02:10.441631: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce 940MX, Compute Capability 5.0
2020-09-03 11:02:10.619742: F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device
Fatal Python error: Aborted
The error indicates that the pre-built binary used in tensorflow, does not support the SM version (compute capability) supported by your actual hardware.
You can refer to below link for supported combinations:
https://www.tensorflow.org/install/source_windows#gpu
Based on this, both 2.1.0 and 2.3.0 require CUDNN 7.4 and CUDA 10.1. You should try with these supported combinations.
[2.3.0 release/rc2/rc0 specific] from https://github.com/tensorflow/tensorflow/releases/tag/v2.3.0 - TF 2.3 includes PTX kernels only for compute capability 7.0 to reduce the TF pip binary size. Earlier releases included PTX for a variety of older compute capabilities.

keras error when trying to get intermediate layer output: Could not create cudnn handle

I am building a model using keras.
I am using:
anaconda (python 3.7)
tensorflow-gpu (2.1)
keras (2.3.1)
cuda (10.1.2)
cudnn (7.6.5)
nvidia driver (445.7)
nvidia gpu: gtx 1660Ti (6GB)
when I am trying to run a model, there is a code that creates an error:
def get_gen_output(gan, noise):
intermediate_model=Model(inputs=gan.input,outputs=gan.layers[24].output)
layer_output = intermediate_model.predict(noise)
return layer_output[0]
this model is a CNN gan. I can run other CNN models well, only this model creates a problem.
the error I get is:
Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
from other questions that faces the same problem, I see that there are two common things that can cause it:
insufficient gpu memory - but I dont think this is the problem, since even if I create a very small model that includes the code snippet from above the error appears. and bigger models without this code work well.
problem with cuda and cudnn compatibility - but based on this link, the version I listed above should work.
any idea what could be the problem and how to fix it? I have been trying to solve this for days now.
if any more information is needed (summary of the model for example), please let me know in the comments and I will add it.
UPDATE: a comment asked me to post the logs:
(base) C:\Users\Moran>ju[yter notebook
'ju[yter' is not recognized as an internal or external command,
operable program or batch file.
(base) C:\Users\Moran>jupyter notebook
[I 16:42:41.966 NotebookApp] Serving notebooks from local directory: C:\Users\Moran
[I 16:42:41.967 NotebookApp] The Jupyter Notebook is running at:
[I 16:42:41.967 NotebookApp] http://localhost:8888/?token=ec3a664897f7d31597f7f4544609cc8c0d7b4db7450b55b1
[I 16:42:41.967 NotebookApp] or http://127.0.0.1:8888/?token=ec3a664897f7d31597f7f4544609cc8c0d7b4db7450b55b1
[I 16:42:41.967 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 16:42:42.000 NotebookApp]
To access the notebook, open this file in a browser:
file:///C:/Users/Moran/AppData/Roaming/jupyter/runtime/nbserver-15820-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=ec3a664897f7d31597f7f4544609cc8c0d7b4db7450b55b1
or http://127.0.0.1:8888/?token=ec3a664897f7d31597f7f4544609cc8c0d7b4db7450b55b1
[I 16:42:47.284 NotebookApp] Kernel started: ae448b14-33fc-471e-a2ae-991be8321434
[W 16:42:47.740 NotebookApp] 404 GET /api/kernels/4ce83e1e-9aa5-4c93-97d8-55dc16480242/channels?session_id=eaa90dc2c0bb4c448d6a01d66f4fbb21 (127.0.0.1): Kernel does not exist: 4ce83e1e-9aa5-4c93-97d8-55dc16480242
[W 16:42:47.757 NotebookApp] 404 GET /api/kernels/4ce83e1e-9aa5-4c93-97d8-55dc16480242/channels?session_id=eaa90dc2c0bb4c448d6a01d66f4fbb21 (127.0.0.1) 18.94ms referer=None
[W 16:42:49.439 NotebookApp] 404 GET /api/kernels/b9e9b610-9c5b-4565-8b85-deb70837c31f/channels?session_id=34072dd627c74e96b496ef73d99601a9 (::1): Kernel does not exist: b9e9b610-9c5b-4565-8b85-deb70837c31f
[W 16:42:49.440 NotebookApp] 404 GET /api/kernels/b9e9b610-9c5b-4565-8b85-deb70837c31f/channels?session_id=34072dd627c74e96b496ef73d99601a9 (::1) 2.00ms referer=None
2020-04-12 16:43:00.321827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-12 16:43:02.652473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-04-12 16:43:02.685848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2020-04-12 16:43:02.693105: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-12 16:43:02.700970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-04-12 16:43:02.708335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-04-12 16:43:02.713049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-04-12 16:43:02.720598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-04-12 16:43:02.726428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-04-12 16:43:02.738007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-12 16:43:02.741940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-12 16:43:02.745942: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2020-04-12 16:43:02.754621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2020-04-12 16:43:02.761464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-12 16:43:02.766394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-04-12 16:43:02.770257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-04-12 16:43:02.773975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-04-12 16:43:02.777827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-04-12 16:43:02.782949: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-04-12 16:43:02.786952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-12 16:43:02.791207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-12 16:43:03.372450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-12 16:43:03.376375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-04-12 16:43:03.379436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-04-12 16:43:03.382400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/device:GPU:0 with 4625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-04-12 16:43:03.966022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1660 Ti computeCapability: 7.5
coreClock: 1.59GHz coreCount: 24 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 268.26GiB/s
2020-04-12 16:43:03.976011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-12 16:43:03.980766: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-04-12 16:43:03.985179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-04-12 16:43:03.988922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-04-12 16:43:03.992744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-04-12 16:43:03.997758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-04-12 16:43:04.001856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-12 16:43:04.006936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-12 16:43:04.009739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-12 16:43:04.014702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-04-12 16:43:04.017351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-04-12 16:43:04.020371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
[W 16:43:04.449 NotebookApp] Replacing stale connection: 4ce83e1e-9aa5-4c93-97d8-55dc16480242:eaa90dc2c0bb4c448d6a01d66f4fbb21
2020-04-12 16:43:05.280820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-12 16:43:06.518456: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-04-12 16:43:06.522375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-04-12 16:43:06.525103: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node 1/convolution}}]]
[W 16:43:06.741 NotebookApp] Replacing stale connection: b9e9b610-9c5b-4565-8b85-deb70837c31f:34072dd627c74e96b496ef73d99601a9
[I 16:43:08.454 NotebookApp] Saving file at /generative models/GAN.ipynb
Kindly remove nvidia cuda toolkit from both anaconda environment as well as system.
sudo apt-get remove nvidia-cuda-toolkit
conda remove cudatoolkit
And, use the following option while calling tensorflow session
Tensorflow
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)
For keras,
from keras.backend.tensorflow_backend import set_session
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
set_session(sess) # set this TensorFlow session as the default session for Keras