ONNX converted TensorFlow saved model runs on CPU but not on GPU - tensorflow

Question

I converted a tensorflow saved model to ONNX format using tf2onnx :
python3 -m tf2onnx.convert --saved-model saved_model/ --output onnx/model.onnx --opset 11
The conversion worked fine and I can run inference with the ONNX model using CPU.
I installed onnxruntime-gpu to run inference with GPU and encountered an error :
RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Relu node. Name:'FirstStageFeatureExtractor/resnet_v1_101/resnet_v1_101/conv1/Relu' Status Message: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:97 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:91 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 2: out of memory ; GPU=0 ; hostname=coincoin; expr=cudaMalloc((void**)&p, size);
Stacktrace:
Stacktrace:
I am the only one using the GPU which is a Titan RTX (24GB of RAM). The model runs fine on GPU using its tensorflow saved model version, with 10GB of the GPU's RAM.
Versions are :
tensorflow 1.14.0
CUDA 10.0
CuDNN 7.6.5
onnx 1.6.0
onnxruntime 1.1.0
tf2onnx 1.9.2
python 3.6
Ubuntu 18.04

Answer 1

according to your information and Versions, maybe two solutions to solve the problem:
the information is out of memory, please check the gpu memory is available yet, if not available, set the gpu memory:
config = tf.ConfigProto()
config.gpu_options.visible_device_list = "0"
config.gpu_options.per_process_gpu_memory_fraction = 0.1
set_session(tf.Session(config=config))
downgrade onnxruntime-gpu, according to your Versions, CUDA 10.0 matchs with onnxruntime-gpu==1.0.0, not 1.1.0, which needs cuda 11
https://github.com/Microsoft/onnxruntime/releases/tag/v1.1.0