I am using tensorflow-gpu 2.3.0 with CUDA_VERSION=8.0.61 and CUDNN_VERSION=6.0.21.
I just run a tensorflow code and get FailedPreconditionError:
'tensorflow.python.framework.errors_impl.FailedPreconditionError: Failed to allocate scratch buffer for device 0 [Op:VarHandleOp] name: Variable/'
What can I do to fix this?
Thanks
Related
I enable GPU by going to >>runtime>> change runtime type >> then choose GPU.
But when I run my code I get this error:
usage: train.py [-h] [--pre PRETRAINED] TRAIN TEST GPU TASK
train.py: error: the following arguments are required: GPU, TASK
this is the part of the code that make error:
! python train.py part_A_train.json part_A_val.json
I also have this warning:
Warning: You are connected to a GPU runtime, but not utilising the GPU.
but by running this code looks like that GPU is active!
I am new with Deep learning. I have a A100 GPU installed with CUDA 11.6. I installed using Conda tensor flow-1.15 and tensorflow gpu - 1.15, cudatoolkit 10.0, python 3.7 but the code I am trying to run from github has given a note as below and it shows errors which I am finding difficult to interpret where I went wrong.The error is displayed as
failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED 2022-06-30 09:37:12.049400: I tensorflow/stream_executor/stream.cc:4925] [stream=0x55d668879990,impl=0x55d668878ac0] did not memcpy device-to-host; source: 0x7f2fe2d0d400 2022-06-30 09:37:12.056385: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at iterator_ops.cc:867 : Cancelled: Operation was cancelled
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Blas GEMM launch failed : a.shape=(25, 25), b.shape=(25, 102400), m=25, n=102400, k=25 [[{{node Hyperprior/HyperAnalysis/layer_Hyperprior_1/MatMul}}]] (1) Internal: Blas GEMM launch failed : a.shape=(25, 25), b.shape=(25, 102400), m=25, n=102400, k=25 [[{{node Hyperprior/HyperAnalysis/layer_Hyperprior_1/MatMul}}]] [[Hyperprior/truediv_3/_3633]]
NOTE: At the moment, we only support CUDA 10.0, Python 3.6-3.7, TensorFlow 1.15, and Tensorflow Compression 1.3. TensorFlow must be installed via pip, not conda. Unfortunately, newer versions of Tensorflow or Python will not work due to various constraints in the dependencies and in the TF binary API.
I am learning tesnorflow from this blog:
http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/
The code i am running is :
https://github.com/dennybritz/cnn-text-classification-tf/blob/master/train.py
I have installed tensorflow from sourcse in a virtual enviroment,in CPU only enviroment using followinbg bazel build command: bazel build --config=mkl ...
here is the exact error:
"2018-01-16 03:15:27.783040: F tensorflow/core/kernels/mkl_maxpooling_op.cc:157] Check failed: dnnPoolingCreateForward_F32( &prim_pooling_fwd, primAttr, algorithm, lt_user_input, params.kernel_size, params.kernel_stride, params.in_offset, dnnBorderZerosAsymm) == E_SUCCESS (-127 vs. 0)
Aborted
"
I have debugged error to the line where sess.run is written, i have beleived it has something to do it mkl_maxpooling, as i had installed tensorflow with mkl optimization of INTEL cpu's
Given below are the steps that I followed:
Build tensorflow 1.4 from source with mkl as mentioned in the question
Cloned the git repo "https://github.com/dennybritz/cnn-text-classification-tf.git"
Ran "python train.py" from "cnn-text-classification-tf" directory(created from git clone)
Code ran without any errors. So it seems like the tensorflow was not properly built from the source. Please confirm that there were no errors while building tensorflow from source.
Following the tutorial on TensorFlow for Poets (Android) (https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2/#0)
Attempting to use Inception model instead of Mobilenet
Trying to strip DecodeJpeg Op from the retrained model using strip_unused.py, but encountered the following error.
Error:
/home/user/tensorflow/bin/python: Error while finding spec for 'tensorflow.python.tools.strip_unused.py' (AttributeError: module 'tensorflow.python.tools.strip_unused' has no attribute '__path__')
Command line:
python -m tensorflow.python.tools.strip_unused.py --input_graph=tf_files/retrained_graph.pb --output_graph=tf_files/stripped_graph.pb --input_node_names="Mul" --output_node_names="final_result" --input_binary=true
Machine:
Ubuntu 16.04 LTS
Python 3.5.2
TensorFlow 1.4.1
Any assistance is greatly appreciated. Thanks!
Might be due to the typo mistake as file extension .py was specified. This seemed to work:
python -m tensorflow.python.tools.strip_unused --input_graph=tf_files/retrained_graph.pb --output_graph=tf_files/stripped_graph.pb --input_node_names="Mul" --output_node_names="final_result" --input_binary=true
Result: 997 ops in the final graph.
I'm running the CIFAR-10 classification from the Tensorflow for the very first time on my laptop with GeForce 940M. I'm running the training with the pre-defined parameters as follows:
python cifar10_train.py
after step 1800 I'm getting the following errors:
E tensorflow/stream_executor/cuda/cuda_event.cc:33] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
E tensorflow/stream_executor/cuda/cuda_driver.cc:1182] failed to enqueue async memcpy from device to host: CUDA_ERROR_ILLEGAL_ADDRESS; host dst: 0x7ff8e9bf26c0; GPU src: 0x5011c0600; size: 16=0x10
F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:105] Unexpected Event status: 1
I tensorflow/stream_executor/stream.cc:3304] stream 0x35e7190 did not block host until done; was already in an error state
Aborted (core dumped)
Does anybody have any idea?
Thanks a lot in advance for your help! Any advice is kindly appreciated!