system auto reboot when tensorflow model is too large - tensorflow

Question

I'm using a nvidia GTX1080 gpu(8GB) to run Inception model on tensorflow, when I set batch_size = 16 and image_size = 400, then after I start the program, my ubuntu14.04 will auto reboot.

Answer 1

Make sure it is not a power supply unit problem. I was observing strange occasional reboots on my development machine. As I was increasing the size of input (batch size, larger NN) the rate of reboots was increasing as well. Turned out to be a PSU problem. A quick check is to limit GPU power consumption and see if this behavior will go away. For instance, you can limit power to about 150 watts with this command (you'll need a sudo rights):
sudo nvidia-smi -pl 150

Answer 2

I tracked the issue down to a faulty power supply. It had enough capacity according to spec, and limiting GPU power consumption by running "nvidia-smi -pl 150" didn't help at all. Probably it couldn't handle bursts in power consumption.
Anyway, after I changed the power supply from "Corsair CX750 Builder Series ATX 80 PLUS" to "Cooler Master V1000", the issue is gone.
See details of my investigation in the TensorFlow GitHub issue.

Answer 3

Changing the GPU power settings will work, if you have PSU with enough power (WATTS).
I limited my GPU's (TITANX) power to max. 200 WATTS using,
sudo nvidia-smi -pl 200
NOTE: Each GPU has power limitations, for e.g. TITANX's power limit is between 125W and 300W. So make sure to give value between those limits.

Answer 4

I was facing similar problems. Even with small batch sizes in both tensorflow and pytorch the pc was restarting by itself. I removed a video card but still no solution. Just nvidia-smi -pl 150 didn't work.
In addition;
sudo nvidia-smi -pm 1
sudo nvidia-smi -lgc 1400
sudo nvidia-smi -lmc 6500
sudo nvidia-smi -gtt 65
sudo nvidia-smi -cc 1
sudo nvidia-smi -pl 165
I added them and now it works with 2 gpu without any problem. These settings are for the RTX2080TI. Edit according to your own video card.
My system:
HP Z800 Workstation
Intel(R) Xeon(R) CPU E5-2643 0 # 3.30GHz
PSU 850W
Ubuntu 20.04
2x RTX 2080TI

Answer 5

I got the exactly same problem after a GTX 2070 installed on DELL T3610. The answer provided by Sergey above solved my problem. Just add a comment for windows users:
Run your command prompt as administrator
go to nvidia-smi directory: typically it is under C:\Program Files\NVIDIA Corporation\NVSMI
run nvidia-smi -pl 150
Then your problem should be solved and you will see the output that the power limit of your GPU has been reduced to 150w. (In my case, reduced to 150w from 185w).

Answer 6

I had a very similar problem but tracked it down to a PATH problem where CUDA 11 got inserted and somehow was overriding my CUDA 10.1 libraries. I am not sure when/how, but it might be related to an upgrade of the Nvidia drivers I had done recently. At least check and make sure your PATH and versions are correct. CUDA 11 will not work with Tensorflow 2.3.1 or prior, at least as of 11/2020 on Windows 10. Please let me know if there is a workaround that I am unaware of, but this was definitely the problem. When I fixed the PATH to point to the CUDA 10.1 path only, everything worked fine and I was able to max out the GPU for over 20 minutes with no restart.

Answer 7

I had the same issue and limiting poser usage resolved it. I had to reduce power supply to 150 as 200 did not work though.