I thought ColabPro+ would allow me to run on a GPU for longer than 24hr, but the VM is getting killed right at 24hrs, while the (Chrome) browser is open, and a python program has been running the whole time using the GPU. I tried running in background mode and in this case it killed the VM in about an hour. What am I missing?
Related
I have Colab Pro+ subscription. It now (I guess from yesterday) runs my notebooks as background. So I can only run two instances. Previously I was able to run 3. When I open Manage Sessions from Runtime > manage Sessions I can see that current colab notebook is running in Background Execution mode.
Sadly there seems to be no way to turn it off. When I click Runtime > Change Runtime Type I get this (no option to uncheck background execution)
Is there any other place, where I can turn background execution off?
I've spent about a day looking for solutions around the web to my issue but none work for me.
Here is my scenario:
I am running Selenium scripts with ChromeDriver using pyATS framework on my Ubuntu 18.04 VM. The VM has 4 GB of memory. I also have setup Jenkins on the machine and am trying to run the pyATS script with the pyATS plugin.
When running headless mode from the terminal, the script runs in the same or faster time than non-headless mode. However, when I run in Jenkins on the same machine, I am getting extreme slowdowns. It looks almost as if Jenkins is running my script in sections, with >2 minutes of delay in between steps at random.
I've tried out Xvfb, headless with various chrome options (noproxy, proxy options, gpu disable, etc), increasing heap memory for jenkins, but I always get the same random 2 min of delay in between script steps.
The script doesn't fail - it will complete eventually. But for a step that I expect to take around 2 min, jenkins will take 10 minutes.
I currently don't have a way to increase the memory my VM has, but are there any other solutions that I can try in the meantime?
Found the issue, I had to set the "--proxy-server" for Chrome to the proxy my VM was running behind. For some reason Firefox was working fine without that option so I didn't think to set this option for Chrome.
I'm stuck in a frustrating loop trying to use the tensorflow estimator API.
When I try to restart my GPU instance, my notebook hangs on initialization.
If I exit the notebook, or switch runtime to CPU and back to GPU again, and try to connect to my instance, I says the instance is busy.
If I switch my runtime to no GPU and restart, the runtime initializes fine, but if I then try and reset the runtime to GPU, the notebook again says it is busy running what I assume to be a hanging GPU task.
So restarting the runtime, exiting the notebook, and switching the runtime to CPU and back to GPU do not seem to help with freeing/restarting the GPU backend.
Is there anything else I can try?
To reset your backend, select the command 'Reset all runtimes...' from the Runtime menu.
Running Windows 10 on my laptop (with a compatible GPU) and just started using tensorflow-gpu 2 days ago. Every time I close the lid to put the laptop to sleep, it restarts when I wake it up. This never happened when I was just running normal tensorflow. I thought I'd see here if this is normal or known before running it up the flagpole on their git repo.
I need to run an intensive CPU task that maxes all cores of my CPU to 100%.
After a few days of running this task, I find that the machine becomes unresponsive and I am no longer able to SSH into it. I then have to restart the machine and begin the task again. This task could take several weeks or even months to compute.
I'd like to find a way to run this task to completion.
I've tried running the task on Debian 8 and Ubuntu Server LTS. Both of these operating system exhibited the same problem. I thought about running the task inside a Virtual Machine and using Cron to snapshot it every hour, but this seems quite extreme and would suffer an overhead.
Why is it that my machine is unstable?
Could it be due to power fluctuations?
Should I try under-clocking the CPU?
Thanks