Google Colab (Colab Pro+) notebook running cell automatically stops usually around 2:45-3pm everyday even if the cell is not idle
Related
I am training a yolov3 model on custom dataset on colab. After training for around 50 minutes the tab got hanged and now the colab is showing the busy status.
It is also showing that red alert mark (49m 17s completed at 1:07 PM) at the bottom. What does it mean?
Do I need to interrupt the execution and start the process again?
I ran into the same issue while performing inference on a pixelcnn, the cell status showed it's running despite a busy kernel (three green dots) & red alert appeared on a status bar.
I kept the code running overnight just to verify and the cell kept running in the background, code generated the samples. However, I could not see the cell execution time.
All workspace variables were still present (%whos)
Please let me know if this is the case.
Since I can't keep colab running without interracting with it, I would like to pause my session. If I press the stop button, I'll lose what's been done in the running cell so far. Is there some way to avoid this?
Recently, I have seen the play button (the one to execute the cell) will have a spinning state with broken line as border and staying like this for a long time. But eventually, the line will turn solid and the thing will execute relatively fast after that. I would like to understand what the "broken" boundary mean? It seems to me it just hangs there waiting for something before actually running any code.
See image attach on that play button.
The spinning dashed line means the execution of the cell is queued behind an earlier execution.
(If you haven't yet executed a cell, it's likely that the backend is blocked by attempting to generate a completion or objection inspection. These should finish within a few seconds.)
I am running a long hyperparameter tuning using TensorFlow on google colab (GPU runtime), which is expected to take a few hours.
However, after over half an hour of training, the "Run cell" button turns from grey to red, with the message reading "Run cell\ cell has not been executed in this session\ executed at unknown time" and the values stop from printing in the output cell. The kernel is probably still running since I can't execute any other cell (the run cell button keeps rotating).
Why is this problem happening?
Is there a way to prevent this from happening?
If not, is there a way to measure the progress of the hyperparameter tuning given that the notebook stops from printing any outputs?
Thanks a lot.
I have a written a intern functional test which copies data from one cell and pastes it to another cell. The problem is every time run the test and copy the data I get a dialogue box with the message
Pasting from clipboard is
currently turned off by your browser.Local Data will be
used
Now because of this popup my functional test freezes and everything else starts to fails. Luckily to avoid this there is a way. .acceptAlert()
.copyfunction()
.sleep(500) //wait for popup
.acceptAlert() //first attempt to copy after login causes alert about using local memory
.pastefunction()
Although this works locally there are times the test still fails when in jenkins. Its not stable. How do I make it stable.