I am training a yolov3 model on custom dataset on colab. After training for around 50 minutes the tab got hanged and now the colab is showing the busy status.
It is also showing that red alert mark (49m 17s completed at 1:07 PM) at the bottom. What does it mean?
Do I need to interrupt the execution and start the process again?
I ran into the same issue while performing inference on a pixelcnn, the cell status showed it's running despite a busy kernel (three green dots) & red alert appeared on a status bar.
I kept the code running overnight just to verify and the cell kept running in the background, code generated the samples. However, I could not see the cell execution time.
All workspace variables were still present (%whos)
Please let me know if this is the case.
Related
This image shows the configuration.
This image shows the last state when the system is stuck in the wait state.
I am using Jitbit MacroRecorder and it is a really helpful tool. The problem is this: By using the IF image found feature, I want the program to execute 2 sequential commands as soon as the image I selected on my screen appears. However, the program is putting too much load on the CPU by running thousands of times until the IF statement is true. What I want is for the program to WAIT until the image in the IF image found statement appears on the screen, that is, not to run thousands of times.
My simple code
The only way for the program to know when the image appears is to continuously "scan" the screen until it sees it. You may be able to use the image NOT found feature inside a loop with a delay. You would look for the image, and if not found, delay a second, and then look for it again. The tradeoff is you will have that delay in there which will keep your program from running the following commands immediately.
From March 2021 google colab has added a captcha that randomly pops up after some time. So it is difficult or not possible to programmatically prevent Google Colab from disconnecting on a timeout. So there's no option other than opening the tab and clicking the captcha button. I often run model which take more training time and sometimes miss going back to it in 90 mins. The captcha can be popped up even while the training is not completed and soon terminates after a few minutes if not clicked.
Are there any solutions so that I can at least get an audio notification when I get a captcha in colab or if not pop-up message in browser or OS so that I can know about the inactivity?
I had found some ways get notified at some target lines in the notebook, say after training is completed. One is the colab browser notification feature, can be checked to true in settings. Other one is an audio solution, you just add these 2 lines of code found here to get an alarm when the code executes at any line in a cell.
What I am looking for is an alarm or pop up message in case of captcha.
This is probably a half-correct answer, but here's my shot at solving this
This article discusses how to bypass Captcha's, and has some code that clicks the check box if a captcha is found. I think it could be modified to have a sound alert instead.
However, Captchas were made to counter these sort of tools, plus I'm not sure about the legality implications etc, so I wouldn't recommend it.
The use of the below js increased the time before google sent a recaptcha for me. Btw, I set this as a bookmark, to call it whenever I need to. Executing the js bit in the console should be equivalent.
javascript:void(setInterval(function(){ console.log("Connect pushed"); document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); },60000))
Recently, I have seen the play button (the one to execute the cell) will have a spinning state with broken line as border and staying like this for a long time. But eventually, the line will turn solid and the thing will execute relatively fast after that. I would like to understand what the "broken" boundary mean? It seems to me it just hangs there waiting for something before actually running any code.
See image attach on that play button.
The spinning dashed line means the execution of the cell is queued behind an earlier execution.
(If you haven't yet executed a cell, it's likely that the backend is blocked by attempting to generate a completion or objection inspection. These should finish within a few seconds.)
I am running a long hyperparameter tuning using TensorFlow on google colab (GPU runtime), which is expected to take a few hours.
However, after over half an hour of training, the "Run cell" button turns from grey to red, with the message reading "Run cell\ cell has not been executed in this session\ executed at unknown time" and the values stop from printing in the output cell. The kernel is probably still running since I can't execute any other cell (the run cell button keeps rotating).
Why is this problem happening?
Is there a way to prevent this from happening?
If not, is there a way to measure the progress of the hyperparameter tuning given that the notebook stops from printing any outputs?
Thanks a lot.