This week, my AlphaFold Colab jobs have been crashing in cell 5. I have run identical sequences previously with no issue. Was an update recently installed in AlphaFold-Colab?
Related
I'm running a CNN Deep Learning Classification model on colab pro+. However, without reason, session is interrupted, and no log is presented in the app.log file.
enter image description here
As can be seen in the image, It's 30th november and the last entry in the app.log was yesterday. The program has just stopped running without reason.
Can you help me?
I've sent a question to colab github acccount.
I have a colab pro plus subscription, and I ran a python code that runs a network using pytorch.
Until now, I succeeded in training my network and using my Google drive to save checkpoints and such.
But now I had a run that ran for around 16 hours, and no checkpoint or any other data was saved - even though the logs clearly show that I have saved the data and even evaluated the metrics on saved data.
Maybe the data was saved on a different folder somehow?
I looked in the drive activity and I could not see any data that was saved.
Has anyone ran to this before?
Any help would be appreciated.
I'm trying to run two notebooks on Google Colab but could only connect one notebook at a time to the virtual machine. There's a pop-up message saying "Too many sessions. You have too many active sessions. Terminate an existing session to continue." when I click the "connect" button on the second notebook. Does anybody know why?
Screenshot:
Edit: I'm using Google Chrome on Windows 10
Edit March 3, 2020: I ended up not using Colab that day, but I came back the next day and was able to run two Colab notebooks just fine. strange. I had this issue a couple of times since I posted this question, but the error disappeared the following day.
I am used to use it with 2 active sessions. I mean, it gives that error when I try to connect for the third notebook. Today, however, I could only connect 1 notebook at a time. It does not permit the second connection. Therefore, the limit changes time to time.
I have the same problem and I found my solution in this issue. I change the Runtime Shape to Standard in the 2nd notebook, it worked for me.
Try this:
Go to Menu "Runtime" > "Manage Sessions", you should see a list of active sessions. Terminate those you don't need. Although you think you are opening only 2 notebooks, some previous sessions may still linger around if you just close the browser tab.
Note: However, when I hit the problem today, I did the above to check, and I have only 1 other session. Usually, I am able to run up to 3 separate sessions. I am not sure if google is dynamically adjusting this based on overall demand. I also suspect ever since they introduce the Pro, priorities may be given to subscribers.
Im not sure but may be collab provides services to a account one at a time..
I was recently working in a notebook on Google Colab and my computer ran out of battery and died. All the progress I had made was not saved anywhere!
I'm very used to having jupyter notebooks, which saves my files pretty much every time I execute a cell.
Is there a way to have an equivalent feature in Google Colab?
Autosave is already implemented in Google Colab, but there is a certain delay between the moment you execute a cell and when the save occurs.
You can try this yourself by going into File>Revision History, executing a cell, and waiting for the list to refresh.
That being said, I have also experienced loss of data in the past, which I can't explain. It might be a glitch.
As a good practice, I try to save every time I remember.
Good luck.
Autosave every 60 seconds by running this "magic command" into a new code cell :
%autosave 60
Colab will confirm it when you run the cell with printing : "Autosave changes every 60 seconds"
To display the list of all magic commands you can use the command :
%lsmagic
Additionally, you can call the Quick Reference Guide, describing all the magic commands and what they do using the command :
%quickref
Enjoy!
I am trying to visualize a training session I trained in a remote server. I used scp to copy the file in my local iMac. I tried to visualize the data by running tensorboard. It runs the tensorboard site but I can't get the visualization. Every chart has a single dot at zero. I get this warning on the terminal.
WARNING:tensorflow:Unable to get first event timestamp for run
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run
470_313_0.0001_2500_200/val
WARNING:tensorflow:Unable to get first event timestamp for run
470_313_0.0001_2500_50/train
WARNING:tensorflow:Unable to get first event timestamp for run
470_313_0.0001_2500_50/val
Any idea what is going on?
I ran into the same problem (with a TensorFlow 1.4 trainer running in the cloud with Google Cloud ML Engine).
Explicitly closing tf.summary.FileWriters with close() solved it in my case.
I ran into a similar problem. There are 2 solutions to it -
Delete all past tfevents files from the directory and keep the latest one (temporary solution).
Create a new directory for building your logs (permanent solution).
In given below picture, first I build logs in directory 2 which resulted in same error/warnings. Later I changed the directory to 3 and build logs there and tensorboard ran successfully.
In my case, the problem was the directory names for the runs were too long. After I manually renamed them, the problem is solved.