how to make directory on the google colab?
I can not make the new directory on google driver on google colab?
If you are looking for creating new folder in the 'Files' section (to the left from notebook) - you can run in code shell:
!mkdir new_folder
Just remember, that Colab is a temporary environment with an idle timeout of 90 minutes and an absolute timeout of 12 hours.
You can use !mkdir foldername command to create folder in the current working directory. It will be by default /content.
If you want to create directory with specific condition then use os.mkdir('folder_path') to create the directory.
import os.path
from os import path
if path.exists('/content/malayalam_keyphrase_extraction') == False:
os.mkdir('/content/malayalam_keyphrase_extraction')
os.chdir('/content/malayalam_keyphrase_extraction')
!pwd
!ls
or
!mkdir malayalam_keyphrase_extraction
NB: malayalam_keyphrase_extraction is the folder name
from google.colab import drive
drive.mount('/content/drive')
mkdir "/content/drive/My Drive/name"
The way you want to do this is as follows:
Mount your drive (i.e. Connect your Colab to your Google Drive):
from google.colab import drive
drive.mount('/content/gdrive')
Use the os library. It has most of your typical bash commands which you can run right from your python script:
import os
path = "gdrive/MyDrive/Rest/Of/Path"
os.mkdir(path)
Do like this, then see the result
You can use feature upload file(s) in GUI.
Related
I'm training a Convolutional Neural Network using Google-Colaboratory. I have my data (images) stored in Google Drive and I'm able to use it correctly. However sometimes the process to read the images is too slow and does not work (other times the process is faster and I have no problem reading the images). In order to read the images from Google Drive I use:
from google.colab import drive
drive.mount('/content/drive')
!unzip -u "/content/drive/My Drive/the folder/files.zip"
IMAGE_PATH = '/content/drive/My Drive/the folder'
file_paths = glob.glob(path.join(IMAGE_PATH, '*.png'))
and sometimes works and other times not or it is too slow :).
Either way I would like to read my data from a folder on my desktop without using google drive but I'm not able to do this.
I'm trying the following:
IMAGE_PATH = 'C:/Users/path/to/my/folder'
file_paths = glob.glob(path.join(IMAGE_PATH, '*.png'))
But I get an error saying that the directory/file does not exist.
Google Colab cannot directly access our local machine dataset because it runs on a separate virtual machine on the cloud. We need to upload the dataset into Google Drive then we can load it into Google Colab’s runtime for model building.
For that you need to follow the steps given below:
Create a zip file of your large dataset and then upload this file in your Google Drive.
Now, open the Google Colab with the same google id and mount the Google Drive using the below code and authorize to access the drive:
from google.colab import drive
drive.mount('/content/drive')
Your uploaded zip file will be available in the Google Colab mounted drive /drive/MyDrive/ in left pane.
To read the dataset into the Google Colab, you need to unzip the folder and extract its contents into the /tmp folder using the code below.
import zipfile
import os
zip_ref = zipfile.ZipFile('/content/drive/MyDrive/train.zip', 'r') #Opens the zip file in read mode
zip_ref.extractall('/tmp') #Extracts the files into the /tmp folder
zip_ref.close()
You can check the extracted file in /drive/train folder in left pane.
Now finally you need to join the path of your dataset to use it in the Google Colab's runtime environment.
train_dataset = os.path.join('/tmp/train/') # dataset
I am trying to set LD_LIBRARY_PATH in google code colab by following statements:
import os
os.environ['LD_LIBRARY_PATH']='/path/to/library/used/by/my/software'
I can see the environment variable added in the environment variables list, checked by (!printenv), but when my exe tries to access the library stored at LD_LIBRARY_PATH, it is not able to find it.
NotFoundError: library_name.so: cannot open shared object file: No such file or directory
I also tried configuring the environment variables through colab-env package (https://pypi.org/project/colab-env/), but I am facing same issue with this approach also.
Can someone give pointers? Thank you.
You are setting up only in your notebook environment.
To configure in linux environment, you can use this:
!export AWS_SHARED_CREDENTIALS_FILE=<PATH HERE>
But I usually do this:
import os
!export AWS_SHARED_CREDENTIALS_FILE=<PATH HERE>
os.environ['AWS_SHARED_CREDENTIALS_FILE'] = <PATH HERE>
Best solution with Colab is to first copy the executable file to Drive :
!cp /content/gdrive/My\ Drive/Colab\ Notebooks/<FILE> /usr/local/bin.
Then allow to execute it:
!chmod 755 /usr/local/bin/<FILE>.
Courtesy of Medium.
I am new to Google Colab. I have used the following code to download a data set from Kaggle:
!pip install kaggle
import os
os.environ['KAGGLE_USERNAME'] = "xxxxxxxxx"
os.environ['KAGGLE_KEY'] = "xxxxxxxxxxxxxxxxxxxxxxxxxx"
!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition
os.chdir('/content/')
#dowloaded data in this directory
I can access all the data now, but where is this data stored? In my Google Drive? But I can't find it. What if I want to access this same data from a different notebook running in Colab? Both notebooks are stored in my Google Drive, but seem to have their own different '/content/' folders.
to store the data in google drive you have to link the drive first using :
from google.colab import drive
drive.mount('/content/drive')
then you can access the drive by navigating through "/content/drive/My Drive/"
so you can download the data to your drive using the following command :
!kaggle competitions download -p "/content/drive/My Drive/" -c dogs-vs-cats-redux-kernels-edition
I was using google colab and using this code to access my google drive:
from google.colab import drive
drive.mount('/content/gdrive')
It works well but the authentification doesn't last long and I don't want to re-enter my credentials all the time. So I tried to use Pydrive to save my credentials to a file (using this answer):
!pip install pydrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
gauth = GoogleAuth()
gauth.LoadCredentialsFile("mycreds.txt")
gauth.Authorize()
drive = GoogleDrive(gauth)
but I can only write files remotely with this solution, and I would like to be able to mount my google drive entirely so that I can easily use unix commands. Is there a way to do that?
PyDrive doesn't create a FUSE mount, so doesn't work for you intended purpose.
Authentication of drive.mount() should last the lifetime of the assigned VM, and no option is going to outlast a VM's assignment, so I don't think what you want is possible today.
I am looking for answer for this too, and it appears that most easiest way to achieve it is that using colab pro version.
you can find more info here
at the moment, I work with 400+ images and upload them with
from google.colab import files
uploaded = files.upload()
This one's working fine but I have to reupload all the images every time I leave my colaboratory. Pretty annoying because the upload takes like 5-10 minutes.
Any possibilities to prevent this? It seems like Colaboratory is saving the files only temporarily.
I have to use Google Colaboratory because I need their GPU.
Thanks in advance :)
As far as I know, there is no way to permanently store data on a Google Colab VM, but there are faster ways to upload data on Colab than files.upload().
For example you can upload your images on Google Drive once and then 1) mount Google Drive directly in your VM or 2) use PyDrive to download your images on your VM. Both of these options should be way faster than uploading your images from your local drive.
Mounting Drive in your VM
Mount Google Drive:
from google.colab import drive
drive.mount('/gdrive')
Print the contents of foo.txt located in the root directory of Drive:
with open('/gdrive/foo.txt') as f:
for line in f:
print(line)
Using PyDrive
Take a look at the first answer to this question.
First Of All Mount Your Google Drive:
# Load the Drive helper and mount
from google.colab import drive
# This will prompt for authorization.
drive.mount('/content/drive')
Result is :
Mounted at /content/drive
For Checking Directory Mounted Run this command:
# After executing the cell above, Drive
# files will be present in "/content/drive/My Drive".
!ls "/content/drive/My Drive"
Result is Something Like This:
07_structured_data.ipynb Sample Excel file.xlsx
BigQuery recipes script.ipynb
Colab Notebooks TFGan tutorial in Colab.txt
Copy of nima colab.ipynb to_upload (1).ipynb
created.txt to_upload (2).ipynb
Exported DataFrame sheet.gsheet to_upload (3).ipynb
foo.txt to_upload.ipynb
Pickle + Drive FUSE example.ipynb variables.pickle
Sample Excel file.gsheet