bug (resolved) with "Mount drive" web-button in colab. Accessing "shared with me" files from google colab (y2020, previous solutions seem to fail) - google-colaboratory

[ Newer Edit]: colab team reported that they corrected the issue on May 27 2020.
I have checked - it works Okay for me now.
Link to issue: https://github.com/googlecolab/colabtools/issues/1205
==================================================================
[New Edit:] It became clear that problem below arises ONLY if mount the google drive to colab by via web interface button "Mount Drive"
and does NOT appear if mount by command line way.
So seems web way is bugged. See details in my own answer below.
It is checked for "Chrome" browser.
==================================================================
[Original question:]
How to access "shared with me" from google colab ? (Interface seems changed now (2020) and previously described solutions does not seem to work).
More details:
The question has been asked several times, and
the solutions described e.g. here : https://stackoverflow.com/a/53887376/625396
The problem that I do not see "Add to My Drive" , but see "Add shortcut to Drive".
After doing it, we can see that via web-interface for google drive, that shortcut indeed appears.
BUT that shortcut canNOT be seen via colab utilities, like
os.listdir() !
So shortcut seems to be invisible for colab, and not clear how to access it.
Below are the screenshot, showing that colab does not see the shortcut to "shared with me"-"cytotrace_datasets", but web-gui of google drive can see.
Here is screenshot what I see by colab (shortcut canNOT be seen):
Here is screenshot what I see by web-gui of google drive (shortcut can be seen):

Brief: do NOT mount google drive by web-interface button "Mount drive" (it is bugged), but do it in the "old" command line way, and you will not have problems.
Details:
After getting excellent answer above and playing with it,
it seems I found some strange thing which results in simpler solution and probably indicates that there is currently a bug with mounting the google drive by the web interface button "Mount drive".
I mean do NOT mount the drive by interface:
But do it in the old way:
and that is all - you will get the access to files which added before with the help of
"Add shortcut to Drive":

Suppose you want to read a shared csv file from drive. You have done "Add shortcut to Drive".
1) At Colab Notebook Connect to your drive.
# Import PyDrive and associated libraries.
# This only needs to be done once per notebook.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
2) Get the id of shared file you want to access.
Open file
-> go to linksharing [https://drive.google.com/open?id=1JKECh3GNry6xbAK6aBSzQtSntD4GTEl ] -> copy the the string after 'id='
3) back to colab
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
file_id = '1JKECh3GNry6xbAK6aBSzQtSntMD4GTEl'
downloaded = drive.CreateFile({'id': file_id}) #important
print(downloaded['title']) # it should print the title of desired file
downloaded.GetContentFile('file.csv')
#Finally, you can read the file as pandas dataframe.
import pandas as pd
df= pd.read_csv('file.csv')
Note : This is my first ever answer to a stack overflow question

Related

Google Colab - Not able to import API key file

I have a code in Google Colab which uses a Python package called Atlite, which in turn retrieves data from the Climate Data Store (CDS) through the use of an API key.
When running this code in Python I just need to have the file containing the key saved in a specific folder and then the code runs perfectly fine.
When I try to run the code in Google Colab the following error arises:
**Exception: Missing/incomplete configuration file: /root/.cdsapirc
**
I have the file ".cdsapirc" in my computer but when I try to import it to the "/root" folder in Google Colab it just does not get imported. I can import a .py file, but when I try to import the ".cdsapirc" file (which is basically a txt file) it does not work.
Could someone please help me to solve this issue?
Thank you!
Regards,
Sebastian
If uploading the .cdsapirc file doesn't work, you could try creating it inside Google Colab using a simple Python script:
uid = "<your uid>"
apikey = "<your api-key"
with open("/root/.cdsapirc", "w") as f:
print("url: https://cds.climate.copernicus.eu/api/v2", file=f)
print(f"key: {uid}:{apikey}", file=f)
You can get the uid and apikey either from CDS after logging in or you open you local .cdsapirc file and look them up there, see here for more information.
There might be a nicer solution by someone more familiar with Google Colab though.

upload file to another Drive Google colab

I created a script generating me a pdf file on a daily basis. It is then sent to the user's drive using a simple drive.mount('/content/drive')
In order to ensure the use of the script, I would like all the pdf files created to be sent to a specific file stored on a specific drive.
Is there a way to specify this path and this drive account after having sent it to the base drive?
I found very few topics going in this direction and not being very clear on the possibility of uploading files to another drive.
If you have any ideas, I'm all ears!
Good day to you all !
I finally found a solution through the PyDrive library.
Here is an example for the curious little ones ;)
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
gfile = drive.CreateFile({'parents': [{'id': 'ID folder'}]})
gfile.SetContentFile("file path")
gfile.Upload()
Once the imports and the creation of the gauth variable have been executed, you will be asked for permissions on your account (don't be surprised)
Hope this helps anyone in need !

TypeError: 'module' object is not callable in Google Colab

I am getting the error when I run this code in google Colab
folders = glob('/content/drive/MyDrive/Data/train/*')
[![enter image description here]
You tried to use glob method directly, take the following code sample which was tested in Google colab environment. It should help to fix your issue.
from google.colab import drive
drive.mount('/gdrive')
import glob
path = glob.glob("/gdrive/MyDrive/*")
for file_or_folder in path:
print(file_or_folder)
For this issue, the path should be defined differently.
For that, initially import glob and drive module as mentioned below
from google.colab import drive
drive.mount('/gdrive')
import glob
And then using this, you can easily access the file as follows
folders = glob.glob('/gdrive/MyDrive/Datasets/Tomato/train/*')
Here if you see the difference in the copied path and the defined path is
'/content/drive/MyDrive/Datasets/Tomato/test' is what coppid form the drive
'/gdrive/MyDrive/Datasets/Tomato/train/*' is how you need to define the path

WebKit2.WebView (and Gnome Browser) doesn't play YouTube videos

Inside a Python GTK+ 3 application I have a simple WebKit2.WebView (no customization) to display a set html string (no live browsing). The html contains a javascript lightbox with some youtube videos. When I open them, the youtube video box shows up but the content doesn't play and I get a black screen saying ("An error occurred. Please try again later. (Playback ID: ...").
Similar image found on the web: https://i.redd.it/gwqswatndiy11.png
I then tried browsing the YouTube website with the Gnome Web browser (v 3.18.11 on Linux Mint 18.3) [based on WebKit] and the same problem occured with every video.
Of course everything works with other browsers (Chromium and Firefox).
Moreover, everything works if I use WebKit1 instead of WebKit2 in my PyGtk+ 3 app.
So I think the problem lies in the Webkit2 options configuration.
Does anyone know how to solve this problem?
Here is an extract of my code. As you can see, I have already tried to set some options but nothing worked.
[Side note: browser_settings.set_enable_media(True) gives me the error AttributeError: 'Settings' object has no attribute 'set_enable_media'; I don't understand why since in the documentation the method exists]
import gi
gi.require_version('Gtk', '3.0')
gi.require_version('GtkSource', '3.0')
gi.require_version('WebKit2', '4.0')
from gi.repository import Gtk, GtkSource, GLib, WebKit2
self.webview = WebKit2.WebView.new()
browser_settings = self.webview.get_settings()
browser_settings.set_property('javascript-can-access-clipboard', bool(True))
# browser_context = self.webview.get_context()
# cookie_manager = browser_context.get_cookie_manager()
# cookie_manager.set_accept_policy(WebKit2.CookieAcceptPolicy.ALWAYS)
# browser_settings.set_enable_xss_auditor(False)
# browser_settings.set_allow_universal_access_from_file_urls(True)
# browser_settings.set_allow_file_access_from_file_urls(True)
# browser_settings.set_enable_media(True)
# browser_settings.set_enable_media_capabilities(True)
# browser_settings.set_enable_media_stream(True)
# browser_settings.set_enable_mediasource(True)
# browser_settings.set_media_playback_allows_inline(True)
self.webview.set_settings(browser_settings)
I also couldn't initially play Youtube videos on Gnome Web (3.38.1) on Ubuntu Budgie 20.10. But I was able to make it work.
I checked Software, searched "gstreamer" and I see that most GStreamer codecs were already installed.
I was able to make Youtube videos work by installing "ffmpeg plugin for GStreamer" and "GStreamer WPEWebKit plugin".

file download from google drive to colaboratory

I was trying to download file from my google drive to colaboratory.
file_id = '1uBtlaggVyWshwcyP6kEI-y_W3P8D26sz'
import io
from googleapiclient.http import MediaIoBaseDownload
request = drive_service.files().get_media(fileId=file_id)
downloaded = io.BytesIO()
downloader = MediaIoBaseDownload(downloaded, request)
done = False
while done is False:
# _ is a placeholder for a progress object that we ignore.
# (Our file is small, so we skip reporting progress.)
_, done = downloader.next_chunk()
downloaded.seek(0)
print('Downloaded file contents are: {}'.format(downloaded.read()))
doing so am getting this error:
NameError: name 'drive_service' is not defined
How to remove this error?
No installing/importing any library. Just put your file id at the end.
!gdown yourFileIdHere
Note: at the time of writing gdown library is preinstalled on Colab.
the easiest method to download a file from google drive to colabo notebook is via the colabo api:
from google.colab import drive
drive.mount('/content/gdrive')
!cp '/content/gdrive/My Drive/<file_path_on_google_drive>' <filename_in_colabo>
Remarks:
By the drive.mount(), you can access any file on your google drive.
'My Drive' is equivalent to 'Google Drive' on your local file system.
The file_path is surrounded with single quotes as the standard directory below the mount point ('My Drive') has a space, and you might also have spaces in your path elsewhere anyway.
Very useful to locate your file and get the file path is the file browser (activated by click on left arrow). It lets you click through your mounted folder structure and copy the file path, see image below.
Here's an easy way to get by. You may either use wget command or requests module in Python to get the job done.
# find the share link of the file/folder on Google Drive
file_share_link = "https://drive.google.com/open?id=0B_URf9ZWjAW7SC11Xzc4R2d0N2c"
# extract the ID of the file
file_id = file_share_link[file_share_link.find("=") + 1:]
# append the id to this REST command
file_download_link = "https://docs.google.com/uc?export=download&id=" + file_id
The string in file_download_link can be pasted in the browser address bar to get the download dialog box directly.
If you use the wget command:
!wget -O ebook.pdf --no-check-certificate "$file_download_link"
Step 1
!pip install -U -q PyDrive
Step 2
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
Step3
file_id = '17Cp4ZxCYGzWNypZo1WPiIz20x06xgPAt' # URL id.
downloaded = drive.CreateFile({'id': file_id})
downloaded.GetContentFile('shaurya.txt')
Step4
!ls #to verify content
OR
import os
print(os.listdir())
You need to define a drive API service client to interact with the Google drive API, for instance:
from googleapiclient.discovery import build
drive_service = build('drive', 'v3')
(see the notebook External data: Drive, Sheets, and Cloud Storage/Drive REST API)
I recommend you use Pydrive to download your file from google drive. I download 500MB dataset for 5s.
1. install Pydrive
!pip install PyDrive
2. OAouth
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)
3. code for download file from google drive
fileId = drive.CreateFile({'id': 'DRIVE_FILE_ID'}) #DRIVE_FILE_ID is file id example: 1iytA1n2z4go3uVCwE_vIKouTKyIDjEq
print fileId['title'] # UMNIST.zip
fileId.GetContentFile('UMNIST.zip') # Save Drive file as a local file
Cheer Mo Jihad
The --id argument has been deprecated so now you simply have to run:
! gdown 1uBtlaggVyWshwcyP6kEI-y_W3P8D26sz
If your file is stored in a variable you can run:
! gdown $my_file_id
You can also use my implementations on google.colab and PyDrive at https://github.com/ruelj2/Google_drive which makes it a lot easier.
!pip install - U - q PyDrive
import os
os.chdir('/content/')
!git clone https://github.com/ruelj2/Google_drive.git
from Google_drive.handle import Google_drive
Gd = Google_drive()
Gd.load_file(local_dir, file_ID)
You can simply copy all your google drive files and folders inside google colab and use it directly using this command
# import drive
from google.colab import drive
drive.mount('/content/drive')
This will ask you for permission after accepting you will have all your google drive inside your colab
If you want to use any file just copy the path