I am running an existing .ipynb which uses TensorFlow and Python3 of course. I am on Colab Pro and set the runtime to "GPU" with High RAM. As soon as I run the notebook I get the error:
Unrecognized runtime "tensorflow"; defaulting to "python3" Notebook settings
What is this, please?
You're likely using a notebook that was not authored in Colab after uploading it to Python.
That notebook requests an environment named "tensorflow", which was likely sensible to the original author. But, that's not the name of any environment in Colab, which selected a Python 3 runtime by default.
You can safely ignore this message. Or, copying or saving the notebook should update the JSON to name the Colab environment used to save it, thereby suppressing the message in the future.
i have a dataset of 30GB which I need to upload to google colab. What is the process to upload it?
It depends on what do you mean by "Have a 30GB dataset". If this dataset is on your local machine, then you need to:
Upload your dataset to Google Drive first
Then mount your Google Drive to your colab-notebook.
If you have the dataset on a server online, then you need to:
Mount your google drive to your notebook
Then, download it to your google drive directly
You can use this cod to mount your google-drive to your notebook:
import os
from google.colab import drive
drive.mount('/content/gdrive')
ROOT = "/content/gdrive/My Drive/"
os.chdir(ROOT)
If your data is on a server, then you can download it directly by running the following code in a notebook cell.
!wget [dataset_url]
If your problem is not enough space, you can change to a GPU runtime to get 350 GB space.
MENU > Runtime > Change runtime type > Hardware accelerator = GPU
The process is the same as #Anwarvic 's answer.
you can get more space by changing GPU to TPU from
MENU > Runtime > Change runtime type > Hardware accelerator = TPU
I want to use Pycharm on my own laptop to connect to our linux cluster and use tensorflow-gpu on it.
However, it says:
ImportError: libcuda.so.1: cannot open shared object file: No such
file or directory.
When I use terminal to connect to cluster and use tensorflow GPU through terminal,there's no problem.
However ,when I use python remote interpreter in Pycharm,the error happens when importing tensorflow-gpu :
ImportError: libcuda.so.1: cannot open shared object file: No such
file or directory
Failed to load the native TensorFlow runtime.
The location 'libcuda.so.1' on the cluster is
'/usr/lib64/nvidia/libcuda.so.1'
and
'/usr/lib64/libcuda.so.1'.
I tried to add them as LD_LIBRARY_PATH to Environment variables in Pycharm run configuration :
LD_LIBRARY_PATH=/usr/lib64/libcuda.so.1\;/usr/lib64/nvidia/libcuda.so.1
but it doesn't work.
I can use other packages like numpy,sklearn normally.
What's more,the corresponding version of my Tensorflow GPU is CUDA 9.0.If the error is about CUDA,it should be like can not find libcuda.so.9,however it shows libcuda.so.1.
I can also use tensorflow-GPU through terminal and ssh well,so I think the problem might be from Pycharm settings?
What do I need to do about Pycharm settings apart from adding LD_LIBRARY_PATH to Environment variables?
I mounted to Google drive account A. Now I want to switch to account B but I cannot do that, because there is no way for me to enter a new authentication key when executing drive.mount().
What I have tried and failed:
restart browser, restart computer
use force_remount=True in drive.mount(), it will only automatically remount account A. Not asking me for new mounting target.
change account A password
change run-time type from GPU to None and back to GPU
open everything in incognito mode
sign out all google accounts
How can I:
forget previous authentication key so it will ask me for a new one?
dismount drive and forget previous authentication key?
I have found the 'Restart runtime...' not to work, and changing permission too much of a hassle.
Luckily, the drive module is equipped with just the function you need:
from google.colab import drive
drive.flush_and_unmount()
You can reset your Colab backend by selecting the 'Reset all runtimes...' item from the Runtime menu.
Be aware, however, that this will discard your current backend.
Another solution for your problem could be to terminate your session and run your code (drive.mount()) again.
Steps:
1) Press "Additional connection options" button. Is the little sign button next to RAM and DISK
2) Select "Manage sessions"
3) Press the "Terminate" button
4) Run again your code (drive.mount()).
Now you will be asked to put your new key.
To force Colab to ask for a new key without waiting or resetting the runtime you can revoke the previous key.
To do this:
go to https://myaccount.google.com/permissions (or manually navigate to Security → Manage third-party access on your Google account page),
on the top right, select your profile image or initial, and then select the account whose drive you want do disconnect from Colab,
select Google Drive File Stream in the Google apps section, then select Remove access.
Executing drive.mount() will now ask for a new key.
Remount does not work when you have recently mounted and unmounted using flush_and_unmount(). The correct steps you should follow is (which worked for me at the time of posting):
After mounting using:
from google.colab import drive
drive.mount('/content/drive')
Unmount using: drive.flush_and_unmount() and you cannot see the 'drive/' folder but TRUST me you should use !rm -rf /content/drive before remounting the drive using:
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
And you will again get the authorise request for a new Gmail account.
You can terminate the session in Runtime -> manage session. That should do the work and you can remount the drive again.
Restarting runtimes and removing access did not help. I discovered that the notebook I was using created directories on the mountpoint:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
I had to first remove the subdirectories on the mountpoint. First ensure that your drive is not actually mounted!
!find /content/drive
/content/drive
/content/drive/My Drive
/content/drive/My Drive/Colab Notebooks
/content/drive/My Drive/Colab Notebooks/assignment4
/content/drive/My Drive/Colab Notebooks/assignment4/output_dir
/content/drive/My Drive/Colab Notebooks/assignment4/output_dir/2020-04-05_16:17:15
The files and directories above was accidentally created by the notebook before I've mounted the drive. Once you are sure (are you sure?) your drive is not mounted then delete the subdirectories.
!rm -rf /content/drive
After this, I was able to mount the drive.
Here is an explanation from their FAQs.
Why do Drive operations sometimes fail due to quota?
Google Drive enforces various limits, including per-user and per-file operation count and bandwidth quotas. Exceeding these limits will trigger Input/output error as above, and show a notification in the Colab UI. A typical cause is accessing a popular shared file, or accessing too many distinct files too quickly. Workarounds include:
Copy the file using drive.google.com and don't share it widely so that other users don't use up its limits.
Avoid making many small I/O reads, instead opting to copy data from Drive to the Colab VM in an archive format (e.g. .zip or.tar.gz files) and unarchive the data locally on the VM instead of in the mounted Drive directory.
Wait a day for quota limits to reset.
https://research.google.com/colaboratory/faq.html#drive-quota
Symptom
/content/drive gets auto-mounted without mounting it and not being asked for Enter your authorization code:.
Cached old state of the drive kept showing up.
The actual Google drive content did not show up.
Terminating, restarting, factory resetting revoking permissions, clear chrome cache did not work.
Flush and unmount google.colab.drive.flush_and_unmount() did not work.
Solution
Create a dummy file inside the mount point /content/drive.
Take a moment and make sure the content /content/drive is not the same with that in the Google drive UI.
Run rm -rf /content/drive.
Run google.colab.drive.flush_and_unmount()
From the menu Runtime -> Factory reset runtime.
Then re-run google.colab.drive.mount('/content/drive', force_remount=True) finally asked for Enter your authorization code.
The current code for the drive.mount() function is found at https://github.com/googlecolab/colabtools/blob/fe964e0e046c12394bae732eaaeda478bc5fa350/google/colab/drive.py
It is a wrapper for the drive executable found at /opt/google/drive/drive. I have found that executable accepts a flag authorize_new_user which can be used to force a reauthentication.
Copy and paste the contents of the drive.py file into your notebook. Then modify the call to d.sendline() currently on line 189 to look like this (note the addition of the authorize_new_user flag):
d.sendline(
('cat {fifo} | head -1 | ( {d}/drive '
'--features=max_parallel_push_task_instances:10,'
'max_operation_batch_size:15,opendir_timeout_ms:{timeout_ms},'
'virtual_folders:true '
'--authorize_new_user=True '
'--inet_family=' + inet_family + ' ' + metadata_auth_arg +
'--preferences=trusted_root_certs_file_path:'
'{d}/roots.pem,mount_point_path:{mnt} --console_auth 2>&1 '
'| grep --line-buffered -E "{oauth_prompt}|{problem_and_stopped}"; '
'echo "{drive_exited}"; ) &').format(
d=drive_dir,
timeout_ms=timeout_ms,
mnt=mountpoint,
fifo=fifo,
oauth_prompt=oauth_prompt,
problem_and_stopped=problem_and_stopped,
drive_exited=drive_exited))
Call either the drive module version of flush_and_unmount() or the one you pasted in, and then call your version of mount() to login as a different user!
I am trying to install Windows 8 on virtual box using a bootable WinPE DVD. The image file (.wim) is on the USB stick.
But when I boot from DVD, I cannot find the USB drive.
I need to execute the command
d:\imagex.exe /apply e:\imageC_20150528.wim 1 c:\
where D - dvd drive and say E - usb drive
But I cannot locate the USB drive from the command prompt.
When I use winPE I use DiskPart to determine the drive letter of the USB drive.
-Diskpart
-list volume
This should display all available drives
If the USB drive was inserted before you booted the system into PE, then the USB drive should already be mounted and have a drive letter. If it doesn't then it is likely the drive isn't supported (you might want to change your boot-image/disk to include additional drivers/tools to get access to your usb controller and connected device).
If your USB drive was inserted after you booted the system into PE (and your hardware is supported in PE) then I usually can access the disk in the following manner (worked even in XP PE):
go to commandline and start diskpart (following commands are in diskpart):
issue command: rescan enter
This 'Locates new disks that might have been added to the computer.'
Depending on diskpart version (and/or it's automount setting) this can be enough to automatically mount the file system for a new basic volume when it adds the drive to the system and assigns a drive letter to the volume.
When it doesn't, follow the following 2 steps:
command: list disk enter
to see a list of disks and some info (to check the drive was found)
command: list volume enter
lists volumes on all disks (to find volume number of the USB drive)
command: select volume=# enter (where #= volume number)
to select the specified volume and shift the focus to it
command: assign enter
to assign the next free letter to the current focused volume
OR: assign letter=# enter (where #= driveletter of choice)
This method usually works for me on most hardware.
See also: Diskpart commandline options on MS Technet
Really - you should be using a fairly current version of winPE - 5.1 is what I'm using - which is the version that parallels windows 8.1. I'm switching over to winPE 10 (they changed the numbering to be consistent with Windows)
The reason I bring this us is because you're using imagex - and dism.exe has largely replaced imagex.exe in more current versions of winPE.
If you're just doing this at the command prompt in a generic winPE, you can use mountvol.exe (with no parameters) to discover the drive letters that are available. Everything shifts around drive-letter-wise when you're in winPE. So it might not be e: any more.
Also, dism and/or imagex would be on the x: drive - which is the virtual drive that the winPE creates when it boots.