Before we could run a script this way
script = 'script.py'
!python '$script'
However, recently, this error occurs
python3: can't open file '$script': [Errno 2] No such file or directory
Is there an updated way to run a script with python variables in colab?
You can try this instead.
!python {script}
Somehow,
!python '$script'
Is working again.
Related
I want to run M-file (Octave) in Colab. first, I upload all file in Google Drive with the following address:
/content/drive/MyDrive/Colab/Corroded/hybrid/FAEICA_ANN/
Then I used the following code to run my file:
!apt install octave
from google.colab import drive
drive.mount('/content/drive')
import sys
import os
py_file_location = "/content/drive/MyDrive/Colab/Corroded/hybrid/FAEICA_ANN"
sys.path.append(os.path.abspath(py_file_location))
!octave -W MainANN_FAEICA.m
But I got:
error: no such file, '/content/MainANN_FAEICA.m'
error: source: error sourcing file '/content/MainANN_FAEICA.m'
Although I added the path, Colab cannot recognize the file.
You've added the directory to Python's system search path, not to Octave's search path.
The simplest way to run this file is to change the working directory to where the file is, then run the file:
os.chdir(py_file_location)
!octave -W MainANN_FAEICA.m
I tried to train the custom object according to https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects and I got an error.
Please tell me any good solutions.
I am going with google colaboratory.
I changed the directory, but it does not change.
dir
--content/
┠darknet-master/
┠build/
┠darknet/
┠x64/
┠data/
┠obj.data
┠
%%bash
cd /content/darknet-master./darknet detector train data/obj.data yolo-obj.cfg darknet53.conv.74 > train_log.txt
Couldn't open file: data/obj.dat
%cd /content/darknet-master/build/darknet/x64
!./darknet detect cfg/yolov3.cfg yolov3.weights data/person.jpg
/content/darknet-master/build/darknet/x64
/bin/bash: ./darknet: No such file or directory
Use %cd or os.chdir rather than %%bash cd...
The reason is that %%bash run commands in a sub-shell. But, I believe what you want to do is to change the working directory of the Python backend running your code.
We've updated the Spark version from 2.2 to 2.3, but admins didn't update the pandas. So our jobs fail with the following error:
ImportError: Pandas >= 0.19.2 must be installed; however, your version was 0.18.1
Our admin team suggested to created a VM downloading latest version from anaconda (using the command conda create -n myenv anaconda).
I did that and after activating the local environment using source activate myenv when I logged into pyspark2 then I found it was picking the new version of pandas.
But when I am submitting a job using spark2-submit command then it is not working. I did added the below configuration in the spark2-submit command
--conf spark.pyspark.virtualenv.enabled=true
--conf spark.pyspark.virtualenv.type=conda
--conf spark.pyspark.virtualenv.requirements=/home/<user>/.conda/requirements_conda.txt --conf spark.pyspark.virtualenv.bin.path=/home/<user>/.conda/envs/myenv/bin
Also I did zipped whole python 2.7 folder and passed that in the --py-files option along with other .py files --py-files /home/<user>/python.zip, but still getting the same version issue for pandas.
I tried to follow the instruction specified in the URL https://community.hortonworks.com/articles/104947/using-virtualenv-with-pyspark.html , but still no luck yet.
How to fix it and be able to spark2-submit with the proper pandas?
I think you may need to define environment variables such as SPARK_HOME and PYTHONPAH pointing to corresponding locations in your virtualenv.
export SPARK_HOME=path_to_spark_in_virtualenv
export PYTHONPATH=$SPARK_HOME/python
So I'm trying to git clone and cd into that directory using Google collab - but I cant cd into it. What am I doing wrong?
!rm -rf SwitchFrequencyAnalysis && git clone https://github.com/ACECentre/SwitchFrequencyAnalysis.git
!cd SwitchFrequencyAnalysis
!ls
datalab/ SwitchFrequencyAnalysis/
You would expect it to output the directory contents of SwitchFrequencyAnalysis - but instead its the root. I'm feeling I'm missing something obvious - Is it something to do with being within the python interpreter? (where is the documentation??)
Demo here.
use
%cd SwitchFrequencyAnalysis
to change the current working directory for the notebook environment (and not just the subshell that runs your ! command).
you can confirm it worked with the pwd command like this:
!pwd
further information about jupyter / ipython magics:
http://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd
As others have pointed out, the cd command needs to start with a percentage sign:
%cd SwitchFrequencyAnalysis
Difference between % and !
Google Colab seems to inherit these syntaxes from Jupyter (which inherits them from IPython).
Jake VanderPlas explains this IPython behaviour here. You can see the excerpt below.
If you play with IPython's shell commands for a while, you might
notice that you cannot use !cd to navigate the filesystem:
In [11]: !pwd
/home/jake/projects/myproject
In [12]: !cd ..
In [13]: !pwd
/home/jake/projects/myproject
The reason is that
shell commands in the notebook are executed in a temporary subshell.
If you'd like to change the working directory in a more enduring way,
you can use the %cd magic command:
In [14]: %cd ..
/home/jake/projects
Another way to look at this: you need % because changing directory is relevant to the environment of the current notebook but not to the entire server runtime.
In general, use ! if the command is one that's okay to run in a separate shell. Use % if the command needs to be run on the specific notebook.
Use os.chdir. Here's a full example:
https://colab.research.google.com/notebook#fileId=1CSPBdmY0TxU038aKscL8YJ3ELgCiGGju
Compactly:
!mkdir abc
!echo "file" > abc/123.txt
import os
os.chdir('abc')
# Now the directory 'abc' is the current working directory.
# and will show 123.txt.
!ls
If you want to use the cd or ls functions , you need proper identifiers before the function names ( % and ! respectively)
use %cd and !ls to navigate
.
!ls # to find the directory you're in ,
%cd ./samplefolder #if you wanna go into a folder (say samplefolder)
or if you wanna go out of the current folder
%cd ../
and then navigate to the required folder/file accordingly
!pwd
import os
os.chdir('/content/drive/My Drive/Colab Notebooks/Data')
!pwd
view this answer for detailed explaination
https://stackoverflow.com/a/61636734/11535267
I believe you'd have to mount the Google Drive first before you do anything else.
from google.colab import drive
drive.mount('/content/drive')
In windows power shell ,I can run scrapy shell 'http://www.hao123.com',
scrapy shell 'http://www.hao123.com
I can run ipython
I can run ipython but not scrapy shell 'http://www.hao123.com'
ipython then scrapy shell 'http://www.hao123.com
In ipython notebook,I can't run scrapy shell 'http://www.hao123.com'also
scrapy shell 'http://www.hao123.com'
File "<ipython-input-3-be4048c8f90b>", line 1
scrapy shell 'http://www.hao123.com'
^
SyntaxError: invalid syntax
Ipython is installed by anaconda,scrapy is installed by pip,anaconda and pip is in different file.
Please help me!
That's not a feature you can have in ipython. scrapy shell is a command, it's own application completely separate from ipython.
However, there are two things you can do:
If you have a Spider and Response objects from somewhere you can simply use scrapy.shell.inspect_repsonse
from scrapy.shell import inspect_response
# You need a scrapy spider object that has crawler instance attached to it
some_spider = Spider()
# you need response object
some_response = Response()
inspect_response(some_spider, some_response)
# new ipython shell will be embedded, like one from scrapy shell command
Otherwise you can spawn a subprocess:
import subprocess
subprocess.call('scrapy shell http://stackoverflow.com', shell=True)
# new ipython shell will be embedded, like one from scrapy shell command
if you are using Anaconda on windows 7 .
Follow: environment --> root --> Open terminal. Then you can write:
scrapy shell 'http://www.hao123.com'