I run command in bash:
ulimit
my_command
How to use ulimit ( or analog function ) in twisted ?
from twisted.internet import protocol, utils, reactor
def r():
utils.getProcessOutputAndValue('my_command')
reactor.callWhenRunning(r)
reactor.run()
The resource stdlib module.
Related
I want to use stockfish in my colab notebook.
I first tried with the chess.engine.SimpleEngine.popen_uci() command, which seems to be outdated, since module 'chess.engine' has no attribute 'SimpleEngine'
So I tried with stockfish itself:
from stockfish import Stockfish
stockfish = Stockfish('/usr/local/lib/python3.7/dist-packages/stockfish')
Here I always get the error:
Permission denied: '/usr/local/lib/python3.7/dist-packages/stockfish'
I googled for it and came up with some good tries:
I tried with !chmod +x '/usr/local/lib/python3.7/dist-packages/stockfish' which compiled, but didn't resolved the problem, and !chmod +x 'stockfish', which didn't find the folder.
So how exactly do I give permission to use the stockfish folder?
You need to set the os.setuid to 0 like this code below:
! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-7.9.2
import os
from subprocess import Popen, PIPE, STDOUT
es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
stdout=PIPE, stderr=STDOUT,
preexec_fn=lambda: os.setuid(0) # as daemon
)
# wait until ES has started
! sleep 30
I have a Centos 7 server with cPanel and I'm working on a Telegram bot for my business needs. The bot should be able to run a terminal command with os.system or subprocess.Popen, however both options do not work when configured through a webhook + wsgi process.
I tested both with bot.polling method and they worked as a charm, however after I switched to webhook method served by flask and wsgi, both stopped working for me. I have tried the following:
mycommand = "python3.6 GoReport.py --id 31-33 --format word"
os.chdir('dir_to_run_command_from')
os.system(mycommand)
and the following one:
mycommand = "python3.6 GoReport.py --id 31-33 --format word"
subprocess.Popen(mycommand, cwd="dir_to_run_command_from", shell=True)
Both options simply do nothing right now. I tried to print them both and received 0 as a response. I wonder if the issue is caused by permissions or something.
I expect both options to work through webhook + wsgi as good as they work through bot.polling method.
I think I got it wrong. Your script writes a report to a specific directory. You do not need a result in your application route.
I wrote a small test application called tryout. It runs in a virtual environment.
$ mkdir tryout
$ cd tryout
$ python3 -m venv tryout
$ source tryout/bin/activate
$ export FLASK_APP=tryout/app
$ export FLASK_ENV=development
$ flask run
Directory structure:
/tryout
/app/*
/bin/*
/include/*
/lib/*
/subdir/*
Application:
# /tryout/app/__init__.py
import sys, os
from flask import Flask
def create_app(env=os.getenv('FLASK_ENV', 'development')):
app = Flask(__name__)
#app.route('/run-script')
def run_script():
import subprocess
cmd = 'python script.py'
cwd = 'subdir'
ret = subprocess.check_output(cmd, cwd=cwd, shell=True)
print(ret)
return ret, 200
return app
app = create_app()
Script:
# /subdir/script.py
import os, sys
def main():
with open('report.txt', 'w+') as fp:
fp.write('Info\n')
sys.stdout.write('It works!')
if __name__ == '__main__':
main()
It works!
A new file named "report.log" is written into the "subdir"-directory.
In Browser appears "It works!".
Hope I could help you or I have no real idea of what you want to do.
If you want to run an external script from inside flask, you could use subprocess to run the script from the command line. This is the right solution.
#app.route('/run-script')
def run_script():
cmd = '<your command here!>'
result = subprocess.check_output(cmd, cwd='<your wordir>', shell=True)
return render_template('results.html', **locals())
Have fun!
#Bogdan Kozlowskyi
Is it possible to pipe on the command line? Do you need to return a result to the user?
cmd = 'first_cmd | tee report.log'
result = subprocess.check_output(cmd, cwd='<your wordir>', shell=True)
Perhaps you should look for shell commands like '>>', '>' and 'tee'.
Seems to be a user-groups permission problem (execute and write).
PROBLEM: I am attempting to run a spark-submit script from my local machine to a cluster of machines. The work done by the cluster uses numpy. I currently get the following error:
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: cannot import name multiarray
DETAIL:
In my local environment I have setup a virtualenv that includes numpy as well as a private repo I use in my project and other various libraries. I created a zip file (lib/libs.zip) from the site-packages directory at venv/lib/site-packages where 'venv' is my virtual environment. I ship this zip to the remote nodes. My shell script for performing the spark-submit looks like this:
$SPARK_HOME/bin/spark-submit \
--deploy-mode cluster \
--master yarn \
--conf spark.pyspark.virtualenv.enabled=true \
--conf spark.pyspark.virtualenv.type=native \
--conf spark.pyspark.virtualenv.requirements=${parent}/requirements.txt \
--conf spark.pyspark.virtualenv.bin.path=${parent}/venv \
--py-files "${parent}/lib/libs.zip" \
--num-executors 1 \
--executor-cores 2 \
--executor-memory 2G \
--driver-memory 2G \
$parent/src/features/pi.py
I also know that on the remote nodes there is a /usr/local/bin/python2.7 folder that includes a python 2.7 install.
so in my conf/spark-env.sh I have set the following:
export PYSPARK_PYTHON=/usr/local/bin/python2.7
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/python2.7
When I run the script I get the error above. If I screen print the installed_distributions I get a zero length list []. Also my private library imports correctly (which says to me it is actually accessing my libs.zip site-packages.). My pi.py file looks something like this:
from myprivatelibrary.bigData.spark import spark_context
spark = spark_context()
import numpy as np
spark.parallelize(range(1, 10)).map(lambda x: np.__version__).collect()
EXPECTATION/MY THOUGHTS:
I expect this to import numpy correctly especially since I know numpy works correctly in my local virtualenv. I suspect this is because I'm not actually using the version of python that is installed in my virtualenv on the remote node. My question is first, how do I fix this and second how do I use my virtualenv installed python on the remote nodes instead of the python that is just manually installed and currently sitting on those machines? I've seen some write-ups on this but frankly they are not well written.
With --conf spark.pyspark.{} and export PYSPARK_PYTHON=/usr/local/bin/python2.7 you set options for your local environment / your driver. To set options for the cluster (executors) use the following syntax:
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON
Furthermore, I guess you should make your virtualenv relocatable (this is experimental, however). <edit 20170908> This means that the virtualenv uses relative instead of absolute links. </edit>
What we did in such cases: we shipped an entire anaconda distribution over hdfs.
<edit 20170908>
If we are talking about different environments (MacOs vs. Linux, as mentioned in the comment below), you cannot just submit a virtualenv, at least not if your virtualenv contains packages with binaries (as is the case with numpy). In that case I suggest you create yourself a 'portable' anaconda, i.e. install Anaconda in a Linux VM and zip it.
Regarding --archives vs. --py-files:
--py-files adds python files/packages to the python path. From the spark-submit documentation:
For Python applications, simply pass a .py file in the place of instead of a JAR, and add Python .zip, .egg or .py files to the search path with --py-files.
--archives means these are extracted into the working directory of each executor (only yarn clusters).
However, a crystal-clear distinction is lacking, in my opinion - see for example this SO post.
In the given case, add the anaconda.zip via --archives, and your 'other python files' via --py-files.
</edit>
See also: Running Pyspark with Virtualenv, a blog post by Henning Kropp.
In windows power shell ,I can run scrapy shell 'http://www.hao123.com',
scrapy shell 'http://www.hao123.com
I can run ipython
I can run ipython but not scrapy shell 'http://www.hao123.com'
ipython then scrapy shell 'http://www.hao123.com
In ipython notebook,I can't run scrapy shell 'http://www.hao123.com'also
scrapy shell 'http://www.hao123.com'
File "<ipython-input-3-be4048c8f90b>", line 1
scrapy shell 'http://www.hao123.com'
^
SyntaxError: invalid syntax
Ipython is installed by anaconda,scrapy is installed by pip,anaconda and pip is in different file.
Please help me!
That's not a feature you can have in ipython. scrapy shell is a command, it's own application completely separate from ipython.
However, there are two things you can do:
If you have a Spider and Response objects from somewhere you can simply use scrapy.shell.inspect_repsonse
from scrapy.shell import inspect_response
# You need a scrapy spider object that has crawler instance attached to it
some_spider = Spider()
# you need response object
some_response = Response()
inspect_response(some_spider, some_response)
# new ipython shell will be embedded, like one from scrapy shell command
Otherwise you can spawn a subprocess:
import subprocess
subprocess.call('scrapy shell http://stackoverflow.com', shell=True)
# new ipython shell will be embedded, like one from scrapy shell command
if you are using Anaconda on windows 7 .
Follow: environment --> root --> Open terminal. Then you can write:
scrapy shell 'http://www.hao123.com'
Environment:
Mac OS X 10.8.5
Apache2 (version that came with the OS)
Python2 (version that came with the OS)
Python3 (installed via Homebrew)
This code returns "testing" in the web browser:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
from __future__ import print_function, division
print("Content-Type: text/html") # HTML is following.
print() # Blank line, end of headers.
print("testing")
But this code returns "Internal Server Error" in the web browser (using python3 this time):
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
print("Content-Type: text/html") # HTML is following.
print() # Blank line, end of headers.
print("testing")
...and in the Apache2 error log:
env: python3: No such file or directory
Premature end of script headers: test_cgi.py
In echo $PATH:
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
Looking deeper, ls -al /usr/bin/python*:
/usr/bin/python
/usr/bin/python-config
/usr/bin/python2.5 -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/bin/python2.5
/usr/bin/python2.5-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/bin/python2.5-config
/usr/bin/python2.6 -> ../../System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
/usr/bin/python2.6-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6-config
/usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/usr/bin/python2.7-config -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-config
/usr/bin/pythonw
/usr/bin/pythonw2.5 -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/bin/pythonw2.5
/usr/bin/pythonw2.6 -> ../../System/Library/Frameworks/Python.framework/Versions/2.6/bin/pythonw2.6
/usr/bin/pythonw2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/pythonw2.7
And in ls -al /usr/local/bin/python*:
/usr/local/bin/python3 -> ../Cellar/python3/3.3.3/bin/python3
/usr/local/bin/python3.3 -> ../Cellar/python3/3.3.3/bin/python3.3
/usr/local/bin/python3.3-config -> ../Cellar/python3/3.3.3/bin/python3.3-config
/usr/local/bin/pythonw3.3 -> ../Cellar/python3/3.3.3/bin/pythonw3.3
Questions:
Since the first item in my PATH is /usr/local/bin, why can't Apache find Python3 ?
How can I tell Apache to use Python3 ?
Thank you for your help :)
I have the exact same problem; I cannot figure out how to get apache to recognize python3. Furthermore, if you specify the interpreter manually:
#!/usr/bin/python
Works, however:
#!/usr/local/bin/python3
Complains about a "malformed header from script" in the apache error log... I have no idea why this is an issue as it should just be running an arbitrary interpreter.
EDIT
Okay, so my problem was actually just me not outputting header information. Your issue is that no python3 environment variable has been set. Try changing the first line to:
#!/usr/local/bin/python3