I'm creating a web-based interface for a number of different command line executables, and am using cherrypy behind apache (using mod_rewrite). I'm very new to this, and am having difficulty getting things configured properly. On my development machine, everything works reasonable well, but when I installed the code on a second machine I can't get anything to work properly.
The basic workflow for the applications is: 1. upload a dataset, 2. process the data (using python with some calls to executables using subprocess.call), 3. display the results on the web page.
After uploading and processing one dataset, everytime I attempt to process a second dataset the system stops responding. I'm not seeing any output in the terminal from the cherrypy process, or in the site log that shows any errors have occurred.
I'm starting cherrypy with the following conf file:
[global]
environment: 'production'
log.error_file: 'logs/site.log'
log.screen: True
tools.sessions.on: True
tools.session.storage_type: "file"
tools.session.storage_path: "sessions/"
tools.sessions.timeout: 60
tools.auth.on: True
tools.caching.on: False
server.socket_host: '0.0.0.0'
server.max_request_body_size: 0
server.socket_timeout: 60
server.thread_pool: 20
server.socket_queue_size: 10
engine.autoreload.on:True
My init.py file:
import cherrypy
import os
import string
from os.path import exists, join
from os import pathsep
from string import split
from mako.template import Template
from mako.lookup import TemplateLookup
from auth import AuthController, require, member_of, name_is
from twopoint import TwoPoint
current_dir = os.path.dirname(os.path.abspath(__file__))
lookup = TemplateLookup(directories=[current_dir + '/templates'])
def findInSubdirectory(filename, subdirectory=''):
if subdirectory:
path = subdirectory
else:
path = os.getcwd()
for root, dirs, names in os.walk(path):
if filename in names:
return os.path.join(root, filename)
return None
class Root:
#cherrypy.expose
#require()
def index(self):
tmpl = lookup.get_template("main.html")
return tmpl.render(usr=WebUtils.getUserName(),source="")
if __name__=='__main__':
conf_path = os.path.dirname(os.path.abspath(__file__))
conf_path = os.path.join(conf_path, "prod.conf")
cherrypy.config.update(conf_path)
cherrypy.config.update({'server.socket_host': '127.0.0.1',
'server.socket_port': 8080});
def nocache():
cherrypy.response.headers['Cache-Control']='no-cache,no-store,must-revalidate'
cherrypy.response.headers['Pragma']='no-cache'
cherrypy.response.headers['Expires']='0'
cherrypy.tools.nocache = cherrypy.Tool('before_finalize',nocache)
cherrypy.config.update({'tools.nocache.on':'True'})
cherrypy.tree.mount(Root(), '/')
cherrypy.tree.mount(TwoPoint(), '/twopoint')
cherrypy.engine.start()
cherrypy.engine.block()
For one example where this occurs, I've got the following javascript function that calls my python code:
function compTwoPoint(dataset,orig){
// call python code to generate images
$.post("/twopoint/compTwoPoint/"+dataset,
function(result){
res=jQuery.parseJSON(result);
if(res.success==true){
showTwoPoint(res.path,orig);
}
else{
alert(res.exception);
$('#display_loading').html("");
}
});
}
This calls the python code:
def twopoint(in_matrix):
"""proprietary code, can't share"""
def twopoint_file(in_file_name,out_file_name):
k = imread(in_file_name);
figure()
imshow(twopoint(k))
colorbar()
savefig(out_file_name,bbox_inches="tight")
close()
class TwoPoint:
#cherrypy.expose
def compTwoPoint(self,dataset):
try:
fnames=WebUtils.dataFileNames(dataset)
twopoint_file(fnames['filepath'],os.path.join(fnames['savebase'],"twopt.png"))
return encoder.iterencode({"success": True})
These functions work together to give the expected result. The problem is that after processing one input file, I am unable to process a second file. I don't seem to get a response from the server.
On the machine where things are working, I'm running python 2.7.6 and cherrypy 3.2.3. On the second machine, I have python 2.7.7 and cherrypy 3.3.0. While this may explain the difference in behavior, I'd like to find a way to make my code portable enough to overcome the difference in version (going from older to newer)
I'm not sure what the problem is, or even what to search for. I would appreciate any guidance or help you can offer.
(edit: Digging a bit more, I discovered something is happening with matplotlib. if I put print statments before and after the figure() command in twopoint_file, only the first one prints. Calling this function directly from a python interpreter (removing cherrypy from the equation) I get the following error:
can't invoke "event" command: application has been destroyed while executing "event generate $w{{ThemeChanged}}"
procedure "ttk::ThemeChanged" line 6 invoked from within "ttk::ThemeChanged"
end edit)
I don't understand what this error means, and haven't had much luck searching.
Old question, but I got the same problem which I fixed by changing backend in Matplotlib:
import matplotlib
matplotlib.use("qt4agg")
Related
i try to deploy an android app. I work with the kivy framework and buildozer in python. My issue is to include the pandas library. This is my simple and working test code:
from kivy.app import App
from kivy.uix.label import Label
import kivy
kivy.require('1.11.1')
import pandas as pd
class TestLibraries(App):
def build(self):
df = pd.DataFrame()
df.loc[0, 'text'] = 'this is pandas'
return Label(text = df.loc[0, 'text'])
if __name__ == '__main__':
TestLibraries().run()
The next step is do define the buildozer .spec file. Here i see two options:
Via requirements: So i modify the .spec file like this
# (list) Application requirements
# comma separated e.g. requirements = sqlite3,kivy
requirements = python3,kivy==1.11.1,pandas
This works very well.
2. Via recipe: I take the recipe from github. and put it into my folder called recipe. After that i modify the .spec file like this
# (str) The directory in which python-for-android should look for your own build recipes (if any)
p4a.local_recipes = /PATH_TO_FOLDER/recipe/
In the buildozer logfile i can read:
Listing '/PATH_TO_FOLDER/.buildozer/android/app/recipe/pandas'...
Compiling 'PATH_TO_FOLDER/.buildozer/android/app/recipe/pandas/__init__.py'...
So buildozer found the recipe but the library is not installed and the app dosen't works.
And the question is: why not?
You might ask me for using the second option because the first option works very well. In the next step i want to write a new recipe. So i have to learn how to include an existing recipe correctly.
I hope you unterstand my problem an have some advices.
Thanks Capa
How to create a Persistent(or any for that matter) REST HANDLER for any given(inbuilt) SPLUNK REST API Endpoint? How to use PersistentServerConnectionApplication class ?
I have gone through https://gist.github.com/LukeMurphey/238004c8976804a8e79570d22721fd99 but cant figure out where to start and how to make one.
There was a great .conf presentation about REST Handlers by James Ervin from a few years ago, https://conf.splunk.com/files/2016/slides/extending-splunks-rest-api-for-fun-and-profit.pdf
Sample code is available from https://github.com/jrervin/splunk-rest-examples
James' echo example is quite straight forward. Make sure you also pay attention to the additions that are necessary in web.conf and restmap.conf.
import os
import sys
if sys.platform == "win32":
import msvcrt
# Binary mode is required for persistent mode on Windows.
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
from splunk.persistconn.application import PersistentServerConnectionApplication
class EchoHandler(PersistentServerConnectionApplication):
def __init__(self, command_line, command_arg):
PersistentServerConnectionApplication.__init__(self)
def handle(self, in_string):
return {'payload': in_string, # Payload of the request.
'status': 200 # HTTP status code
}
Suggest you just get a copy of his app and deploy it, confirm it all works, then modify if for your particular use-case.
When my Flask+PySpark application has a function with #udf or #pandas_udf annotation, it will not start. If I simply remove the annotation, it does start.
If I try to start my application with Flask, the first pass of lexical interpretation of the script is executed. For example, the debugger stops at import lines such as
from pyspark.sql.functions import pandas_udf, udf, PandasUDFType
. However no statement is executed at all, including the initial app = Flask(name) statement. (Could it be some kind of hidden exception? )
If I start my application without Flask, with the same exact function and with the same imports, it does work.
These are the imports:
from pyspark.sql import SQLContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import pandas_udf, udf, PandasUDFType
import pandas as pd
This is the function:
#pandas_udf('string', PandasUDFType.SCALAR)
def pandas_not_null(s):
return s.fillna("_NO_NA_").replace('', '_NO_E_')
This is the statement that is not executed iff #pandas_udf is there:
app = Flask(__name__)
This is how IntelliJ starts Flask:
FLASK_APP = app
FLASK_ENV = development
FLASK_DEBUG = 1
In folder /Users/vivaomengao/projects/dive-platform/cat-intel/divecatintel
/Users/vivaomengao/anaconda/bin/python /Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py --module --multiproc --qt-support=auto --client 127.0.0.1 --port 56486 --file flask run
I'm running MacOS in my own computer.
I found the problem. The problem was that the #pandas_udf annotation required a Spark session at the time that the module is loaded (some kind of "first pass parsing" in Python). To solve the problem, I first called my code that creates a Spark session. Then I imported the module that has the function with the #pandas_udf annotation after. I imported it right inside the caller function and not at the header.
To troubleshoot, I set a breakpoint over the #pandas_udf function (in PyCharm) and stepped into the functions. With that I could inspect the local variables. One of the variables referred to something like "sc" or "_jvm". I knew from a past problem that that happened if the Spark session was not initialized.
After figuring out how to achieve this in a web application using WebPack (https://medium.com/#janos.jarecsni/auto-loading-modules-c8c47472d59b) I was quite sad to see that it is impossible to do in React Native.
I think being able to load all modules in a folder is a rather key technique (this way your code does not have to know about the individual modules), and so it is rather limiting in React Native.
Anyone knows if this is possible using some arcane ways, or any ideas how to work around this?
There's a few ways to achieve this, I have written an article to discuss this in detail. The good news is that there is a Babel plugin that allows you to do this.
https://medium.com/#janos.jarecsni/auto-loading-modules-c8c47472d59b
If you want to import all the modules in the current dir, you can try this one.
import_cur_modules only imports modules in the current dir,
import_sub_modules imports all the modules including sub dirs in the current dir
Note: This function only returns the import lines that need be executed in the __init__.py in the current dir.
import_lines = import_sub_modules(os.path.dirname(__file__))
for line in import_lines:
exec(line)
import os
def import_cur_modules(base_dir: str):
"""import modules in the current dir
Note: This function only returns the import lines that need be executed in the `__init__.py` in base_dir
"""
import_lines = []
for name in os.listdir(base_dir):
if name.endswith('.py') and name != '__init__.py':
import_lines.append(f"from . import {name[:-3]}")
return import_lines
def import_sub_modules(base_dir: str):
"""import all the modules including sub dirs in the current dir
Note: This function only returns the import lines that need be executed in the `__init__.py` in base_dir
"""
import_lines = []
for root, _, files in os.walk(base_dir):
root = os.path.relpath(root, start=base_dir)
for name in files:
if name.endswith('.py') and name != '__init__.py':
path_units = os.path.join(root, name[:-3]).split('/')
if path_units[0] == '.':
path_units.pop(0)
import_lines.append(
f"from .{'.'.join(path_units[:-1])} import {path_units[-1]}"
)
return import_lines
There are several similar questions on stack overflow, and I apologize in advance if I'm breaking etiquette by asking another one, but I just cannot seem to come up with the proper set of incantations to make this work.
I'm trying to use Flask + Flask-SQLAlchemy and then use pytest to manage the session such that when the function-scoped pytest fixture is torn down, the current transation is rolled back.
Some of the other questions seem to advocate using the db "drop all and create all" pytest fixture at the function scope, but I'm trying to use the joined session, and use rollbacks, since I have a LOT of tests. This would speed it up considerably.
http://alexmic.net/flask-sqlalchemy-pytest/ is where I found the original idea, and Isolating py.test DB sessions in Flask-SQLAlchemy is one of the questions recommending using function-level db re-creation.
I had also seen https://github.com/mitsuhiko/flask-sqlalchemy/pull/249 , but that appears to have been released with flask-sqlalchemy 2.1 (which I am using).
My current (very small, hopefully immediately understandable) repo is here:
https://github.com/hoopes/flask-pytest-example
There are two print statements - the first (in example/__init__.py) should have an Account object, and the second (in test/conftest.py) is where I expect the db to be cleared out after the transaction is rolled back.
If you pip install -r requirements.txt and run py.test -s from the test directory, you should see the two print statements.
I'm about at the end of my rope here - there must be something I'm missing, but for the life of me, I just can't seem to find it.
Help me, SO, you're my only hope!
You might want to give pytest-flask-sqlalchemy-transactions a try. It's a plugin that exposes a db_session fixture that accomplishes what you're looking for: allows you to run database updates that will get rolled back when the test exits. The plugin is based on Alex Michael's blog post, with some additional support for nested transactions that covers a wider array of user cases. There are also some configuration options for mocking out connectibles in your app so you can run arbitrary methods from your codebase, too.
For test_accounts.py, you could do something like this:
from example import db, Account
class TestAccounts(object):
def test_update_view(self, db_session):
test_acct = Account(username='abc')
db_session.add(test_acct)
db_session.commit()
resp = self.client.post('/update',
data={'a':1},
content_type='application/json')
assert resp.status_code == 200
The plugin needs access to your database through a _db fixture, but since you already have a db fixture defined in conftest.py, you can set up database access easily:
#pytest.fixture(scope='session')
def _db(db):
return db
You can find detail on how to setup and installation in the docs. Hope this helps!
I'm also having issues with the rollback, my code can be found here
After reading some documentation, it seems the begin() function should be called on the session.
So in your case I would update the session fixture to this:
#pytest.yield_fixture(scope='function', autouse=True)
def session(db, request):
"""Creates a new database session for a test."""
db.session.begin()
yield db.session
db.session.rollback()
db.session.remove()
I didn't test this code, but when I try it on my code I get the following error:
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/main.py", line 90, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
...
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/python.py", line 59, in filter_traceback
INTERNALERROR> return entry.path != cutdir1 and not entry.path.relto(cutdir2)
INTERNALERROR> AttributeError: 'str' object has no attribute 'relto'
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
from unittest import TestCase
# global application scope. create Session class, engine
Session = sessionmaker()
engine = create_engine('postgresql://...')
class SomeTest(TestCase):
def setUp(self):
# connect to the database
self.connection = engine.connect()
# begin a non-ORM transaction
self.trans = self.connection.begin()
# bind an individual Session to the connection
self.session = Session(bind=self.connection)
def test_something(self):
# use the session in tests.
self.session.add(Foo())
self.session.commit()
def tearDown(self):
self.session.close()
# rollback - everything that happened with the
# Session above (including calls to commit())
# is rolled back.
self.trans.rollback()
# return connection to the Engine
self.connection.close()
sqlalchemy doc has solution for the case