Related
There are several similar questions on stack overflow, and I apologize in advance if I'm breaking etiquette by asking another one, but I just cannot seem to come up with the proper set of incantations to make this work.
I'm trying to use Flask + Flask-SQLAlchemy and then use pytest to manage the session such that when the function-scoped pytest fixture is torn down, the current transation is rolled back.
Some of the other questions seem to advocate using the db "drop all and create all" pytest fixture at the function scope, but I'm trying to use the joined session, and use rollbacks, since I have a LOT of tests. This would speed it up considerably.
http://alexmic.net/flask-sqlalchemy-pytest/ is where I found the original idea, and Isolating py.test DB sessions in Flask-SQLAlchemy is one of the questions recommending using function-level db re-creation.
I had also seen https://github.com/mitsuhiko/flask-sqlalchemy/pull/249 , but that appears to have been released with flask-sqlalchemy 2.1 (which I am using).
My current (very small, hopefully immediately understandable) repo is here:
https://github.com/hoopes/flask-pytest-example
There are two print statements - the first (in example/__init__.py) should have an Account object, and the second (in test/conftest.py) is where I expect the db to be cleared out after the transaction is rolled back.
If you pip install -r requirements.txt and run py.test -s from the test directory, you should see the two print statements.
I'm about at the end of my rope here - there must be something I'm missing, but for the life of me, I just can't seem to find it.
Help me, SO, you're my only hope!
You might want to give pytest-flask-sqlalchemy-transactions a try. It's a plugin that exposes a db_session fixture that accomplishes what you're looking for: allows you to run database updates that will get rolled back when the test exits. The plugin is based on Alex Michael's blog post, with some additional support for nested transactions that covers a wider array of user cases. There are also some configuration options for mocking out connectibles in your app so you can run arbitrary methods from your codebase, too.
For test_accounts.py, you could do something like this:
from example import db, Account
class TestAccounts(object):
def test_update_view(self, db_session):
test_acct = Account(username='abc')
db_session.add(test_acct)
db_session.commit()
resp = self.client.post('/update',
data={'a':1},
content_type='application/json')
assert resp.status_code == 200
The plugin needs access to your database through a _db fixture, but since you already have a db fixture defined in conftest.py, you can set up database access easily:
#pytest.fixture(scope='session')
def _db(db):
return db
You can find detail on how to setup and installation in the docs. Hope this helps!
I'm also having issues with the rollback, my code can be found here
After reading some documentation, it seems the begin() function should be called on the session.
So in your case I would update the session fixture to this:
#pytest.yield_fixture(scope='function', autouse=True)
def session(db, request):
"""Creates a new database session for a test."""
db.session.begin()
yield db.session
db.session.rollback()
db.session.remove()
I didn't test this code, but when I try it on my code I get the following error:
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/main.py", line 90, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
...
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/python.py", line 59, in filter_traceback
INTERNALERROR> return entry.path != cutdir1 and not entry.path.relto(cutdir2)
INTERNALERROR> AttributeError: 'str' object has no attribute 'relto'
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
from unittest import TestCase
# global application scope. create Session class, engine
Session = sessionmaker()
engine = create_engine('postgresql://...')
class SomeTest(TestCase):
def setUp(self):
# connect to the database
self.connection = engine.connect()
# begin a non-ORM transaction
self.trans = self.connection.begin()
# bind an individual Session to the connection
self.session = Session(bind=self.connection)
def test_something(self):
# use the session in tests.
self.session.add(Foo())
self.session.commit()
def tearDown(self):
self.session.close()
# rollback - everything that happened with the
# Session above (including calls to commit())
# is rolled back.
self.trans.rollback()
# return connection to the Engine
self.connection.close()
sqlalchemy doc has solution for the case
I'm creating a web-based interface for a number of different command line executables, and am using cherrypy behind apache (using mod_rewrite). I'm very new to this, and am having difficulty getting things configured properly. On my development machine, everything works reasonable well, but when I installed the code on a second machine I can't get anything to work properly.
The basic workflow for the applications is: 1. upload a dataset, 2. process the data (using python with some calls to executables using subprocess.call), 3. display the results on the web page.
After uploading and processing one dataset, everytime I attempt to process a second dataset the system stops responding. I'm not seeing any output in the terminal from the cherrypy process, or in the site log that shows any errors have occurred.
I'm starting cherrypy with the following conf file:
[global]
environment: 'production'
log.error_file: 'logs/site.log'
log.screen: True
tools.sessions.on: True
tools.session.storage_type: "file"
tools.session.storage_path: "sessions/"
tools.sessions.timeout: 60
tools.auth.on: True
tools.caching.on: False
server.socket_host: '0.0.0.0'
server.max_request_body_size: 0
server.socket_timeout: 60
server.thread_pool: 20
server.socket_queue_size: 10
engine.autoreload.on:True
My init.py file:
import cherrypy
import os
import string
from os.path import exists, join
from os import pathsep
from string import split
from mako.template import Template
from mako.lookup import TemplateLookup
from auth import AuthController, require, member_of, name_is
from twopoint import TwoPoint
current_dir = os.path.dirname(os.path.abspath(__file__))
lookup = TemplateLookup(directories=[current_dir + '/templates'])
def findInSubdirectory(filename, subdirectory=''):
if subdirectory:
path = subdirectory
else:
path = os.getcwd()
for root, dirs, names in os.walk(path):
if filename in names:
return os.path.join(root, filename)
return None
class Root:
#cherrypy.expose
#require()
def index(self):
tmpl = lookup.get_template("main.html")
return tmpl.render(usr=WebUtils.getUserName(),source="")
if __name__=='__main__':
conf_path = os.path.dirname(os.path.abspath(__file__))
conf_path = os.path.join(conf_path, "prod.conf")
cherrypy.config.update(conf_path)
cherrypy.config.update({'server.socket_host': '127.0.0.1',
'server.socket_port': 8080});
def nocache():
cherrypy.response.headers['Cache-Control']='no-cache,no-store,must-revalidate'
cherrypy.response.headers['Pragma']='no-cache'
cherrypy.response.headers['Expires']='0'
cherrypy.tools.nocache = cherrypy.Tool('before_finalize',nocache)
cherrypy.config.update({'tools.nocache.on':'True'})
cherrypy.tree.mount(Root(), '/')
cherrypy.tree.mount(TwoPoint(), '/twopoint')
cherrypy.engine.start()
cherrypy.engine.block()
For one example where this occurs, I've got the following javascript function that calls my python code:
function compTwoPoint(dataset,orig){
// call python code to generate images
$.post("/twopoint/compTwoPoint/"+dataset,
function(result){
res=jQuery.parseJSON(result);
if(res.success==true){
showTwoPoint(res.path,orig);
}
else{
alert(res.exception);
$('#display_loading').html("");
}
});
}
This calls the python code:
def twopoint(in_matrix):
"""proprietary code, can't share"""
def twopoint_file(in_file_name,out_file_name):
k = imread(in_file_name);
figure()
imshow(twopoint(k))
colorbar()
savefig(out_file_name,bbox_inches="tight")
close()
class TwoPoint:
#cherrypy.expose
def compTwoPoint(self,dataset):
try:
fnames=WebUtils.dataFileNames(dataset)
twopoint_file(fnames['filepath'],os.path.join(fnames['savebase'],"twopt.png"))
return encoder.iterencode({"success": True})
These functions work together to give the expected result. The problem is that after processing one input file, I am unable to process a second file. I don't seem to get a response from the server.
On the machine where things are working, I'm running python 2.7.6 and cherrypy 3.2.3. On the second machine, I have python 2.7.7 and cherrypy 3.3.0. While this may explain the difference in behavior, I'd like to find a way to make my code portable enough to overcome the difference in version (going from older to newer)
I'm not sure what the problem is, or even what to search for. I would appreciate any guidance or help you can offer.
(edit: Digging a bit more, I discovered something is happening with matplotlib. if I put print statments before and after the figure() command in twopoint_file, only the first one prints. Calling this function directly from a python interpreter (removing cherrypy from the equation) I get the following error:
can't invoke "event" command: application has been destroyed while executing "event generate $w{{ThemeChanged}}"
procedure "ttk::ThemeChanged" line 6 invoked from within "ttk::ThemeChanged"
end edit)
I don't understand what this error means, and haven't had much luck searching.
Old question, but I got the same problem which I fixed by changing backend in Matplotlib:
import matplotlib
matplotlib.use("qt4agg")
I'm trying to organize a large number of CloudWatch alarms for maintainability, and the web console grays out the name field on an edit. Is there another method (preferably something scriptable) for updating the name of CloudWatch alarms? I would prefer a solution that does not require any programming beyond simple executable scripts.
Here's a script we use to do this for the time being:
import sys
import boto
def rename_alarm(alarm_name, new_alarm_name):
conn = boto.connect_cloudwatch()
def get_alarm():
alarms = conn.describe_alarms(alarm_names=[alarm_name])
if not alarms:
raise Exception("Alarm '%s' not found" % alarm_name)
return alarms[0]
alarm = get_alarm()
# work around boto comparison serialization issue
# https://github.com/boto/boto/issues/1311
alarm.comparison = alarm._cmp_map.get(alarm.comparison)
alarm.name = new_alarm_name
conn.update_alarm(alarm)
# update actually creates a new alarm because the name has changed, so
# we have to manually delete the old one
get_alarm().delete()
if __name__ == '__main__':
alarm_name, new_alarm_name = sys.argv[1:3]
rename_alarm(alarm_name, new_alarm_name)
It assumes you're either on an ec2 instance with a role that allows this, or you've got a ~/.boto file with your credentials. It's easy enough to manually add yours.
Unfortunately it looks like this is not currently possible.
I looked around for the same solution but it seems neither console nor cloudwatch API provides that feature.
Note:
But we can copy the existing alram with the same parameter and can save on new name
.
I need to write a "standalone" script in Python to upload sales taxes to the account_tax table in the database using ONLY the ORM module of OpenERP. What I would like to do is something like the pseudo code below.
Can someone provide me a more details on the following:
1) what sys.path's do I need to set
2) what modules do I need to import before importing the "account" module. Currently when I import the "account" module I get the following error:
AssertionError: The report "report.custom" already exists!
3) What is the proper way to get my database cursor. In the code below I am simply calling psycopg2 directly to get a cursor.
If this approach cannot work, can anyone suggest an alternative approach other than writing XML files to load the data from the OpenERP application itself. This process needs to run outside of the the standard OpenERP application.
PSEUDO CODE:
import sys
# set Python paths to access openerp modules
sys.path.append("./openerp")
sys.path.append("./openerp/addons")
# import OpenERP
import openerp
# import the account addon modules that contains the tables
# to be populated.
import account
# define connection string
conn_string2 = "dbname='test2' user='xyz' password='password'"
# get a db connection
conn = psycopg2.connect(conn_string2)
# conn.cursor() will return a cursor object
cursor = conn.cursor()
# and finally use the ORM to insert data into table.
If you wanna do it via web service then have look at the OpenERP XML-RPC Web services
Example code top work with OpenERP Web Services :
import xmlrpclib
username = 'admin' #the user
pwd = 'admin' #the password of the user
dbname = 'test' #the database
# OpenERP Common login Service proxy object
sock_common = xmlrpclib.ServerProxy ('http://localhost:8069/xmlrpc/common')
uid = sock_common.login(dbname, username, pwd)
#replace localhost with the address of the server
# OpenERP Object manipulation service
sock = xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/object')
partner = {
'name': 'Fabien Pinckaers',
'lang': 'fr_FR',
}
#calling remote ORM create method to create a record
partner_id = sock.execute(dbname, uid, pwd, 'res.partner', 'create', partner)
More clearly you can also use the OpenERP Client lib
Example Code with client lib :
import openerplib
connection = openerplib.get_connection(hostname="localhost", database="test", \
login="admin", password="admin")
user_model = connection.get_model("res.users")
ids = user_model.search([("login", "=", "admin")])
user_info = user_model.read(ids[0], ["name"])
print user_info["name"]
You see both way are good but when you use the client lib, code is less and easy to understand while using xmlrpc proxy is lower level calls that you will handle
Hope this will help you.
As per my view one must go for XMLRPC or NETSVC services provided by Open ERP for such needs.
You don't need to import accounts module of Open ERP, there are possibilities that other modules have inherited accounts.tax object and had altered its behaviour as per your business needs.
Eventually if you feed data by calling those methods manually without using Open ERP Web service its possible you'll get undesired result / unexpected failures / inconsistent database state.
You can use Erppeek to browse data, but not sure if you can really upload data to DB, personally I use/prefer XMLRPC
Why don't you use the xmlrpc call of openerp.
it will not need to import account or openerp . and even you can have all orm functionality.
You can use python library to access openerp server using xmlrpc service.
Please check https://github.com/OpenERP/openerp-client-lib
It is officially supported by OpenERP SA.
If you want to interacti directly with the DB, you could just import psycopg2 and:
conn = psycopg2.connect(dbname='dbname', user='dbuser', password='dbpassword', host='dbhost')
cur = conn.cursor()
cur.execute('select * from table where id = %d' % table_id)
cur.execute('insert into table(column1, column2) values(%d, %d)' % (value1, value2))
cur.close()
conn.close()
Why you want to fix it like that?! You should create a localization module and define data in XML files. This is the standard way to fix such a problem in OpenERP.
You want to insert sales taxes for which country? Explain more plz.
from openerp.modules.registry import RegistryManager
registry = RegistryManager.get("databasename")
with registry.cursor() as cr:
user = registry.get('res.users').browse(cr, userid, listids)
print user
Does anyone knows how to load initial data for auth.User using sql fixtures?
For my models, I just got have a < modelname >.sql file in a folder named sql that syncdb does it's job beautifully. But I have no clue how to do it for the auth.User model. I've googled it, but with no success.
Thanks in advance,
Aldo
For SQL fixtures, you'd have to specifically have insert statements for the auth tables. You can find the schema of the auth tables with the command python manage.py sql auth.
The much easier and database-independent way (unless you have some additional SQL magic you want to run), is to just make a JSON or YAML fixture file in the fixtures directory of your app with data like this:
- model: auth.user
pk: 100000
fields:
first_name: Admin
last_name: User
username: admin
password: "<a hashed password>"
You can generate a hashed password quickly in a django shell
>>> from django.contrib.auth.models import User
>>> u = User()
>>> u.set_password('newpass')
>>> u.password
'sha1$e2fd5$96edae9adc8870fd87a65c051e7fdace6226b5a8'
This will get loaded whenever you run syncdb.
You are looking for loaddata:
manage.py loadata path/to/your/fixtureFile
But I think the command can only deal with files in XML, YAML, Python or JSON format (see here). To create such appropriate files, have a look at the dumpdata method.
Thanks for your answers. I've found the solution that works for me, and for coincidence was one of Brian's suggestion. Here it is:
Firs I disconnected the signal that created the Super User after syncdb, for I have my super user in my auth_user fixture:
models.py:
from django.db.models import signals
from django.contrib.auth.management import create_superuser
from django.contrib.auth import models as auth_app
signals.post_syncdb.disconnect(
create_superuser,
sender=auth_app,
dispatch_uid = "django.contrib.auth.management.create_superuser")
Then I created a signal to be called after syncdb:
< myproject >/< myapp >/management/__init__.py
"""
Loads fixtures for files in sql/<modelname>.sql
"""
from django.db.models import get_models, signals
from django.conf import settings
import <myproject>.<myapp>.models as auth_app
def load_fixtures(app, **kwargs):
import MySQLdb
db=MySQLdb.connect(host=settings.DATABASE_HOST or "localhost", \
user=settings.DATABASE_USER,
passwd=settings.DATABASE_PASSWORD, port=int(settings.DATABASE_PORT or 3306))
cursor = db.cursor()
try:
print "Loading fixtures to %s from file %s." % (settings.DATABASE_NAME, \
settings.FIXTURES_FILE)
f = open(settings.FIXTURES_FILE, 'r')
cursor.execute("use %s;" % settings.DATABASE_NAME)
for line in f:
if line.startswith("INSERT"):
try:
cursor.execute(line)
except Exception, strerror:
print "Error on loading fixture:"
print "-- ", strerror
print "-- ", line
print "Fixtures loaded"
except AttributeError:
print "FIXTURES_FILE not found in settings. Please set the FIXTURES_FILE in \
your settings.py"
cursor.close()
db.commit()
db.close()
signals.post_syncdb.connect(load_fixtures, sender=auth_app, \
dispatch_uid = "<myproject>.<myapp>.management.load_fixtures")
And in my settings.py I added FIXTURES_FILE with the path to my .sql file with the sql dump.
One thing that I still haven't found is how to fire this signal only after the tables are created, and not everytime syncdb is fired. A temporary work around for this is use INSERT IGNORE INTO in my sql command.
I know this solution is far from perfect, and critics/improvements/opinions are very welcome!
Regards,
Aldo
There is a trick for this: (tested on Django 1.3.1)
Solution:
python manage.py startapp auth_fix
mkdir auth_fix/fixtures
python manage.py dumpdata auth > auth_fixtures/fixtures/initial_data.json
Include auth_fix in INSTALLED_APPS inside settings.py
Next time you run python manage.py syncdb, Django will load the auth fixture automatically.
Explanation:
Just make an empty app to hold the fixtures folder. Leave __init__py, models.py and views.py in it so that Django recognizes it as an app and not just a folder.
Make the fixtures folder in the app.
python manage.py dumpdata auth will dump the "auth" data in the DB with all the Groups and Users information. The rest of the command simply redirects the output into a file called "initial_data.json" which is the one that Django looks for when you run "syncdb".
Just include auth_fix in INSTALLED_APPS inside settings.py.
This example shows how to do it in JSON but you can basically use the format of your choice.
If you happen to be doing database migrations with south, creating users is very simple.
First, create a bare data migration. It needs to be included in some application. If you have a common app where you place shared code, that would be a good choice. If you have an app where you concentrate user-related code, that would be even better.
$ python manage.py datamigration <some app name> add_users
The pertinent migration code might look something like this:
# encoding: utf-8
import datetime
from south.db import db
from south.v2 import DataMigration
from django.db import models
from django.contrib.auth.models import User
class Migration(DataMigration):
users = [
{
'username': 'nancy',
'email': 'nancy#example.com',
'password': 'nancypassword',
'staff': True,
'superuser': True
},
{
'username': 'joe',
'email': '',
'password': 'joepassword',
'staff': True,
'superuser': False
},
{
'username': 'susan',
'email': 'susan#example.com',
'password': 'susanpassword',
'staff': False,
'superuser': False
}
]
def forwards(self, orm):
"""
Insert User objects
"""
for i in Migration.users:
u = User.objects.create_user(i['username'], i['email'], i['password'])
u.is_staff = i['staff']
u.is_superuser = i['superuser']
u.save()
def backwards(self, orm):
"""
Delete only these users
"""
for i in Migration.users:
User.objects.filter(username=i['username']).delete()
Then simply run the migration and the auth users should be inserted.
$ python manage.py migrate <some app name>
An option is to import your auth.User SQL manually and subsequently dump it out to a standard Django fixture (name it initial_data if you want syncdb to find it). You can generally put this file into any app's fixtures dir since the fixtured data will all be keyed with the proper app_label. Or you can create an empty/dummy app and place it there.
Another option is to override the syncdb command and apply the fixture in a manner as you see fit.
I concur with Felix that there is no non-trivial natural hook in Django for populating contrib apps with SQL.
I simply added SQL statements into the custom sql file for another model. I chose my Employee model because it depends on auth_user.
The custom SQL I wrote actually reads from my legacy application and pulls user info from it, and uses REPLACE rather than INSERT (I'm using MySQL) so I can run it whenever I want.
And I put that REPLACE...SELECT statement in a procedure so that it's easy to run manually or scheduled with cron.