I am trying to generate a database in sqlite3 with pandas and so far everything is fine, it is generated as I need it, the problem is that when trying to access that database through a models created identical to the one created by pandas it breaks and django does not manages to access that database.
I have the following file "output.txt" which was basically generated by a function in views.py
Interface Status Protocol Description
BE9 down down
BE9.1 down down
BE9.100 down down
Lo0 up up
Lo8 up up
Lo30 up up
Lo100 up up ***MERGE LOOPBACK 100****
Lo111 up up Configured by NETCONF
Lo200 up up ***MERGE LOOPBACK 200****
Gi0/0/0/0 down down
Gi0/0/0/1 down down
Gi0/0/0/2 admin-down admin-down
Gi0/0/0/3 admin-down admin-down
Gi0/0/0/4 admin-down admin-down
Gi0/0/0/5 admin-down admin-down
Gi0/0/0/6 admin-down admin-down
and through that same function there is a fragment that takes that info and transfers it to the database:
#Converting txt to Detafrare
df = pd.read_fwf("output.txt")
#debugging df
df["Description"] = (df.iloc[:, 3:].fillna("").astype(str).apply(" ".join, axis=1).str.strip())
df = df.iloc[:, :4]
#Converting to SQLITE3 database,. If doesn't existe, a nre one will be created
connection = sqlite3.connect("db.sqlite3")
#Inserting data into SQLITE database
df.to_sql(
name = "Devices_App_interfaces", #this is the name format given by djangoa los nombres de las tablas.
con = connection,
if_exists = "replace",
index = False)
connection.commit()
connection.close()
#End Process
This is how the table looks visually:
so far everything is perfect, in parallel I create a class in models.py replicating all this format:
class Interfaces(models.Model):
Interface = models.CharField(max_length=50)
Status = models.CharField(max_length=50)
Protocol = models.CharField(max_length=50)
Description = models.CharField(max_length=50)
def __str__(self):
return self.name
So far everything is fine, I run python manage.py makemigrations and python manage.py migrate and everything is ok, it doesn't throw any problems, but if I try to call any value of that class from models.py it doesn't do anything, as if it didn't exist and if For example, I add the class to admin.py to see it from the django administrator, it throws me the following error:
OperationalError at /admin/Devices_App/interfaces/
no such column: Devices_App_interfaces.id
obviously Django can't access that data and therefore throws this error, what am I doing wrong? Is there a better or correct way to handle the load of databases externally and access them to display them in an html?
Since the table already exists and you filled it manually, you need to set managed=False in your model's Meta options.
Make sure you also add an automatic primary key field.
Read more in the documentation.
Related
We are having one django rest framework (DRF) project which should have multiple databases (mongoDB).Each databases should be independed. We are able to connect to one database, but when we are going to another DB for writing connection is happening but data is storing in DB which is first connected.
We changed default DB and everything but no changes.
(Note : Solution should be apt for the usage of serializer. Because we need to use DynamicDocumentSerializer in DRF-mongoengine.
Thanks in advance.
While running connect() just assign an alias for each of your databases and then for each Document specify a db_alias parameter in meta that points to a specific database alias:
settings.py:
from mongoengine import connect
connect(
alias='user-db',
db='test',
username='user',
password='12345',
host='mongodb://admin:qwerty#localhost/production'
)
connect(
alias='book-db'
db='test',
username='user',
password='12345',
host='mongodb://admin:qwerty#localhost/production'
)
models.py:
from mongoengine import Document
class User(Document):
name = StringField()
meta = {'db_alias': 'user-db'}
class Book(Document):
name = StringField()
meta = {'db_alias': 'book-db'}
I guess, I finally get what you need.
What you could do is write a really simple middleware that maps your url schema to the database:
from mongoengine import *
class DBSwitchMiddleware:
"""
This middleware is supposed to switch the database depending on request URL.
"""
def __init__(self, get_response):
# list all the mongoengine Documents in your project
import models
self.documents = [item for in dir(models) if isinstance(item, Document)]
def __call__(self, request):
# depending on the URL, switch documents to appropriate database
if request.path.startswith('/main/project1'):
for document in self.documents:
document.cls._meta['db_alias'] = 'db1'
elif request.path.startswith('/main/project2'):
for document in self.documents:
document.cls._meta['db_alias'] = 'db2'
# delegate handling the rest of response to your views
response = get_response(request)
return response
Note that this solution might be prone to race conditions. We're modifying a Documents globally here, so if one request was started and then in the middle of its execution a second request is handled by the same python interpreter, it will overwrite document.cls._meta['db_alias'] setting and first request will start writing to the same database, which will break your database horribly.
Same python interpreter is used by 2 request handlers, if you're using multithreading. So with this solution you can't start your server with multiple threads, only with multiple processes.
To address the threading issues, you can use threading.local(). If you prefer context manager approach, there's also a contextvars module.
So I created a model for storing credentials from Gmail users.
I wanted to make migrations but it says that there is no such table:
django.db.utils.OperationalError: no such table: mainApp_credentialsmodel
My models:
from django.db import models
# Create your models here.
from django.contrib.auth.models import User
from django.db import models
import json
class CredentialsModel(models.Model):
id = models.ForeignKey(User, primary_key=True,on_delete=models.CASCADE)
credential = models.CharField(max_length=1000)
Calling that model for checking authorization:
SCOPES = 'https://www.googleapis.com/auth/gmail.readonly'
store = CredentialsModel.objects.all()
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('mainApp/client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
service = build('gmail', 'v1', http=creds.authorize(Http()))
python manage.py makemigrations
If that error keep happening, check your migrations folder and check the files inside. Also check If your database is online, in case you have a database online, I've got this problem last week, but it was a problem with azure.
In last case I would create the table (model) again, changing the name to something similar, but If you have a significant amount of data in that table, then I think you can't do that.
It looks like your authorization code - including the query on CredentialsModel - is at module level. This means it runs when the module is imported, which happens before the migration has had a chance to run.
You must ensure that any database-accessing code is inside a function or method and is not invoked globally.
I am new at python, I just learnt how to create an api using flask restless and flask sql-alchemy. I however would like to seed the database with random values. How do I achieve this? Please help.
Here is the api code...
import flask
import flask.ext.sqlalchemy
import flask.ext.restless
import datetime
DATABASE = 'sqlite:///tmp/test.db'
#Create the Flask application and the FLask-SQLALchemy object
app = flask.Flask(__name__)
app.config ['DEBUG'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = DATABASE
db = flask.ext.sqlalchemy.SQLAlchemy(app)
#create Flask-SQLAlchemy models
class TodoItem(db.Model):
id = db.Column(db.Integer, primary_key = True)
todo = db.Column(db.Unicode)
priority = db.Column(db.SmallInteger)
due_date = db.Column(db.Date)
#Create database tables
db.create_all()
#Create Flask restless api manager
manager = flask.ext.restless.APIManager(app, flask_sqlalchemy_db = db)
#Create api end points
manager.create_api(TodoItem, methods = ['GET','POST','DELETE','PUT'], results_per_page = 20)
#Start flask loop
app.run()
I had a similar question and did some research, found something that worked.
The pattern I am seeing is based on registering a Flask CLI custom command, something like: flask seed.
This would look like this given your example. First, import the following into your api code file (let's say you have it named server.py):
from flask.cli import with_appcontext
(I see you do import flask but I would just add you should change these to from flask import what_you_need)
Next, create a function that does the seeding for your project:
#with_appcontext
def seed():
"""Seed the database."""
todo1 = TodoItem(...).save()
todo2 = TodoItem(...).save()
todo3 = TodoItem(...).save()
Finally, register these command with your flask application:
def register_commands(app):
"""Register CLI commands."""
app.cli.add_command(seed)
After you've configured you're application, make sure you call register_commands to register the commands:
register_commands(app)
At this point, you should be able to run: flask seed. You can add more functions (maybe a flask reset) using the same pattern.
From another newbie, the forgerypy and forgerypy3 libraries are available for this purpose (though they look like they haven't been touched in a bit).
A simple example of how to use them by adding them to your model:
class TodoItem(db.Model):
....
#staticmethod
def generate_fake_data(records=10):
import forgery_py
from random import randint
for record in records:
todo = TodoItem(todo=forgery_py.lorem_ipsum.word(),
due_date=forgery_py.date.date(),
priority=randint(1,4))
db.session.add(todo)
try:
db.session.commit()
except:
db.session.rollback()
You would then call the generate_fake_data method in a shell session.
And Miguel Grinberg's Flask Web Development (the O'Reilly book, not blog) chapter 11 is a good resource for this.
We are trying to run code that modifies a document when it loads as part of schema maintenance. We have a document such as
from mongoengine import Document
from mongoengine.fields import IntField, StringField
class User(Document):
version = IntField(default=0)
name = StringField()
Instances of User are created with version=1 and saved.
Later on, we modify this class as follows:
class User(Document):
version = IntField(default=0)
name = StringField(max_length=20)
Some of the version 1 documents now need to be truncated:
def upgrade_1_to_2(self):
self.name = self.name[:20]
We want to automatically run that function whenever a User is retrieved from the database, and only if version == 1. (Future upgrades would be upgrade_2_to_3(), and so on.)
Where in this API can I put code that runs when a document is retrieved?
Looks like it's a good candidate for the post_init signal and you can do a check there.
See: http://docs.mongoengine.org/guide/signals.html
I need to write a "standalone" script in Python to upload sales taxes to the account_tax table in the database using ONLY the ORM module of OpenERP. What I would like to do is something like the pseudo code below.
Can someone provide me a more details on the following:
1) what sys.path's do I need to set
2) what modules do I need to import before importing the "account" module. Currently when I import the "account" module I get the following error:
AssertionError: The report "report.custom" already exists!
3) What is the proper way to get my database cursor. In the code below I am simply calling psycopg2 directly to get a cursor.
If this approach cannot work, can anyone suggest an alternative approach other than writing XML files to load the data from the OpenERP application itself. This process needs to run outside of the the standard OpenERP application.
PSEUDO CODE:
import sys
# set Python paths to access openerp modules
sys.path.append("./openerp")
sys.path.append("./openerp/addons")
# import OpenERP
import openerp
# import the account addon modules that contains the tables
# to be populated.
import account
# define connection string
conn_string2 = "dbname='test2' user='xyz' password='password'"
# get a db connection
conn = psycopg2.connect(conn_string2)
# conn.cursor() will return a cursor object
cursor = conn.cursor()
# and finally use the ORM to insert data into table.
If you wanna do it via web service then have look at the OpenERP XML-RPC Web services
Example code top work with OpenERP Web Services :
import xmlrpclib
username = 'admin' #the user
pwd = 'admin' #the password of the user
dbname = 'test' #the database
# OpenERP Common login Service proxy object
sock_common = xmlrpclib.ServerProxy ('http://localhost:8069/xmlrpc/common')
uid = sock_common.login(dbname, username, pwd)
#replace localhost with the address of the server
# OpenERP Object manipulation service
sock = xmlrpclib.ServerProxy('http://localhost:8069/xmlrpc/object')
partner = {
'name': 'Fabien Pinckaers',
'lang': 'fr_FR',
}
#calling remote ORM create method to create a record
partner_id = sock.execute(dbname, uid, pwd, 'res.partner', 'create', partner)
More clearly you can also use the OpenERP Client lib
Example Code with client lib :
import openerplib
connection = openerplib.get_connection(hostname="localhost", database="test", \
login="admin", password="admin")
user_model = connection.get_model("res.users")
ids = user_model.search([("login", "=", "admin")])
user_info = user_model.read(ids[0], ["name"])
print user_info["name"]
You see both way are good but when you use the client lib, code is less and easy to understand while using xmlrpc proxy is lower level calls that you will handle
Hope this will help you.
As per my view one must go for XMLRPC or NETSVC services provided by Open ERP for such needs.
You don't need to import accounts module of Open ERP, there are possibilities that other modules have inherited accounts.tax object and had altered its behaviour as per your business needs.
Eventually if you feed data by calling those methods manually without using Open ERP Web service its possible you'll get undesired result / unexpected failures / inconsistent database state.
You can use Erppeek to browse data, but not sure if you can really upload data to DB, personally I use/prefer XMLRPC
Why don't you use the xmlrpc call of openerp.
it will not need to import account or openerp . and even you can have all orm functionality.
You can use python library to access openerp server using xmlrpc service.
Please check https://github.com/OpenERP/openerp-client-lib
It is officially supported by OpenERP SA.
If you want to interacti directly with the DB, you could just import psycopg2 and:
conn = psycopg2.connect(dbname='dbname', user='dbuser', password='dbpassword', host='dbhost')
cur = conn.cursor()
cur.execute('select * from table where id = %d' % table_id)
cur.execute('insert into table(column1, column2) values(%d, %d)' % (value1, value2))
cur.close()
conn.close()
Why you want to fix it like that?! You should create a localization module and define data in XML files. This is the standard way to fix such a problem in OpenERP.
You want to insert sales taxes for which country? Explain more plz.
from openerp.modules.registry import RegistryManager
registry = RegistryManager.get("databasename")
with registry.cursor() as cr:
user = registry.get('res.users').browse(cr, userid, listids)
print user