I have my django app running on server 1 which is connected to Amazon RDS (PostgreSQL) present on server2. On a day-to-day basis, I have to interact with database often to write/fetch/query data.
Currently, I am following the django ORM way across the app. I just wanted to know if there would be any improvement in the performance if I use PostgreSQL raw sql quereis using django.db connection cursor over ORM?
Following are the proposed ways for one sample query.
Django ORM
table_name.objects().all()
PostgreSQL raw sql quereis using django.db connection cursor
Can someone please suggest which one of the above will lead to a faster increased in performance provided I have my database running on RDS in another server
Using a raw query will result in close to no performance difference. The ORM indeed uses a few cycles to construct the query, but that is insignificant compared to the work of sending the query, the database receiving, interpreting and running the query, and sending the results back.
In some cases the ORM might indeed make a (over)complicated query, but that is not the case if you fetch all records. It sometimes requires ORM expertise (just like it requires expertise in SQL) to find a query that performs better.
You can furthermore already let the Django ORM do the work for you by using .values() to make it more convenient to construct a DataFrame:
res = pd.DataFrame(table_name.objects.values())
Related
I am developing an application which mimics in someways as notebook. The users login in to the web application, connect to the data-source(database/csv) and then write series of SQL queries. These are typically queries used to compute metrics.
The workflow subsequently is to run these queries on periodic basis to compute the metrics and persist them as time-series data.
Since the users can write SQL queries here, what would be the suggested approach to persist the query in a backend store?
You can store SQL queries as text.
But you must not allow users to define SQL queries and then run them verbatim, without first having a human vet the query to make sure it's not malicious (or even simply errors or unwise queries).
In my use case, ElasticSearch is already configured and has data that can be queried via REST API. I'm wondering if there is a way to write SQL statements that can query this data that ElasticSearch is already configured on.
Example, configure an adapter to ElasticSearch in MS SQLServer and use linkedserver to connect and run normal SQL statements.
I have read about the "river", but it seems to do the opposite of what I'm looking for. Any pointers will really help.
SQL Server is a relational database. It operates with tables in common. Posting requests to some URI is very unusual work for SQL Server. And there is no standard mechanism to do this.
What can you do:
Write a CLR-function to send post-requests
Map result json to some table (it can be difficult, because Elastic Search is document-oriented and SQL Server is not)
So, as for me, this way is very complicated. I advice to use some hand-written service to operate with DB and Elastic Search, and don't try to put all logic to SQL Server.
Something like Elasticsearch-SQL plugin might be of interest to you.
This won't allow you to use it as a linked server in MSSQL, but it will allow whatever application you have, to send SQL queries to the sql API on your ElasticSearch server and get results.
Example:
http://localhost:9200/_sql?sql=select * from indexName limit 10
We have a 3gb file of data from our propriartary CRM system which is using SQL as a database.
The CRM is not meeting our needs and we are thinking about moving to Microsoft access and building our own system from the start.
We were wondering if it is possible to easily migrate the SQL database into access?
Thanks for your time.
First of all, it has been a long time since I've had to use MS-Access (thankfully) but I'm not sure Access is suitable for databases of that size. In my opinion, it's best suited to small, desktop-type applications with few concurrent users.
To answer your question, I believe Access offers a data import feature(see under the External Data ribbon in 2013) - though I'd suspect it might balk at the idea of 3GB of data. Edit: Actually this link suggests the max databsae size is 2GB
What might be more useful however, is its Linked Table feature. If I remember correctly this allows you to access data stored in SQL Server (or a similar RDBMS) which is more suited to large volumes of data through an Access front end - complete with pre-canned forms, queries, reports etc..
It is possible and fairly straight forward to move all of your data tables from SQL Server to Access; however, SQL Server is a much more robust database engine than Access. I would highly recommend against that. I have however had very good success using Access (ADP project files) as a front for the interface and using SQL Server as the database back-end for simple to moderate complexity interfaces. If you are not getting the performance you desire from your SQL Server, you might want to consider query performance tuning and looking into memory and hardware upgrades first. I think you will get better and faster results from doing that.
The simple solution would be to “link” Access to SQL server. That way you continue to use a robust data engine, but are free to use all the reporting and coding features of Access.
In this setup then Access simply becomes a “front end” to the existing SQL database.
And you do NOT want to use an ADP project in Access since they are depreciated.
The process is thus to create a blank standard database, and then use linked tables to SQL server. This will not only eliminate the need to import data (which is likely changing all the time).
I am trying to identify slow queries in a large-scale Django 1.3 web application. As it is kind of difficult to match the raw sql query in the slow query log with the specific ORM statement in the code, I wondered if it is possible to add a SQL comment to the query constructed with the ORM, something like..
Object.objects.filter(Q(pub_date__lte=datetime.now)).comment('query no. 123')
Solution found by using .extra() for raw SQL commands on the django-user mailinglist:
Object.objects.filter(Q(pub_date__lte=datetime.now()).extra(where=['1=1 /* query no. 123 */'])
For those reading in 2022 onwards - there is a much better answer these days:
Google's sqlcommenter project has a Django middleware
[A] Django middleware whose purpose is to augment a SQL statement right before execution, with information about the controller and user code to help with later making database optimization decisions, after those statements are examined from the database server’s logs.
If I use SQLAlchemy's ORM to create objects and store them, does that mean I pretty much also only retrieve the data from the DB via SQLAlchemy? Will the underlying tables created by SQLAlchemy ORM still be sane? Can I still query the DB directly and have useful findings?
The ORM will only create and modify the database records as they're defined. You'll be able to query them just as you normally would. Using SqlAlchemy does not limit your normal database access in any way. SqlAlchemy can output the queries used into log files for seeing what exactly they're doing. It's nothing like html generation where you then don't want to look at the html it created.