In Django 1.2.3 I need to perform some queries that are not feasible with pure Django ORM functions. E.g.
result = MyModel.objects.extra(select={'stddev': 'STDDEV_SAMP(value)'}).values()
But, indeed, I need to run this code on several SQL engines (sqllite, MySQL and MSSQL). So, I should test settings.DATABASES['default']['engine'] and run engine-specific code.
Is there a more Django-like approach to this problem? (e.g. user-definined function to put somewhere so that Django run them according to default database engine).
Thank you
The proper place to store the code for accessing data is in a method in the model layer. That way, the model can:
be environment-aware
construct custom queries
use built-in ORM functions
These can be swapped around, optimized, and tweaked, without the rest of your application having to change a bit, because the rest of your application only manipulates data through your model.
Related
I've created a file <APP>/<MODEL>.sql according to the Django docs in order to make use of a hook to pass arbitrary SQL after syncdb is run. Inside this file are two function declarations for PostgreSQL and a statement that creates a database view. This runs fine in production but -- just as the docs say -- the code is not run for the test database because the use of fixtures is suggested. Now my unittests are missing the crucial database views and functions and thus fail.
How do I test code that relies on raw sql functions / views?
UPDATE
I dug up this ticket which concerns this question directly and also presents a small workaround.
I found the best way to handle this is to put the custom SQL code into Django's migrations.
Django and South (which is the predecessor to Django's own migration framework) both provide commands to create custom (i.e. empty) migrations. The code for creating database views or functions can be put into an empty migration and will be run whenever a new installation of the project is migrated or the test suite is run.
A tutorial on how to use custom migrations for database views with South can be found here. The syntax is a bit different in Django's own migration framework but the documentation about RunSQL explains it all.
Just run them natively like the sql that they are.
or
Use sqlcustom
or
Don't bother with them; you might find yourself swimming upstream to try and make good use of these functions and view via the ORM.
or
Consider another python framework (dare i say it) which is more attuned to using native sql.
And, if I can, does that mean I lose my advantage of treating the results as objects? I find complex queries confusing in many ORMs, not just Django's. But, it is probably because I have never really used an ORM. Does anyone use straight up SQL anymore?
edit: Am I defeating the purpose of having a framework if I bypass the ORM completely? They all have a "nifty" ORM, but when it comes to queries with lots of subqueries, derived tables, it doesn't look pretty.
Using Django's QuerySet API you have different possibilities:
You can use extra() which will return a queryset which evaluates to model objects. Therefore it is, as the name says, somehow limited, because for returning model instances it is necessary to eg. query the model's table. But you have the possibility to add additional SQL eg. the WHERE or ORDER clause. Querysets that use extra() can still use the features of the ORM - like chaining multiple filter() for example.
raw() returns a RawQueryset which also can be iterated over to get model instances, but you loose a lot of features that the ORM would normally provide.
And of course you can execute SQL directly, using a low level connection cursor API (no model instances of course).
Study the documentation on raw queries, there's also a lot of information on eg. how to map a model's fields on the data coming from a raw query and documeting a few gotchas when passing parameters into the query.
To also answer your edited question: I wouldn't use raw SQL when you can do it with the ORM, but of course the ORM is limited and if you need to do some more complex stuff you will always have to switch to SQL (but sometimes using extra() is enough-so you can still use the advantages of the ORM). Don't forget that the ORM works with every DB backend, while the custom SQL might not work with every database.
You can use raw SQL to either return objects; or if you want you can bypass the ORM completely.
Because of performance issue in application we are using in-memory approach for one of my project, in which we are loading all tables in RAM in form of generic collections (using nhibernate).
Issue is that when we were using simple linq to sql approach that time the testing and QA team were easily able to get sql queries using sql profiles for page they were viewing.
but with new approach (in-memory), we are loading all data in collection in one go and then are using linq to get data from that collection, so the testing and QA teams are not able to get the sql queries to verify the business logic and verifying bugs.
Please suggest any solution which can help in this situation, i think its not possible to get sql from linq to object (as all data is already in collection). please suggest any solution/approach/tool which can help me, to generate sql of those linq which is getting run against the collection or any other good solution.
NOTE: i know getting sql out of linq to sql is not possible, i am looking for suggestion which can help my QA and testing team to verify the queries/business logic (like they were doing earlier by capturing sql). like if possible log the linq queries as string which can be further used to be run/analyze.
The only SQL statements that you are running in this situation are the initial SELECT queries to load your data into memory. Once you have done those, you are no longer running "queries", you are instead performing .NET Framework calls.
Given that you have fundamentally changed the architecture of the application, you need to communicate this to the testing and QA teams - they will not be able to "see" what the application is now doing under the hood in the way that they could previously. If this sort of "deep dive" capability is a requirement of your test teams then your architecture is likely to require further modifications.
I would like to build arbitrary queries to a database, by allowing the user to build queries "on the fly". For every object/table, being able to select its attributes, and then "building" the query (that would translate into a SQL statement) and finally launching it, all through a web interface.
The ticketing system "rt" does that, for example, and another example would be the http://gatherer.wizards.com/Pages/Advanced.aspx webpage.
I'm currently programming in rails but any existing solution that implements this (or something similar) would be welcome.
Just be careful when creating dynamically generated queries like this that will need to be executed via sp_executesql (example: ms sql server), etc..... make sure you cover all of your bases to ensure that your application isnt vulnerable to SQL injection attacks as this type of development will essentially get one in a lot of trouble if its done incorrectly.. I would recommend storing all queries in a table and only reading queries from this table to help isolate the queries that are being ran in your application. Just identify them with a label, and allow the EU to choose the label from a dropdown list control on the frontend.
Good luck and I'm not sure of any software that will help assist
Not quite sure what your use case is here but i would say check out the
Doctrine ORM ( Object Relational Mapper )
**Edit
After reading more and looking at the example. I would only suggest Doctrine for a large website.
Then use Doctrines DQL syntax with some javascript/jquery magic for the forms.
Note that the queries you're referencing aren't arbitrary: they're on a very specific problem domain, on a specific set of sql tables.
That said, if I were you I'd look into how people are building sql queries with javascript. Something like these:
http://code.google.com/p/django-querybuilder/
http://css.dzone.com/articles/sqlike-sql-querying-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+zones%2Fria+(RIA+Zone)
http://thechangelog.com/post/4914956307/rel-arel-ported-to-node-js-with-some-changes
That'll at least get you a good idea of the underlying data structures.
In a PHP project I'm working on we need to create some DAL extensions to support multiple database platforms. The main pitfall we have with this is that different platforms have different syntaxes - notable MySQL and MSSQL are quite different.
What would be the best solution to this?
Here are a couple we've discussed:
Class-based SQL building
This would involve creating a class that allows you to build SQL querys bit-by-bit. For example:
$stmt = new SQL_Stmt('mysql');
$stmt->set_type('select');
$stmt->set_columns('*');
$stmt->set_where(array('id' => 4));
$stmt->set_order('id', 'desc');
$stmt->set_limit(0, 30);
$stmt->exec();
It does involve quite a lot of lines for a single query though.
SQL syntax reformatting
This option is much cleaner - it would read SQL code and reformat it based on the input and output languages. I can see this being a much slower solution as far as parsing goes however.
I'd recommend class-based SQL building and recommend Doctrine, Zend_Db or MDB2. And yeah, if it requires more lines to write simple selects but at least you get to rely on a parser and don't need to re-invent the wheel.
Using any DBAL is a trade-off in speed, and not just database execution, but the first time you use either of those it will be more painful than when you are really familiar with it. Also, I'm almost a 100% sure that the code generated is not the fastest SQL query but that's the trade-off I meant earlier.
In the end it's up to you, so even though I wouldn't do it and it sure is not impossible, the question remains if you can actually save time and resources (in the long run) by implementing your own DBAL.
A solution could be to have different sets of queries for different platforms with ID's something like
MySql: GET_USERS = "SELECT * FROM users"
MsSql: GET_USERS = ...
PgSql: GET_USERS = ...
Then on startup you load the needed set of queries and refers then
Db::loadQueries(platform):
$users = $db->query(GET_USERS)
Such a scheme would not take account of all the richness which SQL offers, so you would be better off with code-generated stored procs for all your tables for each DB.
Even if you use parametrized stored procs which are more database model-aware (i.e. they do joins or are user-aware and so are optimized for each vendor), that's still a great approach. I always view the database interface layer as providing more than just simple tables to the application, because that approach can be bandwidth-intensive and roundtrip wasteful.
if you have a set of backends that support it, I would agree that generating stored procedures to form a contract is the best approach. This approach, however, doesnt work if you have a backend that is limited in capabilty with regards to stored procedures in which case you build an abstaction layer to implement SQL or generate target specific sql based on an abstract/limited sql syntax.