I want to find sum of different columns, and I can do it using two ways as follows:
1st way
$obj = $modelClass::findBySql($sql, $params);
$clone = clone $obj;
$grand_amount = $clone->sum('amount');
$grand_tax_amount = $clone->sum('tax_amount');
$grand_total =$clone->sum('total_amount');
2nd way
$grand_amount = $modelClass::findBySql($sql1, $params)->sum('amount');
$grand_tax_amount = $modelClass::findBySql($sql1,$params)->sum('tax_amount');
$grand_total = $modelClass::findBySql($sql1, $params)->sum('total_amount');
Referring above two ways, which will be more efficient? Or both ways will execute same number of queries?
It does not matter if you clone query or not. sum() performs actual SQL query - if you call it three times, you will get three queries.
From microptimization perspective clone should be faster than creating the same query object three times. But you will not see much difference - performing real SQL query will be main bottleneck here, PHP overhead will be negligible.
BTW: Cloning from your example does not make much sense - you're never using original $obj and $clone is used three times. You can use $obj directly and avoid clonning - it make sense only if you want to modify cloned query without modifying original ActiveQuery object.
Related
I have a map with lots drawn on it, when a person select one or more lots I want to grab the information of those lots from the database and return it. The lots are identified by IDs like the ones in "lots_list".
At the moment I'm using a for loop to iterate over the list and fetch the data, passing the ID to a where clause with a placeholder, but the execution is fairly slow this way.
def getLotInfo(lots_list):
lots = []
for lot in lots_list:
try:
connection = psycopg2.connect(user=user,
password=password,
host=host,
port=port,
database=database)
cursor = connection.cursor()
Psql_Query = '''SELECT setor, quadra, lote, area_ocupada FROM iptu_sql_completo WHERE sql
LIKE %s'''
cursor.execute(Psql_Query, (lot,))
lots.append(cursor.fetchone())
print(lots)
except (Exception, psycopg2.Error) as error:
print("Error fetching data from PostgreSQL table", error)
finally:
# closing database connection.
if (connection):
cursor.close()
connection.close()
print("PostgreSQL connection is closed")
return lots
lots_list = ["0830480002", "0830480003", "0830480004"]
Lots = getLotInfo(lots_list)
I tried to use psycopg2 execute_batch command
Psql_Query = '''SELECT setor, quadra, lote, area_ocupada FROM
iptu_sql_completo WHERE sql LIKE %s'''
ppgextra.execute_batch(cursor, Psql_Query, SQLs)
lots.append(cursor.fetchall())
print(lots)
but I get the following error "not all arguments converted during string formatting" I am imagining that's because I should use a placeholder in the query for every item in the list, but if the list is ever changing in size, would there be a way to fix this? The IDs are not always sequential.
My question is: Is there a way to achieve better performance than using the for loop?
Your current code is pretty much the worst case I had in mind here:
What are the pros and cons of performing calculations in sql vs. in your application
What is faster, one big query or many small queries?
Maurice already mentioned the repeated connection overhead. But even with a single connection, this is far from ideal. Instead, run a single query and pass the whole list lots_list as Postgres array:
SELECT setor, quadra, lote, area_ocupada
FROM iptu_sql_completo
WHERE sql = ANY (%s);
Related:
Pass array literal to PostgreSQL function
IN vs ANY operator in PostgreSQL
The situation:
When I try to make a function or stored-proc, I usually start with a plain query window with sql-code. Often, I use #tblvar local tables to hold subsets of data needed later in the script.
During my testing of the developing script, I "select" the contents of the #tblvar-tables to observe the data to make sure it is correct for the scenario being tested.
Then, when I have debugged the complex query I then place that working code into a new stored-proc or user-defined-function.
But first, I need to "remove" or "comment-out" those "select #tblvar-tables" sentences.
I do this using the following sample/example code:
--DEBUG_SELECT
SELECT '#tblvarCostsAll_1' AS 'QueryName', * FROM #tblvarCostsAll WHERE (UID_VEHICLE IN (1628,1638,1672)) ORDER BY DATE_RANGE_CODE, UID_VGROUP, UID_VEHICLE;
--DEBUG_RETURN RETURN;
It becomes simple for me to "search/find" the phrase "--DEBUG_" and adjust the commenting process by joining the separate --DEBUG_SELECT-line with the adjacent SELECT-line.
The Question...
Is there a best practice in how to develop good SQL code from queries to udf-functions and usp-stored-procs?
Thanks...John
I have word model and phrase model
class Word(models.Model):
checked = models.BooleanField(default=False)
class Phrase(models.Model):
words = models.ManyToManyField(Word, null=True,related_name = "phrases")
Word model has attribute checked and many to many connection to phrase
I have a query, something like this:
Phrase.objects.exclude(words__checked=True).filter(words__group__pk__in = groups_ids)
But it works really really slow on big datasets, and the problem is in exclude section, cause without it - it works fast enough
So I found a suggestion that I should use raw sql here,
Performance issue with django exclude
So, how can rewrite this sentence with raw sql ?
(I need this query to both work postgres and mysql due to requirements, or I will need two queries if one query can't achieve this, postgres query has more priority)
So far I 've tried to use .extra syntax,but it didn't work, so asking it here.
I am using Groovy's Sql object to perform queries on a postgres db. The queries are being executed as follows:
List<Map> results = sql.rows("select * from my_table")
List<Map> result2= sql.rows("select * from my_second_table")
I have a groovy method that performs two queries and then does some processing to loop through the data to make a different dataset, however, on some occasions I recieve a postgres exception "This ResultSet is closed" error.
having searched, I originally thought it might be to do with the issue here: SQLException: This ResultSet is closed (running multiple queries and trying to access the data from the resultsets after the fact) - however, we only seem to get the exception on quite high load - which suggests that it isnt as simple as the first dataset is closed on executing the second query as if this was the case I would expect it to happen consistently.
Can anyone shed any light on how Groovy's Sql object handles these situations or suggest what might be going wrong?
Groovy SQL is kind of a weird cat. Easy to use for simple stuff. If you have more complex scenarios you probably are better off using something else. IMHO
I first suggest doing one query, storing the results into a collection, do the second query and store the results in a collection and then do your operations between two collections rather than result sets. If you data is too large for that, find some way to store the data locally before you start doing your aggregation or whatever.
If you don't like that, you might need to checkout the GDK source code to get a better idea what is done with the Sql.getInstance() related to result sets etc. Then you can sidestep whatever land mine you are inadvertently stepping on.
Perhaps
List<Map> results = sql.rows("select * from my_table")
List<Map> result2= sql.rows("select * from my_second_table")
will not work even in plain Java (as already said in the answer you provided when second call is made on statement all resources dedicated during the previous call have to be released). As mentioned by #Todd W Crone Groovy can optimize resources, e.g. release them dynamically or don't release them depending on certain run.
Actually I've tried with only one query. E.g. I've tried to get ResultSet and then iterate through it, like this (don't mind the names of table and field, query is rather simple; and result is one row that contains one column due to LIMIT 1 clause):
def resultSet = sql.executeQuery("SELECT age FROM person WHERE id = 12345 LIMIT 1")
resultSet.next()
and got This ResultSet is closed error. Seems that Groovy optimizes resources and closes ResultSet immediately. I didn't look into the source code of Groovy SDK. I found that eachRow and other methods with closure-style handlers work fine and don't throw This ResultSet is closed error.
Perhaps, methods with closure-style handlers can help you. For example, look at except from the article where rows() method with closure is used:
String query = 'select id as identifier, name as langName from languages'
def rows = db.rows(query, { meta ->
assert meta.tableName == 'languages'
assert meta.columnCount == 2
// ...
})
I am migrating raw PHP code to CakePHP and have some problems. As I have big problems with query to ORM transformation I temporary use raw SQL. All is going nice, but I met the ugly code and don't really know how to make it beautiful. I made DealersController and added function advanced($condition = null) (it will be called from AJAX with parameters 1-15 and 69). function looks like:
switch ($condition) {
case '1':
$cond_query = ' AND ( (d.email = \'\' OR d.email IS NULL) )';
break;
case '2':
$cond_query = ' AND (d.id IN (SELECT dealer_id FROM dealer_logo)';
break;
// There are many cases, some long, some like these two
}
if($user_group == 'group_1') {
$query = 'LONG QUERY WITH 6+ TABLES JOINING' . $cond_query;
} elseif ($user_group == 'group_2'){
$query = 'A LITLE BIT DIFFERENT LONG QUERY WITH 6+ TABLES JOINING' . $cond_query;
} else {
$query = 'A LITLE MORE BIT DIFFERENT LONG QUERY WITH 10+ TABLES JOINING' . $cond_query;
}
// THERE IS $this->Dealer->query($query); and so on
So.. As you see code looks ugly. I have two variants:
1) get out query addition and make model methods for every condition, then these conditions seperate to functions. But this is not DRY, because main 3 big queries is almost the same and if I will need to change something in one - I will need to change 16+ queries.
2) Make small reusable model methods/queries whitch will get out of DB small pieces of data, then don't use raw SQL but play with methods. It would be good, but the performance will be low and I need it as high as possible.
Please give me advice. Thank you!
If you're concerned about how CakePHP makes a database query for every joined table, you might find that the Linkable behaviour can help you reduce the number of queries (where the joins are simple associations on the one table).
Otherwise, I find that creating simple database querying methods at the Model level to get your smaller pieces of information, and then combining them afterwards, is a good approach. It allows you to clearly outline what your code does (through inline documentation). If you can migrate to using CakePHP's find methods instead of raw queries, you will be using the conditions array syntax. So one way you could approach your problem is to have public functions on your Model classes which append their appropriate conditions to an inputted conditions array. For example:
class SomeModel extends AppModel {
...
public function addEmailCondition(&$conditions) {
$conditions['OR'] = array(
'alias.email_address' => null,
'alias.email_address =' => ''
);
}
}
You would call these functions to build up one large conditions array which you can then use to retrieve the data you want from your controller (or from the model if you want to contain it all at the model layer). Note that in the above example, the conditions array is being passed by reference, so it can be edited in place. Also note that any existing 'OR' conditions in the array will be overwritten by this function: your real solution would have to be smarter in terms of merging your new conditions with any existing ones.
Don't worry about 'hypothetical' performance issues - if you've tried to queries and they're too slow, then you can worry about how to increase performance. But for starters, try to write the code as cleanly as possible.
You also might want to consider splitting up that function advanced() call into multiple Controller Actions that are grouped by the similarity of their condition query.
Finally, in case you haven't already checked it out, here's the Book's entry on retrieving data from models. There might be some tricks you hadn't seen before: http://book.cakephp.org/view/1017/Retrieving-Your-Data
If the base part of the query is the same, you could have a function to generate that part of the query, and then use other small functions to append the different where conditions, etc.