SQL optimisation - sql

I want to optimize some of the SQL and just need an opinion on whether I should do it or leave it as is and why I should do it. SQL queries are executed via PHP & Java, I will show an example in PHP which will give an idea of what Im doing.
Main concerns are:
-Maintainability.
-Ease of altering tables without messing with all the legacy code
-Speed of SQL (is it a concern???)
-Readability
Example of what I have right now:
I take a LONG array from a customer (cant make it smaller unfortunately) and update the existing values with the new values provided by a customer in the following way:
$i = 0;
foreach($values as $value)
{
$sql = "UPDATE $someTable SET someItem$i = '$value' WHERE username='$username'";
mysql_query($sql, $con);
$i+=1;
}
Its easy to see from the above example that if the array of values is long, than I execute a lot of SQL statements.
Should I instead do something like:
$i = 0;
$j = count($values);
$sql = "UPDATE $someTable SET ";
foreach($values as $value)
{
if($i < $j) //append values to the sql string up to the last item
{
$sql .= "someItem$i = '$value', ";
}
$i+=1;
}
$sql .= "someItem$i = '$value' WHERE username='$username'"; //add the last item and finish the statement
mysql_query($sql, $con); //execute query once
OR which way should it be done / should I bother making these changes? (there a lot of the type and they all have 100+ items)
Thanks in advance.

The only way you'll get a definitive answer is to run both of these methods and profile it to see how long they take. With that said, I'm confident that running one UPDATE statement with a hundred name value pairs will be faster than running 100 UPDATE statements.

Don't run 100 seperate UPDATE statements!
Use a MySQL wrapper class which, when given an array of name => value pairs will return an SQL UPDATE statement. Its really simple. I'm just looking for the one we use now...
We use something like this (registration required) but adapted a little more to suit our needs. Really basic but very very handy.
For instance, the Update method is just this
/**
* Generate SQL Update Query
* #param string $table Target table name
* #param array $data SQL Data (ColumnName => ColumnValue)
* #param string $cond SQL Condition
* #return string
**/
function update($table,$data,$cond='')
{
$sql = "UPDATE $table SET ";
if (is_string($data)) {
$sql .= $data;
} else {
foreach ($data as $k => $v) {
$sql .= "`" . $k . "`" . " = " . SQL::quote($v) . ",";
}
$sql = SQL::trim($sql , ',');
}
if ($cond != '') $sql .= " WHERE $cond";
$sql .= ";";
return $sql;
}

If you can't change the code, make sure it is enclosed in transaction (if the storage engine is InnoDB) so no non-unique indexes will be updated before commiting transaction (this will speed up the write) and the new row won't be flushed to disk.
If this is MyISAM table, use UPDATE LOW_PRIORTY or lock table before the loop and unlock after read.
Of course, I'm sure you have index on the username column, but just to mention it - you need such index.

Related

Phalcon: specific column on joined PHQL

The below PHQL generates a complex resultset like it should:
$phql = "SELECT User.*, ProductUser.* "
. "FROM ProductUser "
. "INNER JOIN User "
. "WHERE ProductUser.product_id = 5";
Replacing ProductUser.* with an existing column like ProductUser.id causes an error:
MESSAGE: The index does not exist in the row
FILE: phalcon/mvc/model/row.zep
LINE: 67
This is version 2.0.6. Is this a bug or am I making a mistake somewhere? According to the documentation it should be fine.
It was my mistake (expecting the row to always be an object).
I hope this helps someone because looping complex resultsets is not in the documentation.
$result = $this->modelsManager->createQuery($phql)->execute();
if ($result instanceof \Phalcon\Mvc\Model\Resultset\Complex) {
foreach ($result as $rows) {
foreach ($rows as $row) {
if (is_object($row)) {
$modelData = $row->toArray());
// this is what I needed
} else {
$id = $row;
}
}
}
}
First of all you're missing the ON clause in your query.
Anyways, it's easier and more error prone to use Phalcon's query builder for querying:
<?php
// modelManager is avaiable in the default DI
$queryBuilder = $this->modelsManager->createBuilder();
$queryBuilder->from(["product" => 'Path\To\ProductUser']);
// Inner join params are: the model class path, the condition for the 'ON' clause, and optionally an alias for the table
$queryBuilder->innerJoin('Path\To\User', "product.user_id = user.id", "user");
$queryBuilder->columns([
"product.*",
"user.*"
]);
$queryBuilder->where("product.id = 5");
// You can use this line to debug what was generated
// $generatedQuery = $queryBuilder->getPhql();
// var_dump($generatedQuery);
// Finish the query building
$query = $queryBuilder->getQuery();
// Execute it and get the results
$result = $query->execute();
// Now use var_dump to see how the result was fetched
var_dump($result);
exit;

SilverStripe 3 Left Join Missing argument

I have a data object related to some other data objects and I am trying to build a reporting page for them.
So far I've got the code below in my page controller to display a form where I will begin to select filtering options for the report.
However I am getting this error due to the left join:
[Warning] Missing argument 2 for SQLQuery::addLeftJoin()
It would seem that the raw2sql is outputting this when I've debugged:
\'AgeRangeData\', \'CallEvent.AgeRangeData ID=AgeRangeData.ID)\'
I'm assuming that the backslashes is what is causing the error
public function ReportingFilter(){
$DataObjectsList = $this->dbObject('DataObjects')->enumValues();
$fields = new FieldList(
new DropdownField('DataObjects', 'Data Objects', $DataObjectsList)
);
$actions = new FieldList(
new FormAction("FilterObjects", "Filter")
);
return new Form($this, "ReportingFilter", $fields, $actions);
}
public function FilterObjects($data, $form){
$data = $_REQUEST;
$query = new SQLQuery();
$object = $data['DataObjects'];
$leftJoin = Convert::raw2sql("'" . $object . "', 'CallEvent." . $object . " ID={$object}.ID)'");
$query->selectField("CallEvent.ID", "ID");
$query->setFrom('`CallEvent`');
$query->setOrderBy('CallEvent.Created DESC');
$query->addLeftJoin($leftJoin);
return $query;
}
SQLQuery::addLeftJoin() takes two arguments. The first is the table to join on and the second is the "on" clause.
You want:
$query = new SQLQuery();
$query->addLeftJoin($object, '"CallEvent"."ID" = "' . $object . '"ID"');
You'd need to escape $object appropriately, of course.
NB: Your code looks a little fragile as you're not ensuring that you $object actually has a DB table. I recommend you use ClassInfo::baseDataClass($object). This will have the added benefit that it will also sanitise your class name and ensure it's a real class.

CDbMigration::update does not work inside foreach loop

Following this question. There is something wrong, when using CDbMigration::update() inside foreach loop.
This code does not work correctly:
//This is executed inside Yii migration, so $this is CDbMigration.
foreach($idMap as $menuId=>$pageId)
{
$this->update
(
'menus_items',
array('link'=>'/content/show?id='.$pageId),
array('id = '.$menuId)
);
}
For each item in $idMap value of $pageId is always the same and equals value of last item in $idMap array. Therefore, every menu item points to the same URL.
This code works like a charm:
foreach($idMap as $menuId=>$pageId)
{
$sql = "UPDATE `menus_items` SET link = '/content/show?id=".$pageId."' WHERE id = ".$menuId."; ";
Yii::app()->db->createCommand($sql)->execute();
}
For each item in $idMap value of $pageId is always different and equals value of current item in $idMap array. Therefore, every menu item points to correct URL.
The same goes, when executing all statements in one SQL query:
$sql = '';
foreach($idMap as $menuId=>$pageId)
{
$sql .= "UPDATE `menus_items` SET link = '/content/show?id=".$pageId."' WHERE id = ".$menuId."; ";
}
Yii::app()->db->createCommand($sql)->execute();
Again, everything is OK.
Why using CDbMigration::update() fails, while direct SQL execution works like a charm?
I don't think you are providing the criteria parameter properly # array('id = '.$menuId)
. You should use a string if you want to send it like that, putting it in an array presumes you are mapping out the conditions in a key => value pair. Also you should be wrapping the value constraint in quotes id = "$menuId".

Two separate database queries to relate separate tables with Drupal 7 module?

I am developing a custom module for a site I'm working on and have created the following code. This is my first module, so any ideas of what I could be doing better would be appreciate.
As it is, this module works perfectly for me. But, I want to optimize it and be sure that I fix shoddy code.
Thanks!
The function in question is as follows:
// Declared variables for future incrementation
$total=0;
$countOne=0;
$countTwo=0;
$countThree=0;
$countOld=0;
// Call the native global user object from Drupal
global $user;
$userID = $user->uid;
// Check for nodes of given type owned by current user
$sql = db_query("SELECT nid FROM {node} WHERE type = 'content_type' AND uid = " . $userID);
// Iteratively checks each node id against a custom Drupal field on a separate table
foreach ($sql as $record) {
// SQL query for all custom fields attached to the node id given above
$query = db_query("SELECT * FROM {field_birth} WHERE entity_id = " . $record->nid);
$result = $query->fetchObject();
// The unmodified birth format (Y-m-d 00:00:00)
$originalBirth = $result->field_date_of_birth_value;
// The sanitized birth format for comparison (Y-m-d)
$birth = date('Y-m-d', strtotime($originalBirth));
// The current date/time (Y-m-d)
$now = date('Y-m-d');
//Future dates (Y-m-d)
$one_year = date('Y-m-d', strtotime('+1 year', strtotime($birth)));
$two_years = date('Y-m-d', strtotime('+2 years', strtotime($birth)));
$three_years = date('Y-m-d', strtotime('+3 years', strtotime($birth)));
// A count of all records returned before logical statements
$total++;
// Logic to determine the age of the records
if($now < $one_year) {
$countOne++;
}
else if($now >= $one_year && $now < $two_years) {
$countTwo++;
}
else if($now >= $two_years && $now < $three_years) {
$countThree++;
}
else {
$countOld++;
}
My question is, can I avoid having two separate database queries to hit both tables? I am not really sure how to go about that. Also, am I doing things in a manner which will be resource intensive and highly inefficient? As I am not a programmer by trade, I am not certain when code is 'good'. I do want to try my best to make this good code though since it is a module for a website I hope will last a long time.
Thank you stackoverflow community!
EDIT: The code I got working thanks to Mike is as follows. If anyone has a similar question / problem hopefully this will help!
// Join field_birth_table to nodes of given type owned by current user
$sql = db_select('node', 'n');
$sql->join('field_birth_table', 'b', 'n.nid = b.entity_id');
$sql
->fields('b', array('field_birth_field_value', 'entity_id'))
->condition('n.type', 'content_type')
->condition('n.status', '1')
->condition('n.uid', $user->uid)
->addTag('node_access');
$results = $sql->execute();
You can use a left join between the node and field_birth table:
$query = db_select('node', 'n');
$query->leftJoin('field_birth', 'b', '(b.entity_id = n.nid AND b.entity_type = :node)', array(':node' => 'node'));
$query
->fields('b', array())
->condition('n.type', 'content_type')
->condition('n.uid', $user->uid)
$results = $query->execute();

Infinite scroll for sql items

I'm trying to a segment of code which is behaving strange.
I am using the Paul Irish infinite scroll with the JQuery masonry plugin to display content.
The content is pulled by sql and has a set number, currently 20.
This means the page will load the first 20 rows of content from the database, then will load the next 20 upon scrolling down.
I have this functionality on a few pages and it works fine except for one page.
On this page instead of loading the most recent 20 posts, it loads every post in that particular field. Some have over 5000 posts so it literally crashes the page. A major problem as you can imagine.
I have been trying to adapt the code to look for anomalies and have narrowed it down (I think) to one particular segment.
This is the code used on a page that works:
function index() {
$userID = false;
$order = false;
$limit = $this->config->item('pin_load_limit');
$page = $this->uri->segment(3, 1);
$nextOffset = ($page -1) * $limit;
$nextPage = $page +1;
$sql = "SELECT *
FROM
pins";
if ($userID)
$sql .= " WHERE
user_id= $userID ";
if ($order) {
$sql .= " ORDER BY
' $order'";
} else {
$sql .= " ORDER BY time DESC";
}
$sql .= " LIMIT $nextOffset,$limit";
$query = $this->db->query($sql);
if ($query->num_rows() > 0) {
$row = $query->result();
}
This is the code on the page that doesn't work:
function getEachBoardPins($id,$limit=false)
{
$sql = "SELECT
*
FROM
pins
WHERE
board_id = $id
ORDER BY time DESC";
if($limit)
$sql .=" LIMIT 0 ,$limit" ;
$query = $this->db->query($sql);
return $query->result();
}
I know it is to do with the pin load limit but I have tried adding the code to the latter without success.
Perhaps somebody on here who knows more about it can provide some better insight?
I imagine that it is VERY LIKELY that when you call the "getEachBoardPins" function, you aren't setting the second parameter, hence, it is defaulting to false, hence, the limit SQL isn't being appended, so the select statement will run the whole statement.