fetching is very slow in yii using CDBcriteria - yii

i have about 15000 rows in my table using cdbcrieria to fetch result , but it take too much time to load the result , can anyone please tell me the fastest way to fetch the result
i want the count of all fetched rows and method to fetch fastest result.Please help !
following is the code that i am currently using:
$criteria = new CDbCriteria;
// $Criteria->$criteria = 50;
$currentAction = Yii::app()->controller->action->id;
if ($currentAction == 'currentstock') {
$storeid = isset(Yii::app()->session['storeId']) ? Yii::app()->session['storeId'] : 0;
$criteria->condition .= ' storexparts.storeId="' . $storeid . '" AND ( partAppStatus="1" OR partAppStatus="15" OR partAppStatus="17" OR partAppStatus="22" ) ';
} else {
$userId = Yii::app()->user->id;
$storeid = isset(Yii::app()->session['storeId']) ? trim(Yii::app()->session['storeId']) : (isset($_REQUEST['storeId']) ? Yii::app()->session['storeId'] = trim($_REQUEST['storeId']) : '');
$criteria->condition = 'storexparts.storeId="' . $storeid . '"';
$criteria->condition .= ' AND userxfavparts.userId="' . $userId . '"';
$criteria->condition .= ' AND userxfavparts.storeId="' . $storeid . '"';
$criteria->condition .= ' AND (t.partAppStatus="1"';
$criteria->condition .= ' OR t.partAppStatus="15"';
$criteria->condition .= ' OR t.partAppStatus="17"';
$criteria->condition .= ' OR t.partAppStatus="22")';
// $criteria->order = 't.partCreateDate DESC';
}
$criteria->with = array('partUnit0', 'partCat0','fleetname', 'partType0', 'partSubCat0', 'storexparts', 'userxfavparts', 'partBrand0', 'partsalias','partPosXstore');
if (!empty($_GET['filterProperties'])) {
$string = implode("#", explode("\\", $_GET['filterProperties']));
$string = implode("#", explode("'", $string));
$code = stripslashes(trim($string));
$criteria->addCondition("partNumber LIKE CONCAT('%', :filterProperties , '%')
OR partsalias.aliasNumber LIKE CONCAT('%', :filterProperties , '%')
OR partType0.typeName LIKE CONCAT('%', :filterProperties , '%')
OR partCat0.catName LIKE CONCAT('%', :filterProperties , '%')
OR partSubCat0.subCatName LIKE CONCAT('%', :filterProperties , '%')
OR partDesc LIKE CONCAT('%', :filterProperties , '%')
OR storexparts.partQty LIKE CONCAT('%', :filterProperties , '%')
OR partNotes LIKE CONCAT('%', :filterProperties , '%')
OR partBrand0.brandCode LIKE CONCAT('%', :filterProperties , '%')
OR partUnit0.unitShort LIKE CONCAT('%', :filterProperties , '%')
OR partPosXstore.partShelf LIKE CONCAT('%', :filterProperties , '%')
OR partPosXstore.partBin LIKE CONCAT('%', :filterProperties , '%')
OR fleetname.fleetcode LIKE CONCAT('%', :filterProperties , '%')
OR fleetname.fleetserialno LIKE CONCAT('%', :filterProperties , '%')
OR fleetname.fleetmodel LIKE CONCAT('%', :filterProperties , '%')
OR fleettype0.typename LIKE CONCAT('%', :filterProperties , '%')
OR fleetbrand0.brandname LIKE CONCAT('%', :filterProperties , '%')
OR engine.engineno LIKE CONCAT('%', :filterProperties , '%')
OR partsalias.aliasBarcode LIKE CONCAT('%', :filterProperties , '%')
");
$criteria->params = array(':filterProperties' => $code);
}
$criteria->together = true;
if (isset($_REQUEST['Parts']) && !empty($_REQUEST['Parts'])) {
if (!empty($_REQUEST['Parts']['partAliasNumber'])) {
$partId = $_REQUEST['Parts']['partAliasNumber'];
$criteria->addCondition("partNumber LIKE CONCAT('%', :filterProperties , '%')
OR partsalias.aliasNumber LIKE CONCAT('%', :filterProperties , '%')");
$criteria->params = array(':filterProperties' => $partId);
}
}
$criteria->compare('partType0.typeName', $this->typeName, true, 'AND', 'LIKE');
$criteria->compare('partCat0.catName', $this->catName, true, 'AND', 'LIKE');
$criteria->compare('partSubCat0.subCatName', $this->subCatName, true, 'AND', 'LIKE');
$criteria->compare('partDesc', $this->partDesc, true, 'AND', 'LIKE');
$criteria->compare('storexparts.partQty', $this->partQty, true, 'AND', 'LIKE');
$criteria->compare('partNotes', $this->partNotes, true, 'AND', 'LIKE');
$criteria->compare('partUnit0.unitShort', $this->unitShort, true, 'AND', 'LIKE');
$criteria->compare('partBrand0.brandCode', $this->partBrand, true, 'AND', 'LIKE');
$criteria->compare('partPosXstore.partShelf', $this->partShelf, true, 'AND', 'LIKE');
$criteria->compare('partPosXstore.partBin', $this->partBin, true, 'AND', 'LIKE');

Very important: Your select must be optimized before any other thing. If you are using MySQL, you can check their official documentation to do it.
With your select properly optimized, I suggest you to create a database view with your RDMBS to avoid overload your query with fat ActiveRecords with a lot of data and metadata. DML for build definition, in MySQL, looks like:
CREATE OR REPLACE VIEW 'mydatabaseview' AS
(
SELECT
..... your select with all joins and minimal
..... WHERE conditions
)
Backing to the code, create a model with this database view. You can generate model with gii code generator. In generated model, into protected/models/Mydatabaseview.php make sure to define a method primaryKey():
class Mydatabaseview extends ActiveRecord
{
.....
// Define this method manually, because your database view
// don't have primary Key
public function primaryKey()
{
return 'field_from_database_view';
}
....
Finally, in same model in protected/models/Mydatabaseview.php, use your criteria as you want:
public function search()
{
$criteria = new CDbCriteria;
.......... all your conditions
}
If your issue continues, your probably need to use Yii Query Builder instead ActiveRecord. It implies to redesign your original concept.

Related

LINQ Query to search for users with partial name

I have this SQL query:
SELECT
*
FROM
employeeTable
WHERE
(
concat(first_name, ' ', last_name) like concat('%', replace(#MatchString, ' ', '%'), '%')
or
concat(last_name, ' ', first_name) like concat('%', replace(#MatchString, ' ', '%'), '%')
)
It works perfectly by searching both name and last name with partial values., by searching users with partial strings. So for example:
"ric jon" will find Rick Jones, Richard Jonesy, and Jonathan Prichter.
I have the following Linq Query, using Entity Framework:
from employee in context.Employee
where employee.first_name.Contains(matchString)
|| employee.last_name.Contains(matchString)
select employee
But the string "ric jon" does not find anything.
I can't seem to make that linq query work the same as the SQL query.
That SQL wouldn't in fact find those records you sampled with the given "ric jon". What you are asking for is a special filter which IMHO requires a function that is not natively supported by the backend (you didn't specify your backend but anyway at least I don't know a backend that would support this natively). Something like this would work (note that you are getting the Employee records to local, this is not done at backend level):
Func<Employee, string[], bool> match = (emp, _parts) => {
return
((emp.FirstName.IndexOf(_parts[0],StringComparison.CurrentCultureIgnoreCase ) != -1) &&
(_parts.Length == 1 || emp.LastName.IndexOf(_parts[1], StringComparison.CurrentCultureIgnoreCase) != -1)) ||
((_parts.Length == 1 || emp.FirstName.IndexOf(_parts[1], StringComparison.CurrentCultureIgnoreCase) != -1) &&
(emp.LastName.IndexOf(_parts[0], StringComparison.CurrentCultureIgnoreCase) != -1));
};
string search = "ric jon";
var result = context.Employee.AsEnumerable()
.Where(n => match(n, search.Split()));
Update:
You can use this one which would be supported by many backends:
string search = "ric jon";
string[] parts = search.ToLowerInvariant().Split();
string p1 = parts.Length < 1 ? "" :parts[0];
string p2 = parts.Length < 2 ? "" :parts[1];
var result = context.Employee.Where(n =>
((n.FirstName.ToLowerInvariant().Contains(p1) &&
n.LastName.ToLowerInvariant().Contains(p2))) ||
((n.FirstName.ToLowerInvariant().Contains(p2) &&
n.LastName.ToLowerInvariant().Contains(p1))));

Build PostgreSQL SELECT dynamically with different number of parameters

I'd like to write my PostgreSQL (9.3) SELECT dynamically based on the user's fields filled out in the query form:
$cond = '';
if ($_REQUEST['sel_when_a'] != '')
$cond .= ' AND field_lastmod >= $2';
if ($_REQUEST['sel_when_b'] != '')
$cond .= ' AND field_lastmod <= $3';
$sel = ExecSQLP('SELECT field_value, field_lastmod
FROM my_values
WHERE field_value ILIKE \'%\' || $1 || \'%\'
. $cond .
' ORDER BY node_id;'
, array( $_REQUEST['sel_find'], $_REQUEST['sel_when_a'], $_REQUEST['sel_when_b'] ));
(ExecSQLP is my function to execute an SQL statement with the arguments, based on pg_query_params.)
This is simple and elegant, but I get an error message: Query failed: ERROR: bind message supplies 3 parameters, but prepared statement "" requires 1 if sel_when_a ("after") and sel_when_b ("before") are empty.
I think this shouldn't be an error in PostgreSQL as it doesn't do any bad if an unused parameter is supplied. Can I suppress this error? Othervise how to build the statement with different number of parameters? I have a lot of them, not only the 3 above. It'd be terrible to prepare for the several cases with renumbered parameters if PostgreSQL doesn't allow this.
The documentation: http://www.postgresql.org/docs/9.3/interactive/sql-expressions.html#SQL-EXPRESSIONS-PARAMETERS-POSITIONAL is not very useful for this case.
Thanks for your tips. SQL injection vulnerability is not accepted!
(Perhaps similar to How to handle dynamic number of parameters in querystring when building REST api?, but the relation is not always "equals to a string".)
I'd build the parameter array while building query. Like this:
$cond = '';
$params = array( $_REQUEST['sel_find'] );
if ($_REQUEST['sel_when_a'] != '') {
$params[] = $_REQUEST['sel_when_a'];
$cond .= ' AND field_lastmod >= $'.count($params);
}
if ($_REQUEST['sel_when_b'] != '') {
$params[] = $_REQUEST['sel_when_b'];
$cond .= ' AND field_lastmod <= $'.count($params);
}
$sel = ExecSQLP('SELECT field_value, field_lastmod
FROM my_values
WHERE field_value ILIKE \'%\' || $1 || \'%\''
. $cond .
' ORDER BY node_id;'
, $params ));
It's not very sensible to ignore wrong number or parameters - in 99% of cases it would be a programming error.
If you would've used PDO for database connection you could use named parameters like this:
$cond = '';
$params = array( ':sel_find' => $_REQUEST['sel_find'] );
if ($_REQUEST['sel_when_a'] != '') {
$cond .= ' AND field_lastmod >= :sel_when_a';
$params[':sel_when_a'] = $_REQUEST['sel_when_a'];
}
if ($_REQUEST['sel_when_b'] != '') {
$cond .= ' AND field_lastmod <= :sel_when_b';
$params[':sel_when_b'] = $_REQUEST['sel_when_b'];
}
$sel = $dbh->prepare('SELECT field_value, field_lastmod
FROM my_values
WHERE field_value ILIKE \'%\' || :sel_find || \'%\''
. $cond .
' ORDER BY node_id;')->execute($params);

How can I convert this to use PDO?

I would like to use PDO for selecting (searching) a database.
The search 'form' has MULTIPLE fields that can be used.. 1 or many can be filled in to help refine the search. (or there can be many o them left blank/empty)
here is what I have been using (locally):
//localhost details
$db_username="root"; //database user name
$db_password="";//database password
$db_database="test"; //database name
$db_host="localhost";
mysql_connect($db_host,$db_username,$db_password);
#mysql_select_db($db_database) or die("Unable to connect to database.");
if(isset($_POST['submit'])) {
// define the list of fields
$fields = array('first', 'trialdate', 'wcity', 'wstate', 'plantif');
$conditions = array();
//loop through the defined fields
foreach($fields as $field){
// if the field is set and not empty
if(isset($_POST[$field]) && $_POST[$field] != '') {
// create a new condition while escaping the value inputed by the user (SQL Injection)
$conditions[] = "`$field` LIKE '%" . mysql_real_escape_string($_POST[$field]) . "%'";
}
}
//build the query
$query = "SELECT * FROM myTable ";
// if there are conditions defined
if(count($conditions) > 0) {
// append the conditions
$query .= "WHERE " . implode (' OR ', $conditions); // you can change to 'OR', but I suggest to apply the filters cumulative
}
$result = mysql_query($query);
if(isset($_POST['submit'])) {
while($row = mysql_fetch_array($result)) {
echo $row['first'] . "<br />"; //individual value
//build panels that displays everything from row..etc
}
}
}
this has been working fine... but I'd like convert to using the PDO approach.
I gave it a few tries...but am missing something here..
heres what I've tried so far..
//localhost details
$db_username="root"; //database user name
$db_password="";//database password
$db_database="test"; //database name
$db_host="localhost";
//PDO DB connection
$conn = new PDO('mysql:host='.$db_host.'dbname='.$db_database.'charset=utf8', $db_username, $db_password);
if(isset($_POST['submit'])) {
$stmt = $conn->prepare('SELECT * FROM myTable WHERE first LIKE :first OR trialdate LIKE :trialdate OR wcity LIKE :wcity OR wstate LIKE :wstate OR plantif LIKE :plantif');
//build query placeholders (*note: use bindValue for $_POST values)
$stmt->bindValue(':first', '%' . $_POST['first'] . '%');
$stmt->bindValue(':trialdate', '%' . $_POST['trialdate'] . '%');
$stmt->bindValue(':wcity', '%' . $_POST['wcity'] . '%');
$stmt->bindValue(':wstate', '%' . $_POST['wstate'] . '%');
$stmt->bindValue(':plantif', '%' . $_POST['plantif'] . '%');
$stmt->execute();
foreach ($stmt as $row) {
// do something with $row
echo $row['first'] . "<br />"; //individual value
}
}
I could use help on getting the PDO example working with a displayed result/row/value?

Issue with quoting of IS, NULL, NOT, !, and other reserved strings in ON conditions of JOIN clauses in Zend Framework 2

I have an SQL statement, that selets sport classes/courses (courses) with their trainers (trainers) over an association table (courses_trainers). Since some courses have multiple trainers, I use the GROUP_CONCAT(...) function to get the trainer names into one field. Some trainers rows are empty or NULL, so I add a trainers.name IS NOT NULL and a trainers.name != "" condition to the ON clause of the trainers JOIN:
SQL statement
SELECT
courses.id AS id,
GROUP_CONCAT(DISTINCT trainers.name SEPARATOR "|||") AS trainers
...
FROM
courses
...
LEFT JOIN
courses_trainers ON courses.id = courses_trainers.course_id
LEFT JOIN
trainers ON trainer_id = trainers.id
AND trainers.name IS NOT NULL
AND trainers.name != ""
...
...
WHERE `courses`.`id` = '898'
GROUP BY
courses.id
;
OO variant in the CourseTable class
public function findOnceByID($id) {
$concatDelimiter = self::CONCAT_DELIMITER;
$select = new Select();
...
$select->columns(array(
'id', ...
));
$select->from($this->tableGateway->getTable());
$select
...
->join('courses_trainers', 'courses.id = courses_trainers.course_id', array(), Select::JOIN_LEFT)
->join('trainers', 'trainer_id = trainers.id AND trainers.name IS NOT NULL AND trainers.name != ""', array(
'trainers' => new Expression('GROUP_CONCAT(DISTINCT trainers.name SEPARATOR "' . $concatDelimiter . '")')
), Select::JOIN_LEFT)
...
;
$where
->equalTo('courses.id', $id)
;
$select->where($where, Predicate::OP_AND);
$select->group('courses.id');
$resultSet = $this->tableGateway->selectWith($select);
return $resultSet;
}
The generated JOIN code I get looks like this:
LEFT JOIN
`courses_trainers` ON `courses`.`id` = `courses_trainers`.`course_id`
LEFT JOIN
`trainers` ON `trainer_id` = `trainers`.`id`
AND `trainers`.`name` `IS` `NOT` `NULL`
AND `trainers`.`name` `!`= `"``"`
So, here is to much quoted.
How to "explain" to the ZF, that IS, NOT, " etc. should not be quoted?
The join method accepts an expression as its second parameter for the ON clause
->join('trainers', new Expression('trainer_id = trainers.id AND trainers.name IS NOT NULL AND trainers.name != ""'),
Responsible for quoting is Zend\Db\Adapter\Platform\PlatformInterface#quoteIdentifierInFragment(...) (or more precise its implementations, in this case in Zend\Db\Adapter\Platform\Mysql), that gets as second argument an array of "safe words". Zend\Db\Sql\Select#processJoins(...) passes to it the array('=', 'AND', 'OR', '(', ')', 'BETWEEN', '<', '>'). As IS, NOT, ! etc. are not in the list, they are quoted.
The solution is to extend the Zend\Db\Sql\Select and overwrire its processJoins(...), adding the additional "safe words" needed to the list of the second argument in the quoteIdentifierInFragment(...) call:
<?php
namespace MyNamespace\Db\Sql;
use Zend\Db\Adapter\Driver\DriverInterface;
use Zend\Db\Adapter\StatementContainerInterface;
use Zend\Db\Adapter\ParameterContainer;
use Zend\Db\Adapter\Platform\PlatformInterface;
use Zend\Db\Sql\Select as ZendSelect;
class Select extends ZendSelect
{
...
protected function processJoins(PlatformInterface $platform, DriverInterface $driver = null, ParameterContainer $parameterContainer = null)
{
...
// process joins
$joinSpecArgArray = array();
foreach ($this->joins as $j => $join) {
...
$joinSpecArgArray[$j][] = ($join['on'] instanceof ExpressionInterface)
? $this->processExpression($join['on'], $platform, $driver, $this->processInfo['paramPrefix'] . 'join' . ($j+1) . 'part')
: $platform->quoteIdentifierInFragment($join['on'], array('=', 'AND', 'OR', '(', ')', 'BETWEEN', '<', '>', '!', 'IS', 'NULL', 'NOT', '"')); // on
...
}
return array($joinSpecArgArray);
}
...
}

perl called with 2 bind variables when 0 are needed

i have this code:
my (#matches, #vars);
if ($date_to) {
push(#matches, q{ m.mentioned_at <= ? });
push(#vars, $date_to);
}
if ($person) {
push(#matches, q{ t.name like ? });
push(#vars, $person);
}
if ($show) {
push(#matches, q{ t.name like ? });
push(#vars, $show);
}
if (#vars){
my $where = join (' and ', #matches);
$where = qq{where $where} if #matches
$res = $db->do({ page => $.p, per_page => $.s }, q{
select m.id, m.body, m.link,
count(distinct(m.id)) as num_items
from mentions m
$where
group by m.id, m.body, m.link,
order by m.created_at desc
}, #vars );
}
print STDERR Dumper ($where);
$VAR1 = 'where m.mentioned_at <= ? and t.name like ? ';`
print STDERR Dumper (#vars);
$VAR1 = '2012/06/01';
$VAR2 = 'Adby Phil';`
Why do i have this ERROR:
called with 2 bind variables when 0 are needed
Can't use an undefined value as an ARRAY reference
It seems like it does not see the $where clause.
I think you got the order of your parameters to do() switched:
$rv = $dbh->do($statement, \%attr, #bind_values);
Looks like you have \%attr first, then $statement.
Also, be aware that do() doesn't return a statement handle, so you can't fetch the data. You may want a more traditional prepare and execute sequence.