DSL, and SQL query very slow - sql

This question is both about Doctrine and Symfony2.
I've made a query using Doctrine DQL. And Doctrine generates an SQL like this;
SELECT f0_.id AS id0, f0_.nom AS nom1, f0_.prenom AS prenom2, f0_.email AS email3, p1_.move_distance AS move_distance4, a2_.adresse1 AS adresse15, a2_.adresse2 AS adresse26, p3_.nom AS nom7, v4_.nom AS nom8, v4_.url AS url9, v4_.cp AS cp10, v4_.insee AS insee11, v4_.lat AS lat12, v4_.lng AS lng13, COUNT(f0_.id) AS sclr17
FROM person_teacher p1_
INNER JOIN fos_user f0_ ON p1_.id = f0_.id
LEFT JOIN person_lesson p7_ ON f0_.id = p7_.person_id
LEFT JOIN lesson l6_ ON l6_.id = p7_.lesson_id AND (l6_.id = 1)
LEFT JOIN person_teacher_language p9_ ON p1_.id = p9_.personteacher_id
LEFT JOIN language l8_ ON l8_.id = p9_.language_id AND (l8_.id = 1)
LEFT JOIN note_value n10_ ON p1_.id = n10_.personTeacher_id
LEFT JOIN pays p3_ ON f0_.id_pays = p3_.id
LEFT JOIN note n5_ ON n10_.id_note = n5_.id
LEFT JOIN person_teacher_adresse p11_ ON p1_.id = p11_.personteacher_id
LEFT JOIN adresse a2_ ON a2_.id = p11_.adresse_id
LEFT JOIN ville v4_ ON a2_.id_ville = v4_.id
GROUP BY f0_.id LIMIT 2147483647 OFFSET 0;
The problem is about these joins:
LEFT JOIN person_lesson p7_ ON f0_.id = p7_.person_id
LEFT JOIN lesson l6_ ON l6_.id = p7_.lesson_id AND (l6_.id = 1)
LEFT JOIN person_teacher_language p9_ ON p1_.id = p9_.personteacher_id
LEFT JOIN language l8_ ON l8_.id = p9_.language_id AND (l8_.id = 1)
If I remove them, the request works. Long request, but works.
With the joins, the request is infinite (99.9% CPU time used by MySQL after 5mn) or maybe very long, but anyway, far too long.
How to optimize this query?
(PS : I thought the AND (l6_.id = 1) and the AND (l8_.id = 1) would act as "filters" and immediately remove unnecessary rows, but no, it makes things worse: it's faster if I remove those conditions, and add where clause at the end, something like: WHERE (l6_.id = 1) AND (l8_.id = 1))
Here's my DQL code:
$retour = $this->createQueryBuilder('p')
->select(array(
'p.id',
'p.nom',
'p.prenom',
'p.email',
'p.moveDistance',
'a.adresse1',
'a.adresse2',
'pn.nom as pays',
'v.nom AS ville_nom',
'v.url',
'v.cp',
'v.insee',
'v.lat',
'v.lng',
'ROUND(' .
$mul.' * ' .
'ACOS( ' .
'COS( RADIANS( '.$lat.')) * '.
'COS( RADIANS( v.lat )) * '.
'COS( RADIANS( v.lng )-radians('.$lng.')) + '.
'SIN( RADIANS( '.$lat.' )) * '.
'SIN( RADIANS( v.lat )) ' .
')'.
',2) AS distance',
($in_kilometers?'\'km\'':'\'miles\'').' AS unit',
'ROUND( AVG(n.importance), 1) AS importance',
'COUNT(p.id) AS total'
))
->leftJoin('p.noteValues', 'nv')
->leftJoin('p.paysNaissance', 'pn')
->leftJoin('nv.note', 'n')
->leftJoin('p.adresses', 'a')
->leftJoin('a.ville', 'v');
/* (!) Optimizer: find out why if I do a join "ON"
* it endlessly query. I did classical "join" then a "WHERE"
* at the end. Find out why this method is faster:
*/
if ($lesson_id>0) {
$retour = $retour
->leftJoin('p.lessons', 'le');
}
if ($language_id>0) {
$retour = $retour
->leftJoin('p.languages', 'ln');
}
if (($lesson_id>0) && ($language_id>0)) {
$retour = $retour
->where('le.id = :lesson_id')
->andWhere('ln.id = :language_id');
}
elseif ($lesson_id>0) {
$retour = $retour
->where('le.id = :lesson_id');
}
elseif ($language_id>0) {
$retour = $retour
->where('ln.id = :language_id');
}
$retour = $retour
->groupBy('p.id')
->having('distance>:dmin')
->andHaving('distance<=:dmax')
->addOrderBy($order_by_1, $order_sens_1)
->addOrderBy($order_by_2, $order_sens_2);
$params=array(
'dmin' => $distance_min,
'dmax' => $distance_max
);
if ($lesson_id>0) {
$params['lesson_id']= $lesson_id;
}
if ($language_id>0) {
$params['language_id']= $language_id;
}
$retour = $retour->setParameters($params);
$retour = $retour
->setFirstResult( $offset )
->setMaxResults( $limit );
return $retour;

I suggest to put the table with more rows after the FROM part of the select,
and change the line 'LEFT JOIN lesson l6_ ON l6_.id = p7_.lesson_id AND (l6_.id = 1)' this way : It seems that you don't need to join with p7 because you force to be l6_.id=1, so I change to this
'LEFT JOIN lesson l6_ ON (l6_.id = 1)'
Hope this help.

Without a look at your MySQL table definitions, to speed up your query make sure you have defined an index on each column involved in a join, because otherwise mysql must evaluate each record of the tables that miss the index.
Please add more details to your question and I'll edit the answer to reflect the changes.

Related

Convert a $_GET value in PDO

I wrote a basic function to join 3 tables regarding a similar id.
{
$id = $_GET["id"];
$product_edit_query = $this->DB->query('
SELECT * FROM products
LEFT JOIN products_ingredients
ON products_ingredients.product_id = products.product_id
LEFT JOIN products_languages
ON products_languages.product_id = products.product_id
WHERE products.product_id = 73
');
$product_edit = $product_edit_query->fetch();
$this->smarty->assign('product_edit', $product_edit);
}
In the last line I have of my request, i have WHERE products.product_id = 73. 73 is hardcoded, and should be the value of $_GET["id"], but all I tried didn't work to convert it in PDO, any help ?
Solution given by a friend:
{
$stmt = $this->DB->prepare('
SELECT * FROM products
LEFT JOIN products_ingredients
ON products_ingredients.product_id = products.product_id
LEFT JOIN products_languages
ON products_languages.product_id = products.product_id
WHERE products.product_id = :id
');
$stmt->execute([':id' => $_GET["id"]]);
$get_product_base = $stmt->fetch();
$this->smarty->assign('get_product_base', $get_product_base);
}

EclipseLink/TopLink full outer join instead left join

On model Card can have or not CardHolder ( 1:1 ), and I would like getting every cards filter by issuer linked to actived cardHolders plus cards without cardHolders, so I need a full outer join. Although the query below translate to left join returning just cards with cardHolders
final ExpressionBuilder builder = new ExpressionBuilder( Card.class );
Expression queryExp = builder.get( "cardIssuer" ).equal( cardIssuer );
queryExp = queryExp.and( builder.get( "cardStatus" ).get( "statusType" ).equal( "ACTIVATED" ) );
queryExp = queryExp.and( builder.getAllowingNull( "cardHolder" )isNull().or(
builder.get( "cardHolder" ).get( "status" ).get( "status" ).equal( "ACTIVE" ) ) );
Expression orderExpression = builder.get( "cardHolder" ).get( "surname" ).descending();
return getMultiple( queryExp, pageable , Card.class, orderExpression );
Translate query is
SELECT COUNT(t0.CARD_ID) FROM CARD t0 LEFT JOIN CARD_HOLDER t3
ON (t3.CARD_HOLDER_ID = t0.CARD_HOLDER_ID), CARD_HOLDER_STATUS t2, CARD_STATUS t1
WHERE (((((t0.CARD_ISSUER_ID = 10006) AND (t1.STATUS_TYPE = 'ACTIVATED')) AND (t2.STATUS = 'ACTIVE'))
AND (t0.CARD_ID IN ('52683','52692')))
AND ((t1.CARD_STATUS_ID = t0.CARD_STATUS_ID) AND (t2.STATUS_ID = t3.STATUS_ID)))
Due to JPA version the outer join is not properly done, so I found a way through native queries
#NamedNativeQueries( {
#NamedNativeQuery( name = Card.USER_DIRECTORY_BASE,
query = "select * from card c full outer join card_holder ch on c.card_holder_id = ch.card_holder_id "
+ "where c.CARD_ISSUER_ID = ?1 and c.card_status_id = 1 and ( ch.STATUS_ID = 1 or c.CARD_HOLDER_ID is null) "
+ "order by ch.FORENAME asc",
resultClass = Card.class ),
#NamedNativeQuery( name = Card.USER_DIRECTORY_BASE_COUNT,
query = "select count(*) from card c full outer join card_holder ch on c.card_holder_id = ch.card_holder_id "
+ "where c.CARD_ISSUER_ID = ?1 and c.card_status_id = 1 and ( ch.STATUS_ID = 1 or c.CARD_HOLDER_ID is null) "
+ "order by ch.FORENAME asc" )
} )
And getting results
Query query = em.createNamedQuery( Card.USER_DIRECTORY_BASE);
query.setParamenter(1,10000);
query.getResultList();

convert sql subquery with multiple joins to linq

I'm having trouble working in my subselect into LINQ. Here is the SQL followed by the LINQ. The SQL Query runs fine. The problem is incorporating the subquery. Any assistance would be appreciated:
SELECT DISTINCT
cic.CommitmentItemCategoryName + '( ' + cicType.CommitmentItemCategoryTypeName + ' )' AS displayCategory
, 'CategoryType_' + CAST(cic.CommitmentItemCategoryID AS VARCHAR(10)) AS displayCategoryID
, ISNULL(vwPAD.DollarsAllocated, 0) AS DisplayDollarsAllocated
,cic.CommitmentItemCategoryID
FROM
tblCommitmentItemCategory cic
LEFT OUTER JOIN
tblCommitmentItemCategoryType cicType ON cic.CommitmentItemCategoryTypeID = cicType.CommitmentItemCategoryTypeID
LEFT OUTER JOIN
tblAccountDirectToCommitmentItemCategory adToCIC ON adToCIC.CommitmentItemCategoryID IN (SELECT CommitmentItemCategoryID FROM tblCommitmentItemCategory)
LEFT OUTER JOIN
vw_ParentAccountDollarsAllocatedByCommitmentItemCategory vwPAD ON vwPAD.FiscalYear = 2015
AND cic.CommitmentItemCategoryID = vwPAD.CommitmentItemCategoryID
AND vwPAD.AccountDirectParentID = 19
WHERE
adToCIC.AccountDirectParentID = 19
ORDER BY
displayCategory
var queryInner = from cic in MyContext.tblCommitmentItemCategory
select new
{
cic.CommitmentItemCategoryID
};
var queryDollars = (from cic in MyContext.tblCommitmentItemCategory
join cicType in MyContext.tblCommitmentItemCategoryType
on cic.CommitmentItemCategoryTypeID equals cicType.CommitmentItemCategoryTypeID
into t2
from cicType in t2.DefaultIfEmpty()
join adToCIC in MyContext.tblAccountDirectToCommitmentItemCategory
//What goes here?
on ...
// on cic.CommitmentItemCategoryID equals adToCIC.CommitmentItemCategoryID
into t3
from adToCIC in t3.DefaultIfEmpty()
join vw in MyContext.vw_ParentAccountDollarsAllocatedByCommitmentItemCategory
on new { FiscalYear = currentFiscalYear, CommitmentItemCategory = cic.CommitmentItemCategoryID, ParentAccountID = currentParentAccountID }
equals new { FiscalYear = vw.FiscalYear, CommitmentItemCategory = vw.CommitmentItemCategoryID, ParentAccountID = vw.AccountDirectParentID }
into t
from vw in t.DefaultIfEmpty()
where adToCIC.AccountDirectParentID == currentParentAccountID
let displayCategory = cic.CommitmentItemCategoryName + " ( " + cicType.CommitmentItemCategoryTypeName + " )"
// Called CategoryType but it's actually the ID below
let displayCategoryTypeID = "CategoryType_" + cic.CommitmentItemCategoryID.ToString()
//let displayCategoryTypeID = "CategoryType!" + cic.CommitmentItemCategoryID + "_" + adToCIC.AccountDirectToCommitmentItemCategoryID.ToString()
let displayDollarsAllocated = vw.DollarsAllocated == null ? 0 : vw.DollarsAllocated
orderby cic.CommitmentItemCategoryName
select new
{
displayCategory,
displayCategoryTypeID,
cic.CommitmentItemCategoryID,
displayDollarsAllocated
}).Distinct();
Can you convert queryInner to a join? Then use that result in the larger query:
var queryInner =
from atcic in MyContext.tblAccountDirectToCommitmentItemCategory
join cic in MyContext.tblCommitmentItemCategory on atcic.CommitmentItemCategoryID equals cic.CommitmentItemCategoryID
select atcic;
And in the big query:
join adToCIC in queryInner into t3
from adToCIC in t3.DefaultIfEmpty()

Codeigniter: from sql to activerecord, what is the wrong?

I want to try change my sql code to active record, here is the original code and active record try. And the error message. Please tell me where is my wrong? Thanks.
SELECT `emlakilan`.`id`, `emlakdurum`.`durum`,`emlaktip`.`tip`,`semtler`.`semt`,`emlakilan`.`fiyat`,`emlakilan`.`tarih`,`resim`.`r1` FROM ucburcak.`emlakilan`
INNER JOIN `semtler` ON `emlakilan`.`semtId` = `semtler`.`semtid`
INNER JOIN `emlaktip` ON `emlakilan`.`emlakTipId` = `emlaktip`.`id`
INNER JOIN `emlakdurum` ON `emlakilan`.`emlakDurumId` =`emlakdurum`.`id`
LEFT JOIN `resim` ON `emlakilan`.`resimId` = `resim`.`id`
WHERE `emlakdurumId`=3 ORDER BY `emlakilan`.`id` DESC LIMIT 4;
//and here active record function
public function kucuk_ilan($durum ='3') // $durum:1=>tümü 2=>kiralık 3=>satılık 4=>takas
{
$query = $this->db->select('emlakilan.id, emlakdurum.durum, emlaktip.tip, semtler.semt, emlakilan.fiyat, emlakilan.tarih, resim.r1')
->from('emlakilan')
->join('semtler','emlakilan.semtId' === 'semtler.semtid','inner')
->join('emlaktip','emlakilan.emlakTipId'==='emlaktip.id','inner')
->join('emlakdurum','emlakilan.emlakDurumId'==='emlakdurum.id','inner')
->join('resim','emlakilan.resimId'==='resim.id','left')
->where('emlakdurumId'===$durum)
->order_by('emlakilan.id','desc')
->limit(4);
$query = $this->db->get('emlakilan');
return $query->result_array();
}
And, error message:
Error Number: 1066
Not unique table/alias: 'emlakilan'
SELECT emlakilan.id, emlakdurum.durum, emlaktip.tip, semtler.semt, emlakilan.fiyat, emlakilan.tarih, resim.r1 FROM (emlakilan, emlakilan) INNER JOIN semtler ON INNER JOIN emlaktip ON INNER JOIN emlakdurum ON LEFT JOIN resim ON WHERE 0 IS NULL ORDER BY emlakilan.id desc LIMIT 4
Filename: C:\Program Files\EasyPHP-12.1\www\3burcak\system\database\DB_driver.php
Line Number: 330
Thanks for help.
So 1. remove the 'from' line
2. the joins where wrong
try this:
public function kucuk_ilan($durum ='3') // $durum:1=>tümü 2=>kiralık 3=>satılık 4=>takas
{
$query = $this->db->select('emlakilan.id, emlakdurum.durum, emlaktip.tip, semtler.semt, emlakilan.fiyat, emlakilan.tarih, resim.r1')
->from('emlakilan')
->join('semtler','emlakilan.semtId = semtler.semtid','inner')
->join('emlaktip','emlakilan.emlakTipId = emlaktip.id','inner')
->join('emlakdurum','emlakilan.emlakDurumId = emlakdurum.id','inner')
->join('resim','emlakilan.resimId = resim.id','left')
->where('emlakdurumId', $durum)
->order_by('emlakilan.id','desc')
->limit(4);
$query = $this->db->get('emlakilan');
return $query->result_array();
}

Linq to SQL query taking forever

Ok, first I thought I had a problem with how I was querying things. But apparently the problem lies in how linq translates my query to sql.
Here's my linq:
var items = (from p in ctx.bam_Prestatie_AllInstances
join q in ctx.bam_Zending_AllRelationships on p.ActivityID equals q.ReferenceData
join r in ctx.bam_Zending_AllInstances on q.ActivityID equals r.ActivityID
orderby p.LastModified descending
where r.PrestatieOntvangen >= vanaf && r.PrestatieOntvangen <= tm
select new Data.BAMPrestatieInstance
{
Aanvaard = p.PrestatieAanvaard,
Contactnummer = r.ContactNr,
Identificatie = p.Identificatie,
Foutmelding = ((p.Foutmelding == "" || p.Foutmelding == null) && p.PrestatieAanvaard == null) ? "De prestatie is bezig met verwerkt te worden." : p.Foutmelding.Replace("\r\n", " "),
Ontvangen = p.PrestatieZendingOntvangen,
Uitvoerdatum = p.Uitvoerdatum.Replace('-', '/'),
Zender = r.Zender,
PrestatieCode = p.PrestatieCode,
ZendingsNr = r.Zendingnummer.ToString(),
GroepsAanvaarding = r.Identificatie
}).Take(100);
Which gets translated in:
SELECT TOP (100) [t3].[Zender], [t3].[ContactNr] AS [Contactnummer], [t3].[Identificatie], [t3].[value] AS [Uitvoerdatum], [t3].[PrestatieZendingOntvangen] AS [Ontvangen], [t3].[PrestatieAanvaard] AS [Aanvaard], [t3].[value2] AS [Foutmelding], [t3].[PrestatieCode], [t3].[value3] AS [ZendingsNr], [t3].[Identificatie2] AS [GroepsAanvaarding]
FROM (
SELECT [t2].[Zender], [t2].[ContactNr], [t0].[Identificatie], REPLACE([t0].[Uitvoerdatum], #p0, #p1) AS [value], [t0].[PrestatieZendingOntvangen], [t0].[PrestatieAanvaard],
(CASE
WHEN (([t0].[Foutmelding] = #p2) OR ([t0].[Foutmelding] IS NULL)) AND ([t0].[PrestatieAanvaard] IS NULL) THEN CONVERT(NVarChar(3800),#p3)
ELSE REPLACE([t0].[Foutmelding], #p4, #p5)
END) AS [value2], [t0].[PrestatieCode], CONVERT(NVarChar,[t2].[Zendingnummer]) AS [value3], [t2].[Identificatie] AS [Identificatie2], [t2].[PrestatieOntvangen], [t0].[LastModified]
FROM [dbo].[bam_Prestatie_AllInstances] AS [t0]
INNER JOIN [dbo].[bam_Zending_AllRelationships] AS [t1] ON [t0].[ActivityID] = [t1].[ReferenceData]
INNER JOIN [dbo].[bam_Zending_AllInstances] AS [t2] ON [t1].[ActivityID] = [t2].[ActivityID]
) AS [t3]
WHERE ([t3].[PrestatieOntvangen] >= #p6) AND ([t3].[PrestatieOntvangen] <= #p7)
ORDER BY [t3].[LastModified] DESC
As you can see, first it selects EVERYTHING and then it takes the top 100 and does the where. Why is this? Why can't it directly do the top 100, I think the problem why my queries run so long is because of this. Is there a better way to construct my linq query then?
Thanks
Try this:
var items = from p in ctx.bam_Prestatie_AllInstances.OrderByDesc(p => p.LastModified).Take(100)
join q in ctx.bam_Zending_AllRelationships on p.ActivityID equals q.ReferenceData
join r in ctx.bam_Zending_AllInstances on q.ActivityID equals r.ActivityID
where r.PrestatieOntvangen >= vanaf && r.PrestatieOntvangen <= tm
select new Data.BAMPrestatieInstance
{
Aanvaard = p.PrestatieAanvaard,
Contactnummer = r.ContactNr,
Identificatie = p.Identificatie,
Foutmelding = ((p.Foutmelding == "" || p.Foutmelding == null) && p.PrestatieAanvaard == null) ? "De prestatie is bezig met verwerkt te worden." : p.Foutmelding.Replace("\r\n", " "),
Ontvangen = p.PrestatieZendingOntvangen,
Uitvoerdatum = p.Uitvoerdatum.Replace('-', '/'),
Zender = r.Zender,
PrestatieCode = p.PrestatieCode,
ZendingsNr = r.Zendingnummer.ToString(),
GroepsAanvaarding = r.Identificatie
};
Since the sort applies only to your first table, I'd try to force the order by and take 100 to be applied before the join.