query optimisation with join on ordered data - optimization

My question concerns the optimization of a query sql.
My query retrieves a list of members and their last training.
To get the latest training I do a join on the result of a query returning a complete list of training for all members.
This query works but it is very slow, I'm really interrested if someone would have a solution for it to execute faster.
My query (about 16s):
SELECT
m.nom,
m.prenom,
m.ville,
m.maj,
mbf.libelle,
mbf.datefin,
m.id as idmb
FROM
membres m
LEFT JOIN (
select *
from membreform
where idformation = 1
order by datefin DESC
) as mbf ON mbf.idmembre = m.id
WHERE
role > 0 AND visible = 1
group by m.id
ORDER BY m.maj DESC
limit 0 , 20
My data structure :
membreform (1000 entries)
id int(11) NOT NULL AUTO_INCREMENT,
idmembre int(11) NOT NULL,
libelle varchar(128) NOT NULL,
idformation int(11) NOT NULL,
datedebut date NOT NULL,
datefin date NOT NULL DEFAULT '0000-00-00',
descript text NOT NULL,
KEY id (id),
KEY idmembre (idmembre),
KEY idformation (idformation)
membres (500 entries)
id int(3) NOT NULL AUTO_INCREMENT,
nom varchar(255) NOT NULL,
prenom varchar(255) NOT NULL,
ville varchar(255) NOT NULL,
email varchar(255) NOT NULL,
maj datetime NOT NULL,
role tinyint(4) NOT NULL DEFAULT '1',
PRIMARY KEY (id),
KEY role (role),
KEY maj (maj)
I tested this other way (about 0.40s) but I dont find that really clean
SELECT
m.nom,
m.prenom,
m.ville,
m.maj,
m.id as idmb,
(select
libelle
from
membreform
where
idformation = 1
AND m.id = membreform.idmembre
order by datefin DESC
limit 1
) libelle,
(select
datefin
from
membreform
where
idformation = 1
AND m.id = membreform.idmembre
order by datefin DESC
limit 1
) datefin
FROM
membres m
WHERE
role > 0 AND visible = 1
group by m.id
ORDER BY m.maj DESC
limit 0 , 20
I'm open to any suggestions because I am a bit stuck
thank you

You can use temporary table to increase performace. E.g (MS SQL)
1 Step - Get the memberform data into temp table by adding all relevant where conditions
select *
INTO #TempMembreform
from membreform
where idformation = 1 ...
order by datefin DESC
2 step - do the left outer join as you did in the above first example.
E.g.
SELECT
m.nom,
m.prenom,
m.ville,
m.maj,
mbf.libelle,
mbf.datefin,
m.id as idmb
FROM
membres m
LEFT JOIN #TempMembreForm as mbf
ON mbf.idmembre = m.id
WHERE
role > 0 AND visible = 1
group by m.id
ORDER BY m.maj DESC
limit 0 , 20
3 Step - add NOLOCK keyword if the data is not mision crical (e.g. Bank tracnsactions )
select *
INTO #TempMembreform
from membreform WITH(NOLOCK)
where idformation = 1
order by datefin DESC
SELECT
m.nom,
m.prenom,
m.ville,
m.maj,
mbf.libelle,
mbf.datefin,
m.id as idmb
FROM
membres m with(nolock)
LEFT JOIN #TempMembreForm as mbf
ON mbf.idmembre = m.id
WHERE
role > 0 AND visible = 1
group by m.id
ORDER BY m.maj DESC
limit 0 , 20
My profile

Related

Display other column values only for distinct values in derived column

These are my table structures -
Rule
Ruleset_rule_map
What I am trying to do is joining both tables based on rule.id and ruleset_rule_map.rule_id.
Want to fetch data like this-
If you can see I want to remove duplicates from derived column - rule_bucket.
At the same time I want to display rule_order only for distincts in derived column rule_bucket
Have written a query for the same -
select DISTINCT rrm.RULE_ORDER, CONCAT(r.PARENT_RULE,
IIF(r.CHILD_RULE IS NOT NULL, CONCAT('|', r.CHILD_RULE), r.CHILD_RULE),
IIF(r.SUB_CHILD_RULE IS NOT NULL, CONCAT('|', r.SUB_CHILD_RULE), r.SUB_CHILD_RULE),
IIF(r.SUB_SUB_CHILD_RULE IS NOT NULL, CONCAT('|', r.SUB_SUB_CHILD_RULE), r.SUB_SUB_CHILD_RULE) )
AS RULE_BUCKET from [RULE] r
inner join RULESET_RULE_MAP rrm on rrm.RULE_ID = r.ID
and rrm.RULESET_ID = 'AAE97A62-F37E-4454-A008-FF40A102BB25'
and r.PARENT_RULE <> 'N/A' order by rrm.RULE_ORDER;
but with the above query I can only get result like this.
Can some one please help me to write the correct query? Samples to help me solve the above are also welcome.
If you really one one row per rule bucket, just use aggregation:
SELECT MIN(RULE_ORDER), RULE_BUCKET
FROM (SELECT rrm.RULE_ORDER,
CONCAT(r.PARENT_RULE,
IIF(r.CHILD_RULE IS NOT NULL, CONCAT('|', r.CHILD_RULE), r.CHILD_RULE),
IIF(r.SUB_CHILD_RULE IS NOT NULL, CONCAT('|', r.SUB_CHILD_RULE), r.SUB_CHILD_RULE),
IIF(r.SUB_SUB_CHILD_RULE IS NOT NULL, CONCAT('|', r.SUB_SUB_CHILD_RULE), r.SUB_SUB_CHILD_RULE)
) AS RULE_BUCKET
FROM [RULE] r JOIN
RULESET_RULE_MAP rrm
ON rrm.RULE_ID = r.ID AND
rrm.RULESET_ID = 'AAE97A62-F37E-4454-A008-FF40A102BB25' AND
r.PARENT_RULE <> 'N/A'
) r
GROUP BY RULE_BUCKET
ORDER BY MIN(rrm.RULE_ORDER);
You can use the analytical function as follows:
select * from
(select t.*, row_number() over (partition by RULE_BUCKET order by RULE_ORDER desc) as rn
from
(select rrm.RULE_ORDER,
CONCAT(r.PARENT_RULE,
IIF(r.CHILD_RULE IS NOT NULL,
CONCAT('|', r.CHILD_RULE), r.CHILD_RULE),
IIF(r.SUB_CHILD_RULE IS NOT NULL,
CONCAT('|', r.SUB_CHILD_RULE), r.SUB_CHILD_RULE),
IIF(r.SUB_SUB_CHILD_RULE IS NOT NULL,
CONCAT('|', r.SUB_SUB_CHILD_RULE), r.SUB_SUB_CHILD_RULE) )
AS RULE_BUCKET
from [RULE] r
inner join RULESET_RULE_MAP rrm on rrm.RULE_ID = r.ID
and rrm.RULESET_ID = 'AAE97A62-F37E-4454-A008-FF40A102BB25'
and r.PARENT_RULE <> 'N/A') t) t
where rn = 1
order by RULE_ORDER;

Duplicate rows returned even though group by is used

This is my query
SELECT p.book FROM customers_books p
INNER JOIN books b ON p.book = b.id
INNER JOIN bookprices bp ON bp.book = p.book
WHERE b.status = 'PUBLISHED' AND bp.currency_code = 'GBP'
AND p.book NOT IN (SELECT cb.book FROM customers_books cb WHERE cb.customer = 1)
GROUP BY p.book, p.created_date ORDER BY p.created_date DESC
This is the data in my customers_books table,
I expect only 8,6,1 of books IDs to return but query is returning 8,6,1,1
table structures are here
CREATE TABLE "public"."customers_books" (
"id" int8 NOT NULL,
"created_date" timestamp(6),
"book" int8,
"customer" int8,
);
CREATE TABLE "public"."books" (
"id" int8 NOT NULL,
"created_date" timestamp(6),
"status" varchar(255) COLLATE "pg_catalog"."default",
)
CREATE TABLE "public"."bookprices" (
"id" int8 NOT NULL,
"currency_code" varchar(255) COLLATE "pg_catalog"."default",
"book" int8
)
what do you think I am doing wrong here.
I really dont want to use p.created_date in group by but I was forced to use because of order by
You have too many joins in the outer query:
SELECT b.book
FROM books b INNER JOIN
bookprices bp
ON bp.book = p.book
WHERE b.status = 'PUBLISHED' AND bp.currency_code = 'GBP' AND
NOT EXISTS (SELECT 1
FROM customers_books cb
WHERE cb.book = p.book AND cb.customer = 1
) ;
Note that I replaced the NOT IN with NOT EXISTS. I strongly, strongly discourage you from using NOT IN with a subquery. If the subquery returns any NULL values, then NOT IN returns no rows at all. It is better to sidestep this issue just by using NOT EXISTS.

Left join does not return ull values

I have two tables:
AppWindowsEvent:
CREATE TABLE [AppWindowsEvent]
(
[idAppWindowEvent] INT IDENTITY(1,1)
, [idAppWindow] INT
, [idEventType] INT
, [Order] INT
, CONSTRAINT PK_idAppWindowEvent PRIMARY KEY ([idAppWindowEvent])
, CONSTRAINT FK_idAppWindowEvent_AppWindow FOREIGN KEY ([idAppWindow]) REFERENCES [AppWindow]([idAppWindow])
, CONSTRAINT FK_idAppWindowEvent_EventType FOREIGN KEY ([idEventType]) REFERENCES [EventType]([idEventType])
)
Event:
CREATE TABLE [Event]
(
[idEvent] [INT] IDENTITY(1,1) NOT NULL
, [idEventType] [INT] NOT NULL
, [idEntity] [INT] NOT NULL
, CONSTRAINT PK_IdEvent PRIMARY KEY([idEvent])
, CONSTRAINT [FK_Event_EventType] FOREIGN KEY([idEventType]) REFERENCES [EventType] ([idEventType])
)
When i run this query:
SELECT
*
FROM
AppWindowsEvent AWE
LEFT JOIN Event E ON AWE.idEventType = E.idEventType
WHERE
AWE.idMill = 1
AND AWE.idAppWindow = 1
ORDER BY
AWE.[Order] ASC
The result: not return nulls.
And when i run this
SELECT
*
FROM
AppWindowsEvent AWE
LEFT JOIN Event E ON AWE.idEventType = E.idEventType
AND E.[idEntity] = 1234
WHERE
AWE.idMill = 1
AND AWE.idAppWindow = 1
ORDER BY
AWE.[Order] ASC
Result: return nulls.
NOTE:
I need the entire set of data that are and are not already configured, in case you want a specific set of events, in the AND of ON can be filtered by specific idEntity of the Event table and the result returns well, but only for that idEntity, in my case I need all idEntity.
Try this
SELECT *
FROM
AppWindowsEvent AWE
LEFT JOIN Event E ON AWE.idEventType = E.idEventType
WHERE
AWE.idMill = 1
AND AWE.idAppWindow = 1
AND E.[idEntity] = 1234
ORDER BY
AWE.[Order] ASC
Or if you doesn't want appear null valor in second table, you can use Inner Join instead Left Join
SELECT *
FROM
AppWindowsEvent AWE
Inner JOIN Event E ON AWE.idEventType = E.idEventType
AND E.[idEntity] = 1234
WHERE
AWE.idMill = 1
AND AWE.idAppWindow = 1
ORDER BY
AWE.[Order] ASC

Complex conditional SQL statement in SQLite

I'm trying to build a support system in which I now face a complex query. I've got a couple tables in my SQLite table wich look like so (slightly simplified):
CREATE TABLE "assign" (
"id" INTEGER NOT NULL PRIMARY KEY,
"created" DATETIME NOT NULL,
"is_assigned" SMALLINT NOT NULL,
"user_id" INTEGER NOT NULL REFERENCES "user" ("id")
);
CREATE TABLE "message" (
"id" INTEGER NOT NULL PRIMARY KEY,
"created" DATETIME NOT NULL,
"user_id" INTEGER REFERENCES "user" ("id") ,
"text" TEXT NOT NULL
);
CREATE TABLE "user" (
"id" INTEGER NOT NULL PRIMARY KEY,
"name" VARCHAR(255) NOT NULL
);
I now want to do a query which gives me *a list of users for which the last created Assign.is_assigned == False and the last created Message is later than the last created Assign*. So I now have the following (pseudo) query:
SELECT *
FROM user
WHERE ((IF (
SELECT is_assigned
FROM assign
WHERE assign.user_id = user.id
ORDER BY created DESC LIMIT 1
) = False)
AND ((
SELECT created
FROM message
WHERE message.user_id = user.id
ORDER BY created DESC
LIMIT 1
) > (
SELECT created
FROM assign
WHERE assign.user_id = user.id
ORDER BY created DESC
LIMIT 1))
);
This makes sense to me, but unfortunately not to the computer. I guess I need to make use of case statements or even joins or something but I have no clue how. Does anybody have a tip on how to do this?
You don't need the IF in there, and SQLite has no False, but otherwise, your query is quite correct:
SELECT *
FROM "user"
WHERE NOT (SELECT is_assigned
FROM assign
WHERE user_id = "user".id
ORDER BY created DESC
LIMIT 1)
AND (SELECT created
FROM message
WHERE user_id = "user".id
ORDER BY created DESC
LIMIT 1
) > (
SELECT created
FROM assign
WHERE user_id = "user".id
ORDER BY created DESC
LIMIT 1)
Try following query I have created in mysql
SELECT u.id AS 'user',u.name AS 'User_Name', ass.created AS 'assign_created',ass.is_assigned AS 'is_assigned',
msg.created AS 'message_created'
FROM `user` AS u
LEFT JOIN `assign` AS ass ON ass.`user_id` = u.`id`
LEFT JOIN `message` AS msg ON msg.`user_id` = u.id
LEFT JOIN (SELECT u.id AS 'user_id',u.name AS 'username',ass.created AS 'max_ass_created',ass.is_assigned AS 'assigned'
FROM `user` AS u
LEFT JOIN `assign` AS ass ON ass.`user_id` = u.`id`
LEFT JOIN `message` AS msg ON msg.`user_id` = u.`id`
GROUP BY u.id ORDER BY ass.created DESC) AS sub ON sub.user_id = u.id
WHERE (sub.assigned IS FALSE AND msg.created < sub.max_ass_created)
check SQL Fiddle of your scenario
hope this will solve your problem !

get the last record of table in select query

This is a follow up on another problem i had with getting-the-last-record-inserted-into-a-select-query
I am trying to edit a query that Andrea was kind enough to help me with yesterday which works fine for one page but I am trying to create a similar query without much luck.
What I need to to is for every board display the board name, the count of topics and messages linked to that board and the user, topic and date of the last message (which does work)
What i need is to get the board name, the topic and message count
This is my table structure
CREATE TABLE `boards` (
`boardid` int(2) NOT NULL auto_increment,
`boardname` varchar(255) NOT NULL default '',
PRIMARY KEY (`boardid`)
);
CREATE TABLE `messages` (
`messageid` int(6) NOT NULL auto_increment,
`topicid` int(4) NOT NULL default '0',
`message` text NOT NULL,
`author` varchar(255) NOT NULL default '',
`date` datetime(14) NOT NULL,
PRIMARY KEY (`messageid`)
);
CREATE TABLE `topics` (
`topicid` int(4) NOT NULL auto_increment,
`boardid` int(2) NOT NULL default '0',
`topicname` varchar(255) NOT NULL default '',
`author` varchar(255) NOT NULL default '',
PRIMARY KEY (`topicid`)
);
and the query I have come up with based on the query that Andrea did for me. What this query outputs in the boardname, the number of topics and messages (which says 1 even though there are 5), the topic author and messagecount (which isn't needed), the author and date of the last post (which is needed) but not the topic name which is needed
SELECT b.boardname, count( DISTINCT t.topicname ) AS topics, count( lm.message ) AS message, t.author as tauthor,
(select count(message) from messages m where m.topicid = t.topicid) AS messagecount,
lm.author as lauthor, lm.date
FROM topics t
INNER JOIN messages lm
ON lm.topicid = t.topicid AND lm.date = (SELECT max(m2.date) from messages m2)
INNER JOIN boards b
ON b.boardid = t.boardid
GROUP BY t.topicname
This my original query that does what I wanted but get the first post, not the last
SELECT b.boardid, b.boardname, count( DISTINCT t.topicname ) AS topics, count( m.message ) AS message, m.author AS author, m.date AS date, t.topicname AS topic
FROM boards b
INNER JOIN topics t ON t.boardid = b.boardid
INNER JOIN messages m ON t.topicid = m.topicid
INNER JOIN (
SELECT topicid, MAX( date ) AS maxdate
FROM messages
GROUP BY topicid
) test ON test.topicid = t.topicid
GROUP BY boardname
ORDER BY boardname
any help with this much appreciated
You need to define "LAST", in terms of an ORDER BY clause. Once you do that, you can just reverse the direction of your order and add LIMIT 1 to the query.
SELECT b.*, m.*, t,*
(
SELECT COUNT(*)
FROM topics ti
WHERE ti.boardid = b.boardid
) AS topiccount,
(
SELECT COUNT(*)
FROM topics ti, messages mi
WHERE ti.boardid = b.boardid
AND mi.topicid = ti.topicid
) AS messagecount
FROM boards b
LEFT JOIN
messages m
ON m.messageid = (
SELECT mii.messageid
FROM topics tii, messages mii
WHERE tii.boardid = b.boardid
AND mii.topicid = tii.topicid
ORDER BY
mii.date DESC
LIMIT 1
)
LEFT JOIN
topics t
ON t.topicid = m.topicid