SQL select query too slow on webhost, fine on localhost - sql

SELECT c.customers_lastname,
cg.customers_group_name,
dctc.coupons_id AS coupId,
dcto.coupons_id AS coupIdUsed,
dc.coupons_date_start AS coupStart,
count(DISTINCT o.orders_id) AS totalorders,
sum(op.products_quantity * op.final_price) AS ordersum
from
customers c LEFT JOIN customers_groups cg ON cg.customers_group_id = c.customers_group_id
LEFT JOIN (discount_coupons_to_customers dctc
LEFT JOIN discount_coupons dc ON dc.coupons_id = dctc.coupons_id
LEFT JOIN discount_coupons_to_orders dcto ON dcto.coupons_id = dctc.coupons_id
) ON c.customers_id = dctc.customers_id, orders_products op, orders o
WHERE c.customers_id = o.customers_id
AND c.customers_promotions = '0'
AND o.orders_id = op.orders_id
GROUP BY c.customers_id
ORDER BY ordersum DESC LIMIT 0, 10
The above query returns all customers that ever bought anything in our webshop (and some extra data), sorted by total order amount. It runs fine on localhost (a few seconds) but takes up to a minute on remote server. To make matters worse, the query can be modified via a form to include extra bits in the GROUP BY clause like:
HAVING (sum(op.products_quantity * op.final_price) >= 1000
AND/OR count(DISTINCT o.orders_id) > 2)
which doesn't exactly speed things up. there's about 5000 customers and 3000 orders at present. I added a time constraint WHERE order not older than one year but things didnt speed up after that.
i compared my local server and the online one.
localhost linux kernel is 3.2, online 2.6,
localhost php 5.4.4, online 5.3.26,
localhost mysql 5.5, online 5.1,
localhost php memory limit 128M, online 126M.
is there an obvious bottleneck? i sent an email to my webhost but didnt get a response. if I need to swap hosts I will but would like to know what to look out for. cheers,
Edit
using explain: (not sure how to format this, and no idea what it means) returns
`+----+-------------+-------+--------+-----------------------------+--------------+---------+---------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+--------------+---------+---------------------+-------+----------------------------------------------+
| 1 | SIMPLE | c | ALL | PRIMARY | NULL | NULL | NULL | 5541 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | cp | eq_ref | PRIMARY | PRIMARY | 4 | rpc.c.customers_id | 1 | |
| 1 | SIMPLE | dctc | eq_ref | customers_id,customers_id_2 | customers_id | 4 | rpc.c.customers_id | 1 | |
| 1 | SIMPLE | dcto | ref | PRIMARY | PRIMARY | 34 | rpc.dctc.coupons_id | 0 | Using index |
| 1 | SIMPLE | dc | ALL | PRIMARY | NULL | NULL | NULL | 1 | |
| 1 | SIMPLE | cg | ALL | PRIMARY | NULL | NULL | NULL | 5 | Using where; Using join buffer |
| 1 | SIMPLE | o | ALL | PRIMARY | NULL | NULL | NULL | 5010 | Using where; Using join buffer |
| 1 | SIMPLE | op | ALL | NULL | NULL | NULL | NULL | 10675 | Using where; Using join buffer |
+----+-------------+-------+--------+-----------------------------+--------------+---------+---------------------+-------+----------------------------------------------+`

Related

Can't figure out sql join

I'm using nextcloud to track data via the forms app, the table oc_forms_v2_submissions contains the entries:
SELECT * FROM `oc_forms_v2_submissions` WHERE `form_id` = 3;
+----+---------+--------------------------------------------+------------+
| id | form_id | user_id | timestamp |
+----+---------+--------------------------------------------+------------+
| 8 | 3 | anon-user-96684f301d22e7be44f07780a9bffe06 | 1663789158 |
| 9 | 3 | anon-user-a1eaa4f939b59e00b403c046410788aa | 1663835954 |
| 10 | 3 | anon-user-440d0dbe9c107492b6ec1a06d98004a8 | 1663942458 |
+----+---------+--------------------------------------------+------------+
the second table is oc_forms_v2_answers
SELECT * FROM `oc_forms_v2_answers`;
+----+---------------+-------------+-----------------------+
| id | submission_id | question_id | text |
+----+---------------+-------------+-----------------------+
| 10 | 8 | 7 | foo |
| 11 | 9 | 7 | bar |
| 12 | 10 | 7 | foo |
+----+---------------+-------------+-----------------------+
So basically i need to the take all the id entries from table submissions and match them with submission_id from answers and I want to have the data from the text column.
SELECT oc_forms_v2_submissions.id as submission_id
FROM `oc_forms_v2_submissions`
RIGHT JOIN `oc_forms_v2_answers` ON submission_id=oc_forms_v2_answers.submission_id;
This is all i could come up with so far but that returns only the submission_id field and everything triplicated :-D
+---------------+
| submission_id |
+---------------+
| 8 |
| 8 |
| 8 |
| 9 |
| 9 |
| 9 |
| 10 |
| 10 |
| 10 |
+---------------+
Edit:
The updated query still does not get me the field from oc_forms_answers:
SELECT oc_forms_v2_submissions.id as submission_id
FROM `oc_forms_v2_submissions`
RIGHT JOIN `oc_forms_v2_answers` ON oc_forms_v2_submissions.id=oc_forms_v2_answers.submission_id where form_id="3";
that is because you are comparing to identical columns, you need in the ON Clause, the link columns of both tables
Also you can use alias, to reduce the typing time
The RIGHT JOIN would also combine all answers with thes ubmission, but you you will never have more submission as answer, so a LEFT JOIN would gove ou all submissions even if there is no answer
SELECT oc_forms_v2_submissions.id as submission_id
FROM `oc_forms_v2_submissions`
LEFT JOIN `oc_forms_v2_answers` ON oc_forms_v2_submissions.id=oc_forms_v2_answers.submission_id;
This should do the trick (just update the correct naming of columns and tables)
SELECT s.id as submission_id, a.txt FROM submissions s
LEFT JOIN answers a
ON s.id=a.submission_id;
You can check this here in db-fiddle. I've used your info for creating a DB, so WHERE clause is missing but all the rest should give you results you're after.

Joining two tables and show data from one if there is any

I have these two tables that i need to join
fields_data fields
+------------+-----------+------+ +------+-------------+----------+
| relationid | fieldname | data | | name | displayname | position |
+------------+-----------+------+ +------+-------------+----------+
| 2 | ftp | test | | user | Username | top |
| 2 | other | 1234 | | pass | Password | top |
+------------+-----------+------+ | ftp | FTP | top |
| log | Log | top |
| txt | Text | mid |
+------+-------------+----------+
I want to get all the rows from the "fields" table if they have the position "top" AND if a row has a match on name = fieldname from fields_data it should also show the data. This is my join
SELECT
fd.`data`,
fd.`relationid`,
fd.`fieldname`,
f.`name`,
f.`displayname`
FROM `fields` AS f
LEFT OUTER JOIN `fields_data` AS fd
ON fd.`fieldname` = f.`name`
WHERE f.`position`='top' AND (fd.`relationid`='3' OR fd.`relationid` IS NULL)
My problem is that the above query only gives me this result:
+------+------------+-----------+------+-------------+
| data | relationid | fieldname | name | displayname |
+------+------------+-----------+------+-------------+
| NULL | NULL | NULL | user | Username |
| NULL | NULL | NULL | pass | Password |
| NULL | NULL | NULL | log | Log |
+------+------------+-----------+------+-------------+
The field called "ftp" is missing due to it having a relation to "2".. However i still want to display it as result but like the others with NULL in it. And if the SQL query had "fd.relationid='2'" instead of 3 it would give same result, but with the row containing ftp in name, holding data in the three fields.
I hope you get what i mean.. My english is not the best.. Heres the result i want:
with above query containing fd.`relationid`='3'
+------+------------+-----------+------+-------------+
| data | relationid | fieldname | name | displayname |
+------+------------+-----------+------+-------------+
| NULL | NULL | NULL | user | Username |
| NULL | NULL | NULL | pass | Password |
| NULL | NULL | NULL | ftp | FTP |
| NULL | NULL | NULL | log | Log |
+------+------------+-----------+------+-------------+
with above query containing fd.`relationid`='2'
+------+------------+-----------+------+-------------+
| data | relationid | fieldname | name | displayname |
+------+------------+-----------+------+-------------+
| NULL | NULL | NULL | user | Username |
| NULL | NULL | NULL | pass | Password |
| test | 2 | ftp | ftp | FTP |
| NULL | NULL | NULL | log | Log |
+------+------------+-----------+------+-------------+
You want to move the condition to the on clause:
SELECT fd.`data`, fd.`relationid`, fd.`fieldname`, f.`name`, f.`displayname`
FROM `fields` f LEFT OUTER JOIN
`fields_data` fd
ON fd.`fieldname` = f.`name` AND fd.`relationid` = '3'
WHERE f.`position`='top' ;
It is interesting that the semantics of your query and this query are different -- and you found the exact situation: when there is a match on another value, the where clause form filters out the row. This will still keep everything.
As a note, the following also does what you want:
SELECT fd.`data`, fd.`relationid`, fd.`fieldname`, f.`name`, f.`displayname`
FROM `fields` f LEFT OUTER JOIN
(SELECT fd.*
FROM `fields_data` fd
WHERE fd.`relationid` = '3'
) fd
ON fd.`fieldname` = f.`name`
WHERE f.`position` = 'top' ;
I wouldn't recommend writing the query this way, particularly in MySQL (because the subquery is materialized). However, understanding why your version is different from these versions (and why these are the same) is a big step forward in mastering outer joins.

Rewriting this subquery?

I am trying to build a new table such that the values in the existing table are NOT contained (but obviously the following checks for contained) in another table. Following is my table structure:
mysql> explain t1;
+-----------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| point | bigint(20) unsigned | NO | MUL | 0 | |
+-----------+---------------------+------+-----+---------+-------+
mysql> explain whitelist;
+-------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| x | bigint(20) unsigned | YES | | NULL | |
| y | bigint(20) unsigned | YES | | NULL | |
| geonetwork | linestring | NO | MUL | NULL | |
+-------------+---------------------+------+-----+---------+----------------+
My query looks like this:
SELECT point
FROM t1
WHERE EXISTS(SELECT source
FROM whitelist
WHERE MBRContains(geonetwork, GeomFromText(CONCAT('POINT(', t1.point, ' 0)'))));
Explain:
+----+--------------------+--------------------+-------+-------------------+-----------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+--------------------+-------+-------------------+-----------+---------+------+------+--------------------------+
| 1 | PRIMARY | t1 | index | NULL | point | 8 | NULL | 1001 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | whitelist | ALL | _geonetwork | NULL | NULL | NULL | 3257 | Using where |
+----+--------------------+--------------------+-------+-------------------+-----------+---------+------+------+--------------------------+
The query is taking 6 seconds to execute for 1000 records in t1 which is unacceptable for me. How can I rewrite this query using Joins (or perhaps a faster way if that exists) if I don't have a column to join on? Even a stored procedure is acceptable I guess in the worst case. My goal is to finally create a new table containing entries from t1. Any suggestions?
Unless the query optimizer is failing, a WHERE EXISTS construct should result in the same plan as a join with a GROUP clause. Look at optimizing MBRContains(geonetwork, GeomFromText(CONCAT('POINT(', t1.point, ' 0)')))), that's probably where your query is spending all its time. I don't have a suggestion for that, but here's your query written with a JOIN:
Select t1.point
from t1
join whitelist on MBRContains(whitelist.geonetwork, GeomFromText(CONCAT('POINT(', t1.point, ' 0)'))))
group by t1.point
;
or to get the points in t1 not in whitelist:
Select t1.point
from t1
left join whitelist on MBRContains(whitelist.geonetwork, GeomFromText(CONCAT('POINT(', t1.point, ' 0)'))))
where whitelist.id is null
;
This seems like a case where de-nomalizing t1 might be beneficial. Adding a GeomFrmTxt column with a value of GeomFromText(CONCAT('POINT(', t1.point, ' 0)')) could speed up the query you already have.

Eliminate full table scan due to BETWEEN (and GROUP BY)

Description
According to the explain command, there is a range that is causing a query to perform a full table scan (160k rows). How do I keep the range condition and reduce the scanning? I expect the culprit to be:
Y.YEAR BETWEEN 1900 AND 2009 AND
Code
Here is the code that has the range condition (the STATION_DISTRICT is likely superfluous).
SELECT
COUNT(1) as MEASUREMENTS,
AVG(D.AMOUNT) as AMOUNT,
Y.YEAR as YEAR,
MAKEDATE(Y.YEAR,1) as AMOUNT_DATE
FROM
CITY C,
STATION S,
STATION_DISTRICT SD,
YEAR_REF Y FORCE INDEX(YEAR_IDX),
MONTH_REF M,
DAILY D
WHERE
-- For a specific city ...
--
C.ID = 10663 AND
-- Find all the stations within a specific unit radius ...
--
6371.009 *
SQRT(
POW(RADIANS(C.LATITUDE_DECIMAL - S.LATITUDE_DECIMAL), 2) +
(COS(RADIANS(C.LATITUDE_DECIMAL + S.LATITUDE_DECIMAL) / 2) *
POW(RADIANS(C.LONGITUDE_DECIMAL - S.LONGITUDE_DECIMAL), 2)) ) <= 50 AND
-- Get the station district identification for the matching station.
--
S.STATION_DISTRICT_ID = SD.ID AND
-- Gather all known years for that station ...
--
Y.STATION_DISTRICT_ID = SD.ID AND
-- The data before 1900 is shaky; insufficient after 2009.
--
Y.YEAR BETWEEN 1900 AND 2009 AND
-- Filtered by all known months ...
--
M.YEAR_REF_ID = Y.ID AND
-- Whittled down by category ...
--
M.CATEGORY_ID = '003' AND
-- Into the valid daily climate data.
--
M.ID = D.MONTH_REF_ID AND
D.DAILY_FLAG_ID <> 'M'
GROUP BY
Y.YEAR
Update
The SQL is performing a full table scan, which results in MySQL performing a "copy to tmp table", as shown here:
+----+-------------+-------+--------+-----------------------------------+--------------+---------+-------------------------------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------+--------------+---------+-------------------------------+--------+-------------+
| 1 | SIMPLE | C | const | PRIMARY | PRIMARY | 4 | const | 1 | |
| 1 | SIMPLE | Y | range | YEAR_IDX | YEAR_IDX | 4 | NULL | 160422 | Using where |
| 1 | SIMPLE | SD | eq_ref | PRIMARY | PRIMARY | 4 | climate.Y.STATION_DISTRICT_ID | 1 | Using index |
| 1 | SIMPLE | S | eq_ref | PRIMARY | PRIMARY | 4 | climate.SD.ID | 1 | Using where |
| 1 | SIMPLE | M | ref | PRIMARY,YEAR_REF_IDX,CATEGORY_IDX | YEAR_REF_IDX | 8 | climate.Y.ID | 54 | Using where |
| 1 | SIMPLE | D | ref | INDEX | INDEX | 8 | climate.M.ID | 11 | Using where |
+----+-------------+-------+--------+-----------------------------------+--------------+---------+-------------------------------+--------+-------------+
Answer
After using the STRAIGHT_JOIN:
+----+-------------+-------+--------+-----------------------------------+---------------+---------+-------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------+---------------+---------+-------------------------------+------+---------------------------------+
| 1 | SIMPLE | C | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | S | ALL | PRIMARY | NULL | NULL | NULL | 7795 | Using where |
| 1 | SIMPLE | SD | eq_ref | PRIMARY | PRIMARY | 4 | climate.S.STATION_DISTRICT_ID | 1 | Using index |
| 1 | SIMPLE | Y | ref | PRIMARY,STAT_YEAR_IDX | STAT_YEAR_IDX | 4 | climate.S.STATION_DISTRICT_ID | 1650 | Using where |
| 1 | SIMPLE | M | ref | PRIMARY,YEAR_REF_IDX,CATEGORY_IDX | YEAR_REF_IDX | 8 | climate.Y.ID | 54 | Using where |
| 1 | SIMPLE | D | ref | INDEX | INDEX | 8 | climate.M.ID | 11 | Using where |
+----+-------------+-------+--------+-----------------------------------+---------------+---------+-------------------------------+------+---------------------------------+
Related
http://dev.mysql.com/doc/refman/5.0/en/how-to-avoid-table-scan.html
http://dev.mysql.com/doc/refman/5.0/en/where-optimizations.html
Optimize SQL that uses between clause
Thank you!
ONE Request... It looks like you KNOW your data. Add the keyword "STRAIGHT_JOIN" and see the results...
SELECT STRAIGHT_JOIN ... the rest of your query...
Straight-join tells MySql to DO IT AS I HAVE LISTED. So, your CITY table is the first in the FROM list, thus indicating you expect that to be your primary... Additionally, your WHERE clause of the CITY is the immediate filter. With that being said, it will probably fly through the rest of the query...
Hope it helps... Its worked for me with gov't data of millions of records queried and joined to 10+ lookup tables where mySql was trying to think for me.
in order to do efficient between queries you are going to want a b tree index on your YEAR column. for example:
CREATE INDEX id_index USING BTREE ON YEAR_REF (YEAR);
BTREE indexes allow for efficient range queries, if this is in fact the root problem then having an index like this should get rid of the full table scan and have it only scan the part of the table that is in the range. read more about btrees on wikipedia
However, as with any optimisation advice, you should measure to make sure that you don't do more harm than good.
Can you change from searching within a radius to search in a bounding box?
You know the city so you can calculate a bounding box in your application.
Perhaps this
S.LATITUDE_DECIMAL >= latitude_lower and
S.LATITUDE_DECIMAL <= latitude_upper and
S.LONGITUDE_DECIMAL >= longitude_lower and
S.LONGITUDE_DECIMAL <= longitude_upper
could be a little faster?

MySQL join with sort on datetime in joined table

I have 2 large mysql tables: Articles and ArticleTopics. I want to query the DB and retrieve the last 30 articles published for a given topicID. My current query is rather slow. Any ideas on how to improve it?
More details:
The tables:
Articles (~1 million rows)
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| articleId | int(11) | NO | PRI | NULL | auto_increment |
| title | varchar(255) | NO | | NULL | |
| content | longtext | NO | | NULL | |
| pubDate | datetime | NO | MUL | NULL | |
+-----------+--------------+------+-----+---------+----------------+
ArticleTopics (~10 million rows)
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| articleId | int(11) | NO | MUL | NULL | |
| topicId | int(11) | NO | MUL | NULL | |
+-----------+--------------+------+-----+---------+-------+
And my query:
SELECT a.articleId, a.pubDate
FROM Articles a, ArticleTopics t
WHERE t.articleId=a.articleId AND t.topicId=3364
ORDER BY a.pubDate DESC LIMIT 30;
And the EXPLAIN of the query:
+----+-------------+-------+--------+-------------------------------------+-------------------+---------+-------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-------------------------------------+-------------------+---------+-------------------+------+----------------------------------------------+
| 1 | SIMPLE | t | ref | articleId,topicId,topicId_articleId | topicId_articleId | 4 | const | 4281 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | a | eq_ref | PRIMARY,articleId_pubDate | PRIMARY | 4 | t.articleId | 1 | |
+----+-------------+-------+--------+-------------------------------------+-------------------+---------+-------------------+------+----------------------------------------------+
The slowness, I believe, is coming from the ORDER BY a.pubDate DESC. I can greatly improve performance by faking it a bit by instead doing an ORDER BY t.articleId DESC and having an index in ArticleTopics on both articleId & topicId, since in general, the articleIds are in the same order as pubDates. They are not always, however, so it's not ideal. I'd like to be able to sort it on the pubDate.
Update: Added EXPLAIN.
You can rewrite the query in various ways to see if it speeds things up:
SELECT a.articleId, a.pubDate
FROM Articles a
WHERE a.articleId in (
select articleId
from ArticleTopics
where topicId = 3364
)
ORDER BY a.pubDate DESC LIMIT 30;
Or:
SELECT a.articleId, a.pubDate
FROM Articles a
INNER JOIN ArticleTopics t ON t.articleId = a.articleId
WHERE t.topicId = 3364
ORDER BY a.pubDate DESC LIMIT 30;
The important index for both queries is on Articles, and contains articleId as first field.
If article is a large table, with say the entire PDF in binary, you can create an index that fully covers the query. Full coverage means all selected fields are part of the index. For this query, a fully covering index would be (articleId, pubDate).
At this point, do you have an index on topicId? If so, does the index contain only the topicId field?
And maybe you can post the output of the EXPLAIN query.