Slow postgres query with multiple joins to same table - sql
The following query is running very slowly for me:
SELECT r.comp,b.comp,n.comp
FROM paths AS p
INNER JOIN comps AS r ON (p.root=r.id)
INNER JOIN comps AS b ON (p.base=b.id)
INNER JOIN comps AS n ON (p.name=n.id);
Running EXPLAIN (BUFFERS,ANALYZE) gives the following result:
http://explain.depesz.com/s/iKG
Is it (re)building a hash for the comps table for each alias? Anything I can do to make this faster? Note: running two separate queries to join the data myself is faster.
Postgres version: 9.1.9
Machine: Ubuntu 12.04 8 | 4-core Xeon 2.5Ghz | 8GB of RAM
archiving=> \d+ comps
Table "public.comps"
Column | Type | Modifiers | Storage | Description
--------+--------+----------------------------------------------------+----------+-------------
id | bigint | not null default nextval('comps_id_seq'::regclass) | plain |
comp | text | not null | extended |
Indexes:
"comps_pkey" PRIMARY KEY, btree (id)
"comps_comp_key" UNIQUE CONSTRAINT, btree (comp)
"comps_comp_idx" btree (comp)
"comps_id_idx" btree (id)
Has OIDs: no
PostgreSQL does not have any special optimizations for seljoin - so it is expected behavior.
Related
How can I speed up queries with `GROUP BY` in them?
Details: MariaDB: Server version: 10.2.10-MariaDB MariaDB Server The DB table, trans_tbl is using Aria DB engine Table is somewhat large: 126,006,123 rows Server is not at all large: AWS t3 micro w/attached 30GB EBS I applied indexes to this DB table as follows: A primary key: evt_id Another index on the column I want to group by: transaction_type 3 Related Questions: Why is the transaction_type index ignored when I perform the following? SELECT COUNT(evt_id), transaction_type FROM trans_tbl GROUP BY transaction_type If I look at the output from EXPLAIN, I see: MariaDB [my_db]> EXPLAIN SELECT COUNT(evt_id), transaction_type FROM trans_tbl GROUP BY transaction_type; +------+-------------+-----------+------+---------------+------+---------+------+-----------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-----------+------+---------------+------+---------+------+-----------+---------------------------------+ | 1 | SIMPLE | trans_tbl | ALL | NULL | NULL | NULL | NULL | 126006123 | Using temporary; Using filesort | +------+-------------+-----------+------+---------------+------+---------+------+-----------+---------------------------------+ What's confusing me here is that both of the items in the query are indexed. So, shouldn't the index(es) be utilized? Why is the transaction_type index being used in the following case, where all I've done is switched from COUNT(evt_id) -- the primary key -- to COUNT(1). (The column is transaction_type, the index generated from it is called TransType.) MariaDB [my_db]> EXPLAIN SELECT COUNT(1), transaction_type FROM trans_tbl GROUP BY transaction_type; +------+-------------+-----------+-------+---------------+-----------+---------+------+-----------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-----------+-------+---------------+-----------+---------+------+-----------+-------------+ | 1 | SIMPLE | trans_tbl | index | NULL | TransType | 35 | NULL | 126006123 | Using index | +------+-------------+-----------+-------+---------------+-----------+---------+------+-----------+-------------+ The first query (with COUNT(evt_id)) takes 2 minutes & 40 seconds. Since it is not using the indices, that makes sense. But the second query (with COUNT(1)) takes 50 seconds. This makes no sense to me. Shouldn't it take essentially 0 seconds? Can't it just look at the first and last index value of each group, subtract them, and have the count? It seems to me that it is indeed actually counting. What's the point of an index? I guess my more important question is: How do I set up my indexes to allow for grouping on that index to return results almost instantaneously, as I would expect? PS I know the machine is ridiculously underpowered for this size of DB table. But, the table data is not worth throwing a lot of money at it to improve performance. I'd rather just learn to implement Aria indexes properly to gain speed.
COUNT(x) checks x for being NOT NULL before counting the row. COUNT(*) is the usual pattern for counting rows. So... SELECT COUNT(evt_id), transaction_t is just `SELECT FIND_IN_SET(17, '8,12,17,90');`ype FROM trans_tbl GROUP BY transaction_type; decided to do a table scan, then sort and group. SELECT COUNT(*), transaction_type FROM trans_tbl GROUP BY transaction_type; saw INDEX(transaction_type) and said "goodie; I can just scan that index without having to sort." Note: It still has to scan in order to count. But the INDEX is smaller than the table, so it could be done faster. This is also called a "covering" index since all the columns needed in the SELECT are found in that one INDEX. COUNT(1) might be treated the same as COUNT(*), I don't know. INDEX(transaction_type) is essentially identical to INDEX(transaction_type, evt_id). This because the PRIMARY KEY is silently tacked onto any secondary key in InnoDB. I don't know why INDEX(transaction_type, evt_id) was not used. Bottom line: Use COUNT(*). Why not 0 seconds? The counts are not saved anywhere. Anyway, there could be other queries modifying the counts as you run you SELECT. The improvement came from scanning 126M 2-column rows instead of 126M multi-column rows.
ORDER BY [PRIMARY_KEY] has to apply sort-order when it should simply use the index?
according to my Research, ordering by the Primary key (or on any other column with an index) - the query should run without an explicit sort. I also found a blog where this behavior was shown on different databases, one of them being Oracle. However - in my Tests it this was not true - what could be the reason? Bad install-options? Broken Index? (although I ruled that out by creating a completely new table) the query: select * from auftrag_test order by auftragkey the execution plan: Plan Hash Value : 505195503 ----------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost | Time | ----------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 167910 | 44496150 | 11494 | 00:00:01 | | 1 | SORT ORDER BY | | 167910 | 44496150 | 11494 | 00:00:01 | | 2 | TABLE ACCESS FULL | AUFTRAG_TEST | 167910 | 44496150 | 1908 | 00:00:01 | ----------------------------------------------------------------------------------- create table AUFTRAG_TEST ( auftragkey VARCHAR2(40) not null, ... ); alter table AUFTRAG_TEST add constraint PK_AUFTRAG_TEST primary key (AUFTRAGKEY); you might ask yourself why the Primary key would be a varchar field. Well, this is something our bosses have decided. (Actually we put in stringified guids) The blog I found: http://use-the-index-luke.com/sql/sorting-grouping/indexed-order-by P.S.: I think that I found out the Problem. This select does NOT "order by": select * from auftrag_test where auftragkey = 'aabbccddeeffaabbccddeeffaabbccdd' order by auftragkey So - apparently - it does ONLY work, if you filter against an index, with "equality" which wouldn't be very helpful at all. P.P.S: MS-SQL seems to do just what I expected. If I order by the Primary key (with a non clustered unique index) - the sort is "free". In execution plan, and also query time wise.
You should be aware that scanning a big table through an index might take hours Vs. full table scan on the same table that will take only few minutes. In this case travesing through the index is order to save a O(n*log(n)) sort operation, doesn't sound like a good idea. Heap table will yield sort operation. IOT (Index orginized Table, also knows as "clustered index") is already sorted. create table t_heap (i int primary key,j int); create table t_iot (i int primary key,j int) organization index; select * from t_heap order by i; select * from t_iot order by i;
SQL Query 2 tables null results
I was asked this question in an interview: From the 2 tables below, write a query to pull customers with no sales orders. How many ways to write this query and which would have best performance. Table 1: Customer - CustomerID Table 2: SalesOrder - OrderID, CustomerID, OrderDate Query: SELECT * FROM Customer C RIGHT OUTER JOIN SalesOrder SO ON C.CustomerID = SO.CustomerID WHERE SO.OrderID = NULL Is my query correct and are there other ways to write the query and get the same results?
Answering for MySQL instead of SQL Server, cause you tagged it later with SQL Server, so I thought (since this was an interview question, that it wouldn't bother you, for which DBMS this is). Note though, that the queries I wrote are standard sql, they should run in every RDBMS out there. How each RDBMS handles those queries is another issue, though. I wrote this little procedure for you, to have a test case. It creates the tables customers and orders like you specified and I added primary keys and foreign keys, like one would usually do it. No other indexes, as every column worth indexing here is already primary key. 250 customers are created, 100 of them made an order (though out of convenience none of them twice / multiple times). A dump of the data follows, posted the script just in case you want to play around a little by increasing the numbers. delimiter $$ create procedure fill_table() begin create table customers(customerId int primary key) engine=innodb; set #x = 1; while (#x <= 250) do insert into customers values(#x); set #x := #x + 1; end while; create table orders(orderId int auto_increment primary key, customerId int, orderDate timestamp, foreign key fk_customer (customerId) references customers(customerId) ) engine=innodb; insert into orders(customerId, orderDate) select customerId, now() - interval customerId day from customers order by rand() limit 100; end $$ delimiter ; call fill_table(); For me, this resulted in this: CREATE TABLE `customers` ( `customerId` int(11) NOT NULL, PRIMARY KEY (`customerId`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; INSERT INTO `customers` VALUES (1),(2),(3),(4),(5),(6),(7),(8),(9),(10),(11),(12),(13),(14),(15),(16),(17),(18),(19),(20),(21),(22),(23),(24),(25),(26),(27),(28),(29),(30),(31),(32),(33),(34),(35),(36),(37),(38),(39),(40),(41),(42),(43),(44),(45),(46),(47),(48),(49),(50),(51),(52),(53),(54),(55),(56),(57),(58),(59),(60),(61),(62),(63),(64),(65),(66),(67),(68),(69),(70),(71),(72),(73),(74),(75),(76),(77),(78),(79),(80),(81),(82),(83),(84),(85),(86),(87),(88),(89),(90),(91),(92),(93),(94),(95),(96),(97),(98),(99),(100),(101),(102),(103),(104),(105),(106),(107),(108),(109),(110),(111),(112),(113),(114),(115),(116),(117),(118),(119),(120),(121),(122),(123),(124),(125),(126),(127),(128),(129),(130),(131),(132),(133),(134),(135),(136),(137),(138),(139),(140),(141),(142),(143),(144),(145),(146),(147),(148),(149),(150),(151),(152),(153),(154),(155),(156),(157),(158),(159),(160),(161),(162),(163),(164),(165),(166),(167),(168),(169),(170),(171),(172),(173),(174),(175),(176),(177),(178),(179),(180),(181),(182),(183),(184),(185),(186),(187),(188),(189),(190),(191),(192),(193),(194),(195),(196),(197),(198),(199),(200),(201),(202),(203),(204),(205),(206),(207),(208),(209),(210),(211),(212),(213),(214),(215),(216),(217),(218),(219),(220),(221),(222),(223),(224),(225),(226),(227),(228),(229),(230),(231),(232),(233),(234),(235),(236),(237),(238),(239),(240),(241),(242),(243),(244),(245),(246),(247),(248),(249),(250); CREATE TABLE `orders` ( `orderId` int(11) NOT NULL AUTO_INCREMENT, `customerId` int(11) DEFAULT NULL, `orderDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`orderId`), KEY `fk_customer` (`customerId`), CONSTRAINT `orders_ibfk_1` FOREIGN KEY (`customerId`) REFERENCES `customers` (`customerId`) ) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8; INSERT INTO `orders` VALUES (1,247,'2013-06-24 19:50:07'),(2,217,'2013-07-24 19:50:07'),(3,8,'2014-02-18 20:50:07'),(4,40,'2014-01-17 20:50:07'),(5,52,'2014-01-05 20:50:07'),(6,80,'2013-12-08 20:50:07'),(7,169,'2013-09-10 19:50:07'),(8,135,'2013-10-14 19:50:07'),(9,115,'2013-11-03 20:50:07'),(10,225,'2013-07-16 19:50:07'),(11,112,'2013-11-06 20:50:07'),(12,243,'2013-06-28 19:50:07'),(13,158,'2013-09-21 19:50:07'),(14,24,'2014-02-02 20:50:07'),(15,214,'2013-07-27 19:50:07'),(16,25,'2014-02-01 20:50:07'),(17,245,'2013-06-26 19:50:07'),(18,182,'2013-08-28 19:50:07'),(19,166,'2013-09-13 19:50:07'),(20,69,'2013-12-19 20:50:07'),(21,85,'2013-12-03 20:50:07'),(22,44,'2014-01-13 20:50:07'),(23,103,'2013-11-15 20:50:07'),(24,19,'2014-02-07 20:50:07'),(25,33,'2014-01-24 20:50:07'),(26,102,'2013-11-16 20:50:07'),(27,41,'2014-01-16 20:50:07'),(28,94,'2013-11-24 20:50:07'),(29,43,'2014-01-14 20:50:07'),(30,150,'2013-09-29 19:50:07'),(31,218,'2013-07-23 19:50:07'),(32,131,'2013-10-18 19:50:07'),(33,77,'2013-12-11 20:50:07'),(34,2,'2014-02-24 20:50:07'),(35,45,'2014-01-12 20:50:07'),(36,230,'2013-07-11 19:50:07'),(37,101,'2013-11-17 20:50:07'),(38,31,'2014-01-26 20:50:07'),(39,56,'2014-01-01 20:50:07'),(40,176,'2013-09-03 19:50:07'),(41,223,'2013-07-18 19:50:07'),(42,145,'2013-10-04 19:50:07'),(43,26,'2014-01-31 20:50:07'),(44,62,'2013-12-26 20:50:07'),(45,195,'2013-08-15 19:50:07'),(46,153,'2013-09-26 19:50:07'),(47,179,'2013-08-31 19:50:07'),(48,104,'2013-11-14 20:50:07'),(49,7,'2014-02-19 20:50:07'),(50,209,'2013-08-01 19:50:07'),(51,86,'2013-12-02 20:50:07'),(52,110,'2013-11-08 20:50:07'),(53,204,'2013-08-06 19:50:07'),(54,187,'2013-08-23 19:50:07'),(55,114,'2013-11-04 20:50:07'),(56,38,'2014-01-19 20:50:07'),(57,236,'2013-07-05 19:50:07'),(58,79,'2013-12-09 20:50:07'),(59,96,'2013-11-22 20:50:07'),(60,37,'2014-01-20 20:50:07'),(61,207,'2013-08-03 19:50:07'),(62,22,'2014-02-04 20:50:07'),(63,120,'2013-10-29 20:50:07'),(64,200,'2013-08-10 19:50:07'),(65,51,'2014-01-06 20:50:07'),(66,181,'2013-08-29 19:50:07'),(67,4,'2014-02-22 20:50:07'),(68,123,'2013-10-26 19:50:07'),(69,108,'2013-11-10 20:50:07'),(70,55,'2014-01-02 20:50:07'),(71,76,'2013-12-12 20:50:07'),(72,6,'2014-02-20 20:50:07'),(73,18,'2014-02-08 20:50:07'),(74,211,'2013-07-30 19:50:07'),(75,53,'2014-01-04 20:50:07'),(76,216,'2013-07-25 19:50:07'),(77,32,'2014-01-25 20:50:07'),(78,74,'2013-12-14 20:50:07'),(79,138,'2013-10-11 19:50:07'),(80,197,'2013-08-13 19:50:07'),(81,221,'2013-07-20 19:50:07'),(82,118,'2013-10-31 20:50:07'),(83,61,'2013-12-27 20:50:07'),(84,28,'2014-01-29 20:50:07'),(85,16,'2014-02-10 20:50:07'),(86,39,'2014-01-18 20:50:07'),(87,3,'2014-02-23 20:50:07'),(88,46,'2014-01-11 20:50:07'),(89,189,'2013-08-21 19:50:07'),(90,59,'2013-12-29 20:50:07'),(91,249,'2013-06-22 19:50:07'),(92,127,'2013-10-22 19:50:07'),(93,47,'2014-01-10 20:50:07'),(94,178,'2013-09-01 19:50:07'),(95,141,'2013-10-08 19:50:07'),(96,188,'2013-08-22 19:50:07'),(97,220,'2013-07-21 19:50:07'),(98,15,'2014-02-11 20:50:07'),(99,175,'2013-09-04 19:50:07'),(100,206,'2013-08-04 19:50:07'); Okay, now to the queries. Three ways came to my mind, I omitted the right join that MDiesel did, because it's actually just another way of writing left join. It was invented for lazy sql developers, that don't want to switch table names, but instead just rewrite one word. Anyway, first query: select c.* from customers c left join orders o on c.customerId = o.customerId where o.customerId is null; Results in an execution plan like this: +----+-------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ | 1 | SIMPLE | c | index | NULL | PRIMARY | 4 | NULL | 250 | Using index | | 1 | SIMPLE | o | ref | fk_customer | fk_customer | 5 | wtf.c.customerId | 1 | Using where; Using index | +----+-------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ Second query: select c.* from customers c where c.customerId not in (select distinct customerId from orders); Results in an execution plan like this: +----+--------------------+--------+----------------+---------------+-------------+---------+------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+--------+----------------+---------------+-------------+---------+------+------+--------------------------+ | 1 | PRIMARY | c | index | NULL | PRIMARY | 4 | NULL | 250 | Using where; Using index | | 2 | DEPENDENT SUBQUERY | orders | index_subquery | fk_customer | fk_customer | 5 | func | 2 | Using index | +----+--------------------+--------+----------------+---------------+-------------+---------+------+------+--------------------------+ Third query: select c.* from customers c where not exists (select 1 from orders o where o.customerId = c.customerId); Results in an execution plan like this: +----+--------------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+--------------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ | 1 | PRIMARY | c | index | NULL | PRIMARY | 4 | NULL | 250 | Using where; Using index | | 2 | DEPENDENT SUBQUERY | o | ref | fk_customer | fk_customer | 5 | wtf.c.customerId | 1 | Using where; Using index | +----+--------------------+-------+-------+---------------+-------------+---------+------------------+------+--------------------------+ We can see in all execution plans, that the customers table is read as a whole, but from the index (the implicit one as the only column is primary key). This may change, when you select other columns from the table, that are not in an index. The first one seems to be the best. For each row in customers only one row in orders is read. The id column suggests, that MySQL can do this in one step, as only indexes are involved. The second query seems to be the worst (though all 3 queries shouldn't perform too bad). For each row in customers the subquery is executed (the select_type column tells this). The third query is not much different in that it uses a dependent subquery, but should perform better than the second query. Explaining the small differences would lead to far now. If you're interested, here's the manual page that explains what each column and their values mean here: EXPLAIN output Finally: I'd say, that the first query will perform best, but as always, in the end one has to measure, to measure and to measure.
I can thing of two other ways to write this query: SELECT C.* FROM Customer C LEFT OUTER JOIN SalesOrder SO ON C.CustomerID = SO.CustomerID WHERE SO.CustomerID IS NULL SELECT C.* FROM Customer C WHERE NOT C.CustomerID IN(SELECT CustomerID FROM SalesOrder)
The solutions involving outer joins will perform better than a solution using NOT IN.
How can I speed up a count(*) which is already using indexes? (MyISAM)
I have a 3 large tables (10k, 10k, and 100M rows) and am trying to do a simple count on a join of them, where all the joined columns are indexed. Why does the COUNT(*) take so long, and how can I speed it up (without triggers and a running summary)? mysql> describe SELECT COUNT(*) FROM `metaward_alias` INNER JOIN `metaward_achiever` ON (`metaward_alias`.`id` = `metaward_achiever`.`alias_id`) INNER JOIN `metaward_award` ON (`metaward_achiever`.`award_id` = `metaward_award`.`id`) WHERE `metaward_award`.`owner_id` = 8; +----+-------------+-------------------+--------+-------------------------------------------------------+----------------------------+---------+---------------------------------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------------------+--------+-------------------------------------------------------+----------------------------+---------+---------------------------------+------+-------------+ | 1 | SIMPLE | metaward_award | ref | PRIMARY,metaward_award_owner_id | metaward_award_owner_id | 4 | const | 1552 | | | 1 | SIMPLE | metaward_achiever | ref | metaward_achiever_award_id,metaward_achiever_alias_id | metaward_achiever_award_id | 4 | paul.metaward_award.id | 2498 | | | 1 | SIMPLE | metaward_alias | eq_ref | PRIMARY | PRIMARY | 4 | paul.metaward_achiever.alias_id | 1 | Using index | +----+-------------+-------------------+--------+-------------------------------------------------------+----------------------------+---------+---------------------------------+------+-------------+ 3 rows in set (0.00 sec) But actually running the query takes about 10 minutes, and I'm on MyISAM so the tables are fully locked down for that duration
I guess the reason is that you do a huge join over three tables (without applying where clause first, the result would be 10k * 10k * 100M = 1016 rows). Try to reorder joins (for example start with metaward_award, then join only metaward_achiever see how long that takes, then try to plug metaward_alias, possibly using subquery to force your preferred evaluation order). If that does not help you might have to denormalize your data, for example by storing number of aliases for particular metaward_achiever. Then you'd get rid of one join altogether. Maybe you can even cache the sums for metaward_award, depending on how and how often is your data updated. Other thing that might help is getting all your database content into RAM :-)
Make sure you have indexes on: metaward_alias id metaward_achiever alias_id metaward_achiever award_id metaward_award id metaward_award owner_id I'm sure many people will also suggest to count on a specific column, but in MySql this doesn't make any difference for your query. UPDATE: You could also try to set the condition on the main table instead of one of the joined tables. That would give you the same result, but it could be faster (I don't know how clever MySql is): SELECT COUNT(*) FROM `metaward_award` INNER JOIN `metaward_achiever` ON (`metaward_achiever`.`award_id` = `metaward_award`.`id`) INNER JOIN `metaward_alias` ON (`metaward_alias`.`id` = `metaward_achiever`.`alias_id`) WHERE `metaward_award`.`owner_id` = 8
10 minutes is way too long for that query. I think you must have a really small key cache. You can get its size in bytes with: SELECT ##key_buffer_size First off, you should run ANALYZE TABLE or OPTIMIZE TABLE. They'll sort your index and can slightly improve the performance. You should also see if you can use more compact types for your columns. For instance, if you're not going to have more than 16 millions owners or awards or aliases, you can change your INT columns into MEDIUMINT (UNSIGNED, of course). Perhaps even SMALLINT in some cases? That will reduce your index footprint and you'll fit more of it in the cache.
MySQL, reading this EXPLAIN statement
I have a query which is starting to cause some concern in my application. I'm trying to understand this EXPLAIN statement better to understand where indexes are potentially missing: +----+-------------+-------+--------+---------------+------------+---------+-------------------------------+------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+--------+---------------+------------+---------+-------------------------------+------+---------------------------------+ | 1 | SIMPLE | s | ref | client_id | client_id | 4 | const | 102 | Using temporary; Using filesort | | 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | www_foo_com.s.user_id | 1 | | | 1 | SIMPLE | a | ref | session_id | session_id | 4 | www_foo_com.s.session_id | 1 | Using index | | 1 | SIMPLE | h | ref | email_id | email_id | 4 | www_foo_com.a.email_id | 10 | Using index | | 1 | SIMPLE | ph | ref | session_id | session_id | 4 | www_foo_com.s.session_id | 1 | Using index | | 1 | SIMPLE | em | ref | session_id | session_id | 4 | www_foo_com.s.session_id | 1 | | | 1 | SIMPLE | pho | ref | session_id | session_id | 4 | www_foo_com.s.session_id | 1 | | | 1 | SIMPLE | c | ALL | userfield | NULL | NULL | NULL | 1108 | | +----+-------------+-------+--------+---------------+------------+---------+-------------------------------+------+---------------------------------+ 8 rows in set (0.00 sec) I'm trying to understand where my indexes are missing by reading this EXPLAIN statement. Is it fair to say that one can understand how to optimize this query without seeing the query at all and just look at the results of the EXPLAIN? It appears that the ALL scan against the 'c' table is the achilles heel. What's the best way to index this based on constant values as recommended on MySQL's documentation? | Note, I also added an index to userfield in the cdr table and that hasn't done much good either. Thanks. --- edit --- Here's the query, sorry -- don't know why I neglected to include it the first pass through. SELECT s.`session_id` id, DATE_FORMAT(s.`created`,'%m/%d/%Y') date, u.`name`, COUNT(DISTINCT c.id) calls, COUNT(DISTINCT h.id) emails, SEC_TO_TIME(MAX(DISTINCT c.duration)) duration, (COUNT(DISTINCT em.email_id) + COUNT(DISTINCT pho.phone_id) > 0) status FROM `fa_sessions` s LEFT JOIN `fa_users` u ON s.`user_id`=u.`user_id` LEFT JOIN `fa_email_aliases` a ON a.session_id = s.session_id LEFT JOIN `fa_email_headers` h ON h.email_id = a.email_id LEFT JOIN `fa_phones` ph ON ph.session_id = s.session_id LEFT JOIN `fa_email_aliases` em ON em.session_id = s.session_id AND em.status = 1 LEFT JOIN `fa_phones` pho ON pho.session_id = s.session_id AND pho.status = 1 LEFT JOIN `cdr` c ON c.userfield = ph.phone_id WHERE s.`partner_id`=1 GROUP BY s.`session_id`
I assume you've looked here to get more info about what it is telling you. Obviously the ALL means its going through all of them. The using temporary and using filesort are talked about on that page. You might want to look at that. From the page: Using filesort MySQL must do an extra pass to find out how to retrieve the rows in sorted order. The sort is done by going through all rows according to the join type and storing the sort key and pointer to the row for all rows that match the WHERE clause. The keys then are sorted and the rows are retrieved in sorted order. See Section 7.2.12, “ORDER BY Optimization”. Using temporary To resolve the query, MySQL needs to create a temporary table to hold the result. This typically happens if the query contains GROUP BY and ORDER BY clauses that list columns differently. I agree that seeing the query might help to figure things out better.
My advice? Break the query into 2 and use a temporary table in the middle. Reasonning The problem appears to be that table c is being table scanned, and that this is the last table in the query. This is probably bad: if you have a table scan, you want to do it at the start of the query, so it's only done once. I'm not a MySQL guru, but I have spent a whole lot of time optimising queries on other DBs. It looks to me like the optimiser hasn't worked out that it should start with c and work backwards. The other thing that strikes me is that there are probably too many tables in the join. Most optimisers struggle with more than 4 tables (because the number of possible table orders is growing exponentially, so checking them all becomes impractical). Having too many tables in a join is the root of 90% of performance problems I have seen. Give it a go, and let us know how you get on. If it doesn't help, please post the SQL, table definitions and indeces, and I'll take another look. General Tips Feel free to look at this answer I gave on general performance tips. A great resource MySQL Documentation for EXPLAIN
Well looking at the query would be useful, but there's at least one thing that's obviously worth looking into - the final line shows the ALL type for that part of the query, which is generally not great to see. If the suggested possible key (userfield) makes sense as an added index to table c, it might be worth adding it and seeing if that reduces the rows returned for that table in the search.
Query Plan The query plan we might hope the optimiser would choose would be something like: start with sessions where partner_id=1 , possibly using an index on partner_id, join sessions to users, using an index on user_id join sessions to phones, where status=1, using an index on session_id and possibly status join sessions to phones again using an index on session_id and phone_id ** join phones to cdr using an index on userfield join sessions to email_aliases, where status=1 using an index on session_id and possibly status join sessions to email_aliases again using an index on session_id and email_id ** join email_aliases to email_headers using an index on email_id ** by putting 2 fields in these indeces, we enable the optimiser to join to the table using session_id, and immediately find out the associated phone_id or email_id without having to read the underlying table. This technique saves us a read, and can save a lot of time. Indeces I would create: The above query plan suggests these indeces: fa_sessions ( partner_id, session_id ) fa_users ( user_id ) fa_email_aliases ( session_id, email_id ) fa_email_headers ( email_id ) fa_email_aliases ( session_id, status ) fa_phones ( session_id, status, phone_id ) cdr ( userfield ) Notes You will almost certainly get acceptable performance without creating all of these. If any of the tables are small ( less than 100 rows ) then it's probably not worth creating an index. fa_email_aliases might work with ( session_id, status, email_id ), depending on how the optimiser works.