(Using MySQL 4.1.22)
I can't get this query of mine to use an index on a large table(200k+ rows), it is doing a full table scan on it. The query takes about 1.2 seconds right now. I want to get it to less than .2 seconds if possible.
Here is my query:
SELECT st_issues.issue_id, st_issues.cat_id,st_categories.name AS cat_name, st_issues.status_id,st_statuses.name AS status_name, st_issues.priority_id,st_priorities.name AS priority_name,st_priorities.color AS color, st_issues.assigned_cid,assigned_u.firstname,assigned_u.lastname,assigned_u.screenname, message, rating, created_by_email,created_by_cid,created_by_uid,by_user.firstname AS by_firstname,by_user.lastname AS by_lastname,by_user.screenname AS by_screenname, st_issues.browser,from_url,created_by_store,created,st_issues.stamp
FROM st_issues
JOIN st_categories ON (st_issues.cat_id=st_categories.cat_id)
JOIN st_statuses ON (st_issues.status_id=st_statuses.status_id)
JOIN st_priorities ON (st_issues.priority_id=st_priorities.priority_id)
LEFT JOIN users AS assigned_u ON (assigned_u.cid=st_issues.assigned_cid)
LEFT JOIN users AS by_user ON (by_user.uid=st_issues.created_by_uid)
LEFT JOIN st_issue_changes ON (st_issues.issue_id=st_issue_changes.issue_id AND change_id=0)
WHERE st_issues.assigned_cid=0
The results of explain:
1, 'SIMPLE', 'st_issues', 'ALL', '', '', , '', 4, 'Using where'
1, 'SIMPLE', 'st_categories', 'eq_ref', 'PRIMARY', 'PRIMARY', 1, 'sg.st_issues.cat_id', 1, ''
1, 'SIMPLE', 'st_priorities', 'eq_ref', 'PRIMARY', 'PRIMARY', 1, 'sg.st_issues.priority_id', 1, ''
1, 'SIMPLE', 'assigned_u', 'ref', 'cid', 'cid', 8, 'sg.st_issues.assigned_cid', 1, ''
1, 'SIMPLE', 'st_statuses', 'ALL', 'PRIMARY', '', , '', 4, 'Using where'
1, 'SIMPLE', 'by_user', 'ALL', '', '', , '', 221623, ''
1, 'SIMPLE', 'st_issue_changes', 'eq_ref', 'PRIMARY', 'PRIMARY', 6, 'sg.st_issues.issue_id,const', 1, ''
Obviously the problem is with the join on 'by_user' since it isn't using an index.
Here is some of the definition of the 'users' table:
CREATE TABLE `users` (
`cid` double unsigned NOT NULL auto_increment,
`uid` varchar(20) NOT NULL default '',
...
`firstname` varchar(20) default NULL,
`lastname` varchar(20) default NULL,
...
PRIMARY KEY (`uid`),
...
) ENGINE=InnoDB
Anyone have any ideas of why it is not using the primary key in the join?
Anyone have any ideas or hints of how to speed up this query more?
(I can add the table definitions of the other tables if needed/wanted)
Edit:
Here is the table definition for st_issues:
CREATE TABLE `st_issues` (
`issue_id` int(10) unsigned NOT NULL auto_increment,
`cat_id` tinyint(3) unsigned NOT NULL default '0',
`status_id` tinyint(3) unsigned NOT NULL default '0',
`priority_id` tinyint(3) unsigned NOT NULL default '0',
`assigned_cid` int(10) unsigned NOT NULL default '0',
`rating` tinyint(4) default NULL,
`created_by_email` varchar(255) NOT NULL default '',
`created_by_cid` int(10) unsigned NOT NULL default '0',
`created_by_uid` varchar(20) NOT NULL default '',
`created_by_store` tinyint(3) unsigned NOT NULL default '0',
`browser` varchar(255) NOT NULL default '',
`from_url` varchar(255) NOT NULL default '',
`created` datetime NOT NULL default '0000-00-00 00:00:00',
`stamp` datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (`issue_id`),
KEY `idx_create_by_cid` (`created_by_cid`),
KEY `idx_create_by_uid` (`created_by_uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Is that the whole of the definition of the users table?
Because it says:
) ENGINE=InnoDB
whereas st_issues says:
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
If your two tables are using different collations, the two string datatypes for uid and created_by_uid are different, and MySQL must do a character set coercion before it can compare them, thus defeating your index.
It's always best to ensure you use the same character set/collation for all text in your database.
I did some testing and found the following changes helped:
Add index on st_issues.assigned_cid.
Change primary key of users table to cid instead of uid.
Change join condition for by_user to use cid instead of uid:
LEFT JOIN users AS by_user ON (by_user.cid=st_issues.created_by_cid)
Then I got the following EXPLAIN report (though with zero rows of data):
+----+-------------+------------------+--------+---------------+--------------+---------+-------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+--------+---------------+--------------+---------+-------------------------------+------+-------------+
| 1 | SIMPLE | st_issues | ref | assigned_cid | assigned_cid | 4 | const | 1 | |
| 1 | SIMPLE | st_categories | eq_ref | PRIMARY | PRIMARY | 1 | test.st_issues.cat_id | 1 | |
| 1 | SIMPLE | st_statuses | eq_ref | PRIMARY | PRIMARY | 1 | test.st_issues.status_id | 1 | |
| 1 | SIMPLE | st_priorities | eq_ref | PRIMARY | PRIMARY | 1 | test.st_issues.priority_id | 1 | |
| 1 | SIMPLE | assigned_u | eq_ref | PRIMARY | PRIMARY | 8 | test.st_issues.assigned_cid | 1 | |
| 1 | SIMPLE | by_user | eq_ref | PRIMARY | PRIMARY | 8 | test.st_issues.created_by_cid | 1 | |
| 1 | SIMPLE | st_issue_changes | eq_ref | PRIMARY | PRIMARY | 8 | test.st_issues.issue_id,const | 1 | Using index |
+----+-------------+------------------+--------+---------------+--------------+---------+-------------------------------+------+-------------+
This shows that the optimizer has selected an index for each table, which it didn't in your version of the query. I had to guess on the definition for your lookup tables.
Another thing I would suggest is to define your lookup tables st_categories and st_statuses with a natural key, the name of the category or status. Then reference that natural key from the st_issues table, instead of using a tinyint pseudokey. The advantage is that you don't have to perform those joins to get the name of the category or status; it's already in the st_issues table.
Related
I have written Perl code that parses a text file and uses a hash to tally up the number of times a US State abbreviation appears in each file/record. I end up with something like this.
File: 521
OH => 4
PA => 1
IN => 2
TX => 3
IL => 7
I am struggling to find a way to store such hash results in an SQL database. I am using mariadb. Because the structure of the data itself varies, one file will have some states and the next may have others. For example, one file may contain only a few states, the next may contain a group of completely different states. I am even having trouble conceptualizing the table structure. What would be the best way to store data like this in a database?
There are many possible ways to store the data.
For sake of simplicity see if the following approach will be an acceptable solution for your case. The solution is base on use one table with two indexes based upon id and state columns.
CREATE TABLE IF NOT EXISTS `state_count` (
`id` INT NOT NULL,
`state` VARCHAR(2) NOT NULL,
`count` INT NOT NULL,
INDEX `id` (`id`),
INDEX `state` (`state`)
);
INSERT INTO `state_count`
(`id`,`state`,`count`)
VALUES
('251','OH',4),
('251','PA',1),
('251','IN',2),
('251','TX',3),
('251','IL',7);
Sample SQL SELECT output
MySQL [dbs0897329] > SELECT * FROM state_count;
+-----+-------+-------+
| id | state | count |
+-----+-------+-------+
| 251 | OH | 4 |
| 251 | PA | 1 |
| 251 | IN | 2 |
| 251 | TX | 3 |
| 251 | IL | 7 |
+-----+-------+-------+
5 rows in set (0.000 sec)
MySQL [dbs0897329]> SELECT * FROM state_count WHERE state='OH';
+-----+-------+-------+
| id | state | count |
+-----+-------+-------+
| 251 | OH | 4 |
+-----+-------+-------+
1 row in set (0.000 sec)
MySQL [dbs0897329]> SELECT * FROM state_count WHERE state IN ('OH','TX');
+-----+-------+-------+
| id | state | count |
+-----+-------+-------+
| 251 | OH | 4 |
| 251 | TX | 3 |
+-----+-------+-------+
2 rows in set (0.001 sec)
It's a little unclear in what direction your question goes. But if you want a good relational model to store the data into, that would be three tables. One for the files. One for the states. One for the count of the states in a file. For example:
The tables:
CREATE TABLE file
(id integer
AUTO_INCREMENT,
path varchar(256)
NOT NULL,
PRIMARY KEY (id),
UNIQUE (path));
CREATE TABLE state
(id integer
AUTO_INCREMENT,
abbreviation varchar(2)
NOT NULL,
PRIMARY KEY (id),
UNIQUE (abbreviation));
CREATE TABLE occurrences
(file integer,
state integer,
count integer
NOT NULL,
PRIMARY KEY (file,
state),
FOREIGN KEY (file)
REFERENCES file
(id),
FOREIGN KEY (state)
REFERENCES state
(id),
CHECK (count >= 0));
The data:
INSERT INTO files
(path)
VALUES ('521');
INSERT INTO states
(abbreviation)
VALUES ('OH'),
('PA'),
('IN'),
('TX'),
('IL');
INSERT INTO occurrences
(file,
state,
count)
VALUES (1,
1,
4),
(1,
2,
1),
(1,
3,
2),
(1,
4,
3),
(1,
4,
7);
The states of course would be reused. Fill the table with all 50 and use them. They should not be inserted for every file again.
You can fill occurrences explicitly with a count of 0 for file where the respective state didn't appear, if you want to distinguish between "I know it's 0." and "I don't know the count.", which would then be encoded through the absence of a corresponding row. If you don't want to distinguish that and no row means a count of 0, you can handle that in queries by using outer joins and coalesce() to "translate" to 0.
I suppose it is quite simple, I just can't get the hang of it.
At the moment I have the following code:
SELECT s.first_name, s.last_name, s.staff_id, SUM(p.amount) AS 'Revenue'
FROM payment p
JOIN staff s
ON s.staff_id = p.staff_id
GROUP BY s.staff_id
This gives me the 2 staff members and their revenue but I'm still missing the yearly part.
I'm yet again using the sakila database, if anybody could help me I would really appreciate it, thanks in regards
Edit for the tables:
-- sakila.staff definition
CREATE TABLE `staff` (
`staff_id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`address_id` smallint(5) unsigned NOT NULL,
`picture` blob DEFAULT NULL,
`email` varchar(50) DEFAULT NULL,
`store_id` tinyint(3) unsigned NOT NULL,
`active` tinyint(1) NOT NULL DEFAULT 1,
`username` varchar(16) NOT NULL,
`password` varchar(40) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
`last_update` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`staff_id`),
KEY `idx_fk_store_id` (`store_id`),
KEY `idx_fk_address_id` (`address_id`),
CONSTRAINT `fk_staff_address` FOREIGN KEY (`address_id`) REFERENCES `address` (`address_id`) ON UPDATE CASCADE,
CONSTRAINT `fk_staff_store` FOREIGN KEY (`store_id`) REFERENCES `store` (`store_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4;
CREATE TABLE `payment` (
`payment_id` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`customer_id` smallint(5) unsigned NOT NULL,
`staff_id` tinyint(3) unsigned NOT NULL,
`rental_id` int(11) DEFAULT NULL,
`amount` decimal(5,2) NOT NULL,
`payment_date` datetime NOT NULL,
`last_update` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
PRIMARY KEY (`payment_id`),
KEY `idx_fk_staff_id` (`staff_id`),
KEY `idx_fk_customer_id` (`customer_id`),
KEY `fk_payment_rental` (`rental_id`),
CONSTRAINT `fk_payment_customer` FOREIGN KEY (`customer_id`) REFERENCES `customer` (`customer_id`) ON UPDATE CASCADE,
CONSTRAINT `fk_payment_rental` FOREIGN KEY (`rental_id`) REFERENCES `rental` (`rental_id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_payment_staff` FOREIGN KEY (`staff_id`) REFERENCES `staff` (`staff_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=16050 DEFAULT CHARSET=utf8mb4;
This can be done by simply adding the year to the GROUP BY clause, like this:
SELECT s.first_name, s.last_name, s.staff_id, SUM(p.amount) AS 'Revenue'
, YEAR(payment_date) AS year
FROM payment p
JOIN staff s
ON s.staff_id = p.staff_id
GROUP BY s.staff_id, year
;
This means, generate a SUM for each group associated with rows having the same (staff_id, year) pairs.
The result:
+------------+-----------+----------+----------+------+
| first_name | last_name | staff_id | Revenue | year |
+------------+-----------+----------+----------+------+
| Mike | Hillyer | 1 | 33255.38 | 2005 |
| Mike | Hillyer | 1 | 234.09 | 2006 |
| Jon | Stephens | 2 | 33646.95 | 2005 |
| Jon | Stephens | 2 | 280.09 | 2006 |
+------------+-----------+----------+----------+------+
I have the following three tables:
CREATE TABLE group (
id SERIAL PRIMARY KEY,
name VARCHAR NOT NULL,
insert_date TIMESTAMP WITH TIME ZONE NOT NULL
);
CREATE TABLE customer (
id SERIAL PRIMARY KEY,
ext_id VARCHAR NOT NULL,
insert_date TIMESTAMP WITH TIME ZONE NOT NULL
);
CREATE TABLE customer_in_group (
id SERIAL PRIMARY KEY,
customer_id INT NOT NULL,
group_id INT NOT NULL,
insert_date TIMESTAMP WITH TIME ZONE NOT NULL,
CONSTRAINT customer_id_fk
FOREIGN KEY(customer_id)
REFERENCES customer(id),
CONSTRAINT group_id_fk
FOREIGN KEY(group_id)
REFERENCES group(id)
)
I need to find all of the groups which have not had any customer_in_group entities' group_id column reference them within the last two years. I then plan to delete all of the customer_in_groups that reference them, and finally delete that group after finding them.
So basically given the following two groups and the following 3 customer_in_groups
Group
| id | name | insert_date |
|----|--------|--------------------------|
| 1 | group1 | 2011-10-05T14:48:00.000Z |
| 2 | group2 | 2011-10-05T14:48:00.000Z |
Customer In Group
| id | group_id | customer_id | insert_date |
|----|----------|-------------|--------------------------|
| 1 | 1 | 1 | 2011-10-05T14:48:00.000Z |
| 2 | 1 | 1 | 2020-10-05T14:48:00.000Z |
| 3 | 2 | 1 | 2011-10-05T14:48:00.000Z |
I would expect just to get back group2, since group1 has a customer_in_group referencing it inserted in the last two years.
I am not sure how I would write the query that would find all of these groups.
As a starter, I would recommend enabling on delete cascade on foreing keys of customer_in_group.
Then, you can just delete the rows you want from groups, and it will drop the dependent rows in the child table. For this, you can use not exists:
delete from groups g
where not exists (
select 1
from customer_in_group cig
where cig.group_id = g.id and cig.insert_date >= now() - interval '2 year'
)
Forgive me if this question already ask,not much of db guy here ,
here is what i tried,
select row_number() over (partition by name order by challanto_date) , *
from (
select
rma,
p.id,
p.name,
challanto_date,
CURRENT_TIMESTAMP as fromDate
from challan_to_vendor cv
left join challan_to_vendor_detail cvd on cv.id = cvd.challan_to_vendor_id
inner join main_product p on p.id = cvd.product_id
union all
select
rma,
p.id,
p.name,
CURRENT_TIMESTAMP as toDate,
challan_date
from challan_from_vendor cv
left join challan_from_vendor_detail cvd on cv.id = cvd.challan_from_vendor_id
inner join main_product p on p.id = cvd.product_id
) as a
Here is my create table script :
challan_from_vendor
CREATE TABLE public.challan_from_vendor
(
id character varying NOT NULL,
date_ad date,
rma integer DEFAULT 1,
CONSTRAINT psk PRIMARY KEY (id)
)
challan_from_vendor_detail
CREATE TABLE public.challan_from_vendor_detail
(
id character varying NOT NULL,
challan_from_id character varying,
product_id character varying,
CONSTRAINT psks PRIMARY KEY (id),
CONSTRAINT fsks FOREIGN KEY (challan_from_id)
REFERENCES public.challan_from_vendor (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
challan_to_vendor;
CREATE TABLE public.challan_to_vendor
(
id character varying NOT NULL,
date_ad date,
rma integer DEFAULT 1,
CONSTRAINT pk PRIMARY KEY (id)
)
challan_to_vendor_detail
CREATE TABLE public.challan_to_vendor_detail
(
id character varying NOT NULL,
challan_to_id character varying,
product_id character varying,
CONSTRAINT pks PRIMARY KEY (id),
CONSTRAINT fks FOREIGN KEY (challan_to_id)
REFERENCES public.challan_to_vendor (id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
product
CREATE TABLE public.product
(
id character varying NOT NULL,
product_name character varying,
CONSTRAINT pks PRIMARY KEY (id)
)
Here is my table structures and desire output.
challan_from_vendor
| id | rma | date |
|:-----------|------------:|:------------:|
| 12012 | 0001 | 2018-11-10
| 123121 | 0001 | 2018-11-11
challan_to_vendor
| id | rma | date |
|:-----------|------------:|:------------:|
| 12 | 0001 | 2018-12-10
| 123 | 0001 | 2018-12-11
challan_from_vendor_detail
| id | challan_from_vendor_id | product_id |
|:-----------|------------:|:------------:|
| 121 | 12012 | 121313
| 1213 | 12012 | 131381
challan_to_vendor_detail
challan_from_vendor_detail
| id | challan_to_vendor_id | product_id |
|:-----------|------------------------|:------------:|
| 121 | 12 | 121313
| 1213 | 123 | 131381
product
| id | product_name |
|:-----------|------------:|
| 191313 | apple |
| 89113 | banana |
Output
| ram | product_id | challan_from_date | challan_to_date|
|:-----------|------------:|:-----------------:|:--------------:|
| 0001 | 191313| 2018-11-10 |2018-11-11 |
| 0001 | 89113 | 2018-12-10 |2018-12-11 |
There is some strange things in the query you have tried so it is not clear what tables, how they are related or what the columns are in those tables.
So by some guessing I give you this to start of with:
select
main_product.*,
challan_to_vendor.toDate,
challan_from_vendor.fromDate
from main_product
join challan_to_vendor using(product_id)
join challan_from_vendor using(product_id)
If you explain more about your db an what you want out of it I might be able to help you more.
Edit: So I could not run your create statements in my db since there was naming conflicts among other minor things. Here is some advice on the create process that I find useful:
Let the id's be integer instead of character varying otherwise it is probably a name-column and should not be named id. You also used integer-id's in your examples.
Use SERIAL PRIMARY KEY (see tutorial) to help you with the key creation. This also removes the naming-conflict since the constraints are given implicit unique names.
Use the same column-name for the same thing in all places to avoid confusion by having multiple things called id after a join plus that it simplify's the join. So for example the id of the product should be product_id in all places, that way you could use using(product_id) as your join condition.
So with the advises given above here's how I would create one of your table and then query them:
CREATE TABLE public.challan_to_vendor_detail
(
challan_to_vendor_detail_id SERIAL PRIMARY KEY,
challan_to_vendor_id integer,
product_id integer,
CONSTRAINT fks FOREIGN KEY (challan_to_vendor_id)
REFERENCES public.challan_to_vendor (challan_to_vendor_id) MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
);
select
product_name,
challan_to_vendor.date_ad as date_to,
challan_from_vendor.date_ad as date_from
from product
join challan_to_vendor_detail using(product_id)
join challan_to_vendor using(challan_to_vendor_id)
join challan_from_vendor_detail using(product_id)
join challan_from_vendor using(challan_from_vendor_id)
Unfortunately the overall db-design does not make sense to me so I do not know if this is what you expect.
Good luck!
I have this query below that I've rewritten a dozen different ways, yet I am unable to get it optimized and loaded in under 8 seconds. If I can get it to 2s, that would be good. 1s or less would be optimal.
This query retrieves a list of books that are currently available for sale or trade, and performs a bit of filtering. This query takes about 9-10 seconds.
SELECT
listing.for_sale,
listing.for_trade,
MIN(listing.price) AS from_price,
MAX(listing.price) AS to_price,
IF (NOW() > CONVERT_TZ(listing.date_sale_expiration, '+00:00', '-7:00'), 1, 0) AS expired,
COUNT(listing.book_id) AS 'listing_count',
book.id AS 'book_id',
book.title AS 'book_title',
book.date_released AS 'date_released',
book.publisher AS 'publisher',
book.photo_cover AS 'photo_cover',
publisher.name AS 'publisher_name',
COALESCE((SELECT COUNT(*) FROM listing l1 WHERE l1.book_id = book.id AND l1.status IN ('in_active_deal', 'complete')), 0) AS 'number_sold',
(SELECT 1 FROM listing l2 WHERE l2.status = 'available' AND l2.book_id = book.id AND l2.member_id = 1234 LIMIT 1) AS 'hasListing',
(SELECT 1 FROM wishlist w1 WHERE w1.book_id = book.id AND w1.member_id = 1234 LIMIT 1) AS 'hasWishlist'
FROM listing
INNER JOIN member ON
listing.member_id = member.id
AND member.transaction_limit <> 0
AND member.banned <> 1
AND member.date_last_login > DATE_SUB(CURDATE(), INTERVAL 120 DAY)
INNER JOIN book ON
listing.book_id = book.id
AND book.released = 1
INNER JOIN publisher ON
book.publisher_id = publisher.id
WHERE
listing.type = 'book'
AND listing.status = 'available'
AND (listing.for_trade = 1 OR (listing.for_sale = 1 AND NOW() < COALESCE(CONVERT_TZ(listing.date_sale_expiration, '+00:00', '-7:00'), 0)))
AND (
EXISTS (SELECT 1 FROM listing l3 LEFT JOIN book b ON l3.book_id = b.id WHERE l3.member_id = 1234 AND b.publisher_id = book.publisher_id AND l3.status = 'available' AND l3.type = 'book' AND (l3.for_trade = 1 OR (l3.for_sale = 1 AND NOW() < COALESCE(CONVERT_TZ(l3.date_sale_expiration, '+00:00', '-7:00'), 0))) LIMIT 1)
OR member.publisher_only <> 1
OR member.id = 1234
)
AND (
EXISTS (SELECT 1 FROM wishlist w WHERE w.member_id = member.id AND w.type = 'book' AND (w.type, w.book_id) IN (SELECT l4.type, l4.book_id FROM listing l4 WHERE 1234 = l4.member_id AND l4.status = 'available' AND (l4.for_trade = 1 OR (l4.for_sale = 1 AND NOW() < COALESCE(DATE_SUB(l4.date_sale_expiration, INTERVAL 7 HOUR), 0)))) LIMIT 1)
OR member.wishlist_only <> 1
OR member.id = 1234
)
GROUP BY
book.id
ORDER BY
book.date_released DESC
LIMIT 30;
These are my tables:
CREATE TABLE `listing` (
`id` int(10) unsigned NOT NULL auto_increment,
`member_id` int(10) unsigned NOT NULL,
`type` enum('book','audiobook','accessory') NOT NULL,
`book_id` int(10) unsigned default NULL,
`audiobook_id` int(10) unsigned default NULL,
`accessory_id` int(10) unsigned default NULL,
`date_created` datetime NOT NULL,
`date_modified` datetime NOT NULL,
`date_sale_expiration` datetime default NULL,
`status` enum('available','in_active_deal','complete','deleted') NOT NULL default 'available',
`for_sale` tinyint(1) unsigned NOT NULL default '0',
`for_trade` tinyint(1) unsigned NOT NULL default '0',
`price` decimal(10,2) default NULL,
`condition` tinyint(1) unsigned default NULL,
`notes` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `ix_accessory` (`accessory_id`,`member_id`,`type`,`status`),
KEY `ix_book` (`book_id`,`member_id`,`type`,`status`),
KEY `ix_member` (`member_id`,`status`,`date_created`),
KEY `ix_audiobook` (`audiobook_id`,`member_id`,`type`,`status`),
KEY `ix_status` (`status`,`accessory_id`,`for_trade`,`member_id`)
) ENGINE=MyISAM AUTO_INCREMENT=281724 DEFAULT CHARSET=utf8
CREATE TABLE `member` (
`id` int(10) unsigned NOT NULL auto_increment,
`email` varchar(200) NOT NULL,
`screen_name` varchar(25) default NULL,
`date_last_login` datetime default NULL,
`wishlist_only` tinyint(1) unsigned NOT NULL default '1',
`platform_only` tinyint(1) unsigned NOT NULL default '0',
`transaction_limit` smallint(6) NOT NULL default '5',
`banned` tinyint(1) unsigned NOT NULL default '0',
`notes` text,
PRIMARY KEY (`id`),
KEY `ix_email` (`email`),
KEY `ix_screen_name` (`screen_name`),
KEY `ix_transaction_limit` (`transaction_limit`)
) ENGINE=MyISAM AUTO_INCREMENT=50842 DEFAULT CHARSET=utf8
CREATE TABLE `publisher` (
`id` int(10) unsigned NOT NULL auto_increment,
`name` varchar(128) NOT NULL,
`date_updated` datetime default NULL,
PRIMARY KEY (`id`),
KEY `ix_name` (`name`)
) ENGINE=MyISAM AUTO_INCREMENT=129 DEFAULT CHARSET=utf8
CREATE TABLE `book` (
`id` int(10) unsigned NOT NULL auto_increment,
`publisher_id` int(10) unsigned default NULL,
`name` varchar(128) NOT NULL,
`description` text,
`keywords` varchar(200) default NULL,
`date_released` varchar(10) default NULL,
`genre` varchar(50) default NULL,
`subgenre` varchar(50) default NULL,
`author` varchar(100) default NULL,
`date_updated` datetime default NULL,
`photo_cover` varchar(50) default NULL,
`weight_oz` decimal(7,2) default NULL,
`released` tinyint(2) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `ix_genre` (`genre`),
KEY `ix_name` (`name`),
KEY `ix_released` (`released`,`date_released`),
KEY `ix_keywords` (`keywords`)
) ENGINE=MyISAM AUTO_INCREMENT=87329 DEFAULT CHARSET=utf8
CREATE TABLE `wishlist` (
`id` int(10) unsigned NOT NULL auto_increment,
`member_id` int(10) unsigned NOT NULL,
`type` enum('book','audiobook','accessory') NOT NULL,
`book_id` int(10) unsigned default NULL,
`audiobook_id` int(10) unsigned default NULL,
`accessory_id` int(10) unsigned default NULL,
`date_created` datetime NOT NULL,
`date_modified` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `ix_accessory` (`accessory_id`,`member_id`,`type`),
KEY `ix_book` (`book_id`,`member_id`,`type`),
KEY `ix_member_accessory` (`member_id`,`accessory_id`),
KEY `ix_member_date_created` (`member_id`,`date_created`),
KEY `ix_member_book` (`member_id`,`book_id`),
KEY `ix_member_audiobook` (`member_id`,`audiobook_id`),
KEY `ix_audiobook` (`audiobook_id`,`member_id`,`type`)
) ENGINE=MyISAM AUTO_INCREMENT=241886 DEFAULT CHARSET=utf8
And here is the result when I run EXPLAIN:
+----+--------------------+-----------+----------------+---------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-----------+----------------+---------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
| 1 | PRIMARY | member | range | PRIMARY,ix_transaction_limit | ix_transaction_limit | 2 | NULL | 19617 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | listing | ref | ix_game,ix_member,ix_status | ix_member | 5 | live_database001.member.id,const | 7 | Using where |
| 1 | PRIMARY | book | eq_ref | PRIMARY,ix_released | PRIMARY | 4 | live_database001.listing.book_id | 1 | Using where |
| 1 | PRIMARY | publisher | eq_ref | PRIMARY | PRIMARY | 4 | live_database001.book.publisher_id | 1 | |
| 6 | DEPENDENT SUBQUERY | w | ref | ix_member_accessory,ix_member_date_created,ix_member_book,ix_member_publisher | ix_member_accessory | 4 | live_database001.member.id | 6 | Using where |
| 7 | DEPENDENT SUBQUERY | l4 | index_subquery | ix_book,ix_member,ix_status | ix_book | 11 | func,const,func,const | 1 | Using where |
| 5 | DEPENDENT SUBQUERY | l3 | ref | ix_book,ix_member,ix_status | ix_member | 5 | const,const | 63 | Using where |
| 5 | DEPENDENT SUBQUERY | b | eq_ref | PRIMARY | PRIMARY | 4 | live_database001.l3.book_id | 1 | Using where |
| 4 | DEPENDENT SUBQUERY | w1 | ref | ix_book,ix_member_accessory,ix_member_date_created,ix_member_game,ix_member_publisher | ix_book | 9 | func,const | 1 | Using where; Using index |
| 3 | DEPENDENT SUBQUERY | l2 | ref | ix_book,ix_member,ix_status | ix_book | 9 | func,const | 1 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | l1 | ref | ix_book,ix_status | ix_book | 5 | func | 10 | Using where; Using index |
+----+--------------------+-----------+----------------+--------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
This brings me to a couple questions:
1. The member table is using ix_transaction_limit, and as a result is searching through 19k+ rows. Since I am specifying a member.id, shouldn't this be using PRIMARY and shouldn't the rows be 1? How can I fix this?
2. How does the key_len affect the performance?
3. I've seen other complex queries which dealt with 100's of millions of rows take less time. How is it that only 19k rows are taking so long?
(I'm still very green with MySQL Optimization, so I'd really love to understand the how's & why's)
Any suggestions small or big is greatly appreciated, thank you in advance!
Not sure what transaction limit does but at a guess it seems like a strange choice to have an index on. What might help is an index on date_last_login. At the moment the query is filtering member and then joining listing to it - ie. its going through all the member records with the appropriate transaction limit and using the member id to find the listing.
After dropping the index on the member table, I was still having the same problems. In fact, that made it even worse. So my ultimate solution was to completely re-write the query from scratch.
And consequently, changing the order in the conditionals made a big difference as well. So moving the and member_id = 1234 and the and wishlish_only <> 1 up above the subquery was a huge improvement.
Thanks for all your help!