SELECT column_name as name,
data_type,
concat(character_maximum_length,numeric_precision,datetime_precision) as length,
concat(numeric_precision,datetime_precision) as prec,
is_nullable as isnullable,
COALESCE(numeric_scale,'0') as scale,
null as definition,
null as iscomputed,
null as is_sparse,
column_default as "default",
null as is_persisted
FROM information_schema.columns
WHERE (table_schema, table_name) = ('test', 'testtable')
ORDER BY ordinal_position;
Current output:
name | data_type | length | prec | isnullable | scale | definition | iscomputed | is_sparse | default | is_persisted
----------------------+-----------------------------+--------+------+------------+-------+------------+------------+-----------+--------------------+--------------
keytestpostgresql | uuid | | | NO | 0 | | | | uuid_generate_v4() |
testsmalldatetime | timestamp without time zone | 0 | 0 | YES | 0 | | | | |
testdatetime | timestamp without time zone | 3 | 3 | YES | 0 | | | | |
testint | integer | 32 | 32 | YES | 0 | | | | |
testdatenotime | date | 0 | 0 | YES | 0 | | | | |
testnmemo | text | | | YES | 0 | | | | |
testfloat | double precision | 53 | 53 | YES | 0 | | | | |
testboolean | boolean | | | YES | 0 | | | | |
biginttest | bigint | 64 | 64 | YES | 0 | | | | |
testtinyint | smallint | 16 | 16 | YES | 0 | | | | |
testsmallint | smallint | 16 | 16 | YES | 0 | | | | |
testuniqueidentifier | uuid | | | YES | 0 | | | | |
testnumeric | numeric | 5 | 5 | YES | 2 | | | | |
updoperation | smallint | 16 | 16 | NO | 0 | | | | 0 |
upddate | timestamp without time zone | 3 | 3 | NO | 0 | | | | now() |
(15 rows)
Want column length to return or show value in output as -1 if blank or null
Want column prec to return or show value in output as 0 if blank or null
Please refer below Query:
SELECT column_name as name,
data_type,
coalesce(character_maximum_length, numeric_precision, datetime_precision, -1) as length,
concat(numeric_precision,datetime_precision) as prec,
is_nullable as isnullable,
COALESCE(numeric_scale,'0') as scale,
null as definition,
null as iscomputed,
null as is_sparse,
column_default as "default",
null as is_persisted
FROM information_schema.columns
WHERE (table_schema, table_name) = ('test', 'testtable')
ORDER BY ordinal_position;
I have a long running Oracle Query which uses a bunch of:
WHERE EXISTS (SELECT NULL FROM Table WHERE TableColumn IN (...))
Instead of using SELECT NULL, which goes through the entire table to find criteria, can't I just put FETCH NEXT 1 ROW ONLY after it since I only care if TableColumn is IN (...)?
Like this:
WHERE EXISTS (SELECT NULL FROM Table WHERE TableColumn IN (...) FETCH NEXT 1 ROW ONLY)
So the WHERE EXISTS would be evaluated quicker.
EDIT:
Below is the query plan without the FETCH NEXT clause attached:
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost | Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 75 | 521611 | |
| 1 | SORT AGGREGATE | | 1 | 75 | | |
| 2 | HASH JOIN | | 531266 | 39844950 | 521611 | |
| 3 | TABLE ACCESS FULL | ACCT | 47574 | 523314 | 418 | |
| 4 | HASH JOIN | | 531224 | 33998336 | 521185 | |
| 5 | INDEX FAST FULL SCAN | PK_ACTVTYP | 454 | 2270 | 2 | |
| 6 | HASH JOIN | | 531224 | 31342216 | 521177 | |
| 7 | INDEX FULL SCAN | PK_ACTVCAT | 67 | 335 | 1 | |
| 8 | HASH JOIN SEMI | | 531224 | 28686096 | 521169 | |
| 9 | NESTED LOOPS SEMI | | 531224 | 28686096 | 521169 | |
| 10 | STATISTICS COLLECTOR | | | | | |
| 11 | HASH JOIN RIGHT SEMI | | 531224 | 25498752 | 112887 | |
| 12 | TABLE ACCESS FULL | AMSACTVGRPEMPL | 2364 | 35460 | 10 | |
| 13 | TABLE ACCESS FULL | ACTV | 12779986 | 421739538 | 112712 | |
| 14 | INDEX RANGE SCAN | ACTVSUBACTV_DX2 | 163091724 | 978550344 | 251246 | |
| 15 | INDEX FAST FULL SCAN | ACTVSUBACTV_DX2 | 163091724 | 978550344 | 251246 | |
------------------------------------------------------------------------------------------------
Below is the query plan with the FETCH NEXT clause attached:
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost | Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 69 | 113148 | |
| 1 | SORT AGGREGATE | | 1 | 69 | | |
| 2 | FILTER | | | | | |
| 3 | HASH JOIN | | 531221 | 36654249 | 113144 | |
| 4 | TABLE ACCESS FULL | ACCT | 47574 | 523314 | 418 | |
| 5 | HASH JOIN | | 531179 | 30808382 | 112718 | |
| 6 | INDEX FAST FULL SCAN | PK_ACTVTYP | 454 | 2270 | 2 | |
| 7 | HASH JOIN | | 531179 | 28152487 | 112710 | |
| 8 | INDEX FULL SCAN | PK_ACTVCAT | 67 | 335 | 1 | |
| 9 | HASH JOIN RIGHT SEMI | | 531179 | 25496592 | 112702 | |
| 10 | TABLE ACCESS FULL | AMSACTVGRPEMPL | 2167 | 32505 | 10 | |
| 11 | TABLE ACCESS FULL | ACTV | 12778893 | 421703469 | 112527 | |
| 12 | VIEW | | 1 | 13 | 4 | |
| 13 | WINDOW BUFFER PUSHED RANK | | 8 | 48 | 4 | |
| 14 | INDEX RANGE SCAN | ACTVSUBACTV_DX2 | 8 | 48 | 4 | |
------------------------------------------------------------------------------------------------
From what I see, it looks like without the FETCH NEXT it's adding overhead by more TABLE ACCESS FULL
EDIT #2
Adding AND ROWNUM = 1 instead of FETCH NEXT 1 ROW ONLY:
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost | Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 54 | 128114 | |
| 1 | SORT AGGREGATE | | 1 | 54 | | |
| 2 | FILTER | | | | | |
| 3 | HASH JOIN | | 12779902 | 690114708 | 113296 | |
| 4 | TABLE ACCESS FULL | ACCT | 47574 | 523314 | 418 | |
| 5 | HASH JOIN | | 12778893 | 549492399 | 112713 | |
| 6 | MERGE JOIN CARTESIAN | | 30418 | 304180 | 31 | |
| 7 | INDEX FULL SCAN | PK_ACTVCAT | 67 | 335 | 1 | |
| 8 | BUFFER SORT | | 454 | 2270 | 30 | |
| 9 | INDEX FAST FULL SCAN | PK_ACTVTYP | 454 | 2270 | 0 | |
| 10 | TABLE ACCESS FULL | ACTV | 12778893 | 421703469 | 112517 | |
| 11 | COUNT STOPKEY | | | | | |
| 12 | INLIST ITERATOR | | | | | |
| 13 | INDEX UNIQUE SCAN | PK_AMSACTVGRPEMPL | 1 | 15 | 2 | |
| 14 | COUNT STOPKEY | | | | | |
| 15 | INDEX RANGE SCAN | ACTVSUBACTV_DX2 | 2 | 12 | 4 | |
------------------------------------------------------------------------------------------------
The FETCH NEXT is new in 12c, and to avoid the performance issue causing it add
hint like below
WHERE EXISTS (SELECT /*+ first_rows(1)*/* FROM Table WHERE TableColumn IN (...) FETCH NEXT 1 ROW ONLY)
try it and check its query plan
Note: I recommend to add indexes on table ACCT ,ACTV to enhance its performance.
My logic in this query is right (well im 80% sure it is). but its been running for 2h 23min and still going, was wondering if some one could maybe help me make this run a bit more efficiently as i don't think its that intense of a query
SELECT b.bridge_no, COUNT(*) AS comment_cnt
FROM iacd_asset b INNER JOIN iacd_note c
ON REGEXP_LIKE(c.comments, '(^|\W)BN' || b.bridge_no || '(\W|$)', 'i')
inner join ncr_note e on c.note_id=e.note_id
inner join ncr f on e.ncr_id=f.ncr_id
inner join ncr_iac g on f.ncr_id=g.ncr_id
WHERE c.create_dt >= date'2015-01-01'
AND c.create_dt < date'2015-03-12'
AND length(b.bridge_no) > 1
AND g.scheme in (1, 3, 5, 6, 7, 8, 9, 9, and about 10 more values)
GROUP BY b.bridge_no
ORDER BY comment_cnt;
in short the query should be making a bunch of joins, and then filtering the joined table by schemes (g.scheme in....) , and then parsing the notes field for anything with BN in it.
PLAN TABLE, ok i have never used one before, but i believe this is the plan table
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| OPERATION | OPTIONS | OBJECT_OWNER | OBJECT_NAME | OBJECT_ALIAS | OBJECT_INSTANCE | OBJECT_TYPE | OPTIMIZER | ID | PARENT_ID | DEPTH | POSITION | COST | CARDINALITY | BYTES | CPU_COST | IO_COST | TEMP_SPACE | ACCESS_PREDICATES | FILTER_PREDICATES | PROJECTION | TIME | QBLOCK_NAME |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| SELECT STATEMENT | | | | | | | ALL_ROWS | 0 | | 0 | 281,503 | 281,503 | 40 | 4,480 | 148,378,917,975 | 215,677 | | | | | 458 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| SORT | ORDER BY | | | | | | | 1 | 0 | 1 | 1 | 281,503 | 40 | 4,480 | 148,378,917,975 | 215,677 | | | | (#keys=1) COUNT(*)[22], "B"."BRIDGE_NO"[NUMBER,22] | 458 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| HASH | GROUP BY | | | | | | | 2 | 1 | 2 | 1 | 281,503 | 40 | 4,480 | 148,378,917,975 | 215,677 | | | | (#keys=1) "B"."BRIDGE_NO"[NUMBER,22], COUNT(*)[22] | 458 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| HASH JOIN | | | | | | | | 3 | 2 | 3 | 1 | 281,497 | 16,084 | 1,801,408 | 148,366,537,976 | 215,677 | 24,126,000 | "G"."NCR_ID"="F"."NCR_ID" | | (#keys=1) "B"."BRIDGE_NO"[NUMBER,22] | 458 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| HASH JOIN | | | | | | | | 4 | 3 | 4 | 1 | 96,996 | 209,778 | 21,607,134 | 13,549,630,814 | 90,985 | 22,725,000 | "E"."NCR_ID"="F"."NCR_ID" | | (#keys=1) "F"."NCR_ID"[NUMBER,22], "B"."BRIDGE_NO"[NUMBER,22] | 158 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| HASH JOIN | | | | | | | | 5 | 4 | 5 | 1 | 42,595 | 208,419 | 20,216,643 | 5,484,063,163 | 40,162 | 9,839,000 | "C"."NOTE_ID"="E"."NOTE_ID" | REGEXP_LIKE ("C"."COMMENTS",'(^|\W)BN'||TO_CHAR("B"."BRIDGE_NO")||'(\W|$)','i') | (#keys=1) "B"."BRIDGE_NO"[NUMBER,22], "E"."NCR_ID"[NUMBER,22] | 70 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| PARTITION RANGE | SINGLE | | | | | | | 6 | 5 | 6 | 1 | 1,039 | 104,603 | 8,577,446 | 62,280,224 | 1,011 | | | | "C"."NOTE_ID"[NUMBER,22], "C"."COMMENTS"[VARCHAR2,4000] | 2 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| TABLE ACCESS | FULL | IACDB | IACD_NOTE | C#SEL$1 | 2 | TABLE | ANALYZED | 7 | 6 | 7 | 1 | 1,039 | 104,603 | 8,577,446 | 62,280,224 | 1,011 | | | "C"."CREATE_DATE"<TO_DATE(' 2014-12-31 00:00:00', 'syyyy-mm-dd hh24:mi:ss') | "C"."NOTE_ID"[NUMBER,22], "C"."COMMENTS"[VARCHAR2,4000] | 2 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| MERGE JOIN | CARTESIAN | | | | | | | 8 | 5 | 6 | 2 | 24,267 | 12,268,270 | 184,024,050 | 2,780,501,758 | 23,033 | | | | (#keys=0) "B"."BRIDGE_NO"[NUMBER,22], "E"."NCR_ID"[NUMBER,22], "E"."NOTE_ID"[NUMBER,22] | 40 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| TABLE ACCESS | FULL | IACDB | IACD_ASSET | B#SEL$1 | 1 | TABLE | ANALYZED | 9 | 8 | 7 | 1 | 7 | 40 | 160 | 560,542 | 7 | | | LENGTH(TO_CHAR("B"."BRIDGE_NO"))>1 | "B"."BRIDGE_NO"[NUMBER,22] | 1 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| BUFFER | SORT | | | | | | | 10 | 8 | 7 | 2 | 24,259 | 308,248 | 3,390,728 | 2,779,941,216 | 23,026 | | | | (#keys=0) "E"."NCR_ID"[NUMBER,22], "E"."NOTE_ID"[NUMBER,22] | 40 | |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| TABLE ACCESS | FULL | IACDB | IACD_NCR_NOTE | E#SEL$2 | 4 | TABLE | ANALYZED | 11 | 10 | 8 | 1 | 606 | 308,248 | 3,390,728 | 69,498,530 | 576 | | | | "E"."NCR_ID"[NUMBER,22], "E"."NOTE_ID"[NUMBER,22] | 1 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| INDEX | FAST FULL SCAN | IACDB | PK_IACDNCR_NCRID | F#SEL$3 | | INDEX (UNIQUE) | ANALYZED | 12 | 4 | 5 | 2 | 31,763 | 22,838,996 | 137,033,976 | 3,248,120,913 | 30,322 | | | | "F"."NCR_ID"[NUMBER,22] | 52 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
| TABLE ACCESS | FULL | IACDB | IACD_NCR_IAC | G#SEL$4 | 8 | TABLE | ANALYZED | 13 | 3 | 4 | 2 | 181,461 | 1,731,062 | 15,579,558 | 134,407,812,606 | 121,833 | | | ALL THE SCHEMES CHCECKS | "G"."NCR_ID"[NUMBER,22] | 295 | SEL$81719215 |
+------------------+----------------+--------------+------------------+--------------+-----------------+----------------+-----------+----+-----------+-------+----------+---------+-------------+-------------+-----------------+---------+------------+-----------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------+------+--------------+
Hopefully thats legible enough
interms of indexes i assume only the fields that are being sorted is importent
crate_dt is not indexed
scheme id is indexed
Maybe my order in query is wrong...
The plan shows you're doing FULL TABLE SCAN of IACD_NOTE and IACD_ASSET, and then doing a CARETESIAN join of them, because you have provided no criteria for linking one record in IACD_ASSET to a set of records in IACD_NOTE.
That's not my definition of a non-intense query, and the eye-popping values for CPU cost bear that out.
You need to replace this ..,
FROM iacd_asset b INNER JOIN iacd_note c
ON REGEXP_LIKE(c.comments, '(^|\W)BN' || b.bridge_no || '(\W|$)', 'i')
... with an actual join on indexed columns. It would be helpful if Notes were linked to Assets by a foreign key of BRIDGE_NO or similar. I don't know your data model. Then you can use that regex as an additional filter in the WHERE clause.
Also you join to three further tables, to get to something which allows an additional filter on SCHEME. Again, I don't know your data model but this seems pretty inefficient.
Unfortunately this is the sort of tuning which relies on domain knowledge. Fixing this query requires understanding of the data - its volume, distribution and skew, the data model itself and the business logic your query implements. This is way beyond the scope of the advice we can offer in StackOverflow.
One thing to consider, but it is a big decision would be to index the comments with a free text index. However, that has lots of ramifications (especially space and database admin). Find out more.
I have the following SQL generated from my Rails app, it is trying to get a list of all auto models that have live adverts in a marketplace app & from mysql:
SELECT `models`.* FROM `models`
INNER JOIN `autos` ON autos.model_id = models.id
INNER JOIN `ads` ON `ads`.id = `autos`.ad_id
WHERE (ads.ad_status_id = 4 AND pub_start_date < NOW() AND pub_end_date > NOW() AND models.manufacturer_id = 50 )
GROUP BY models.id ORDER BY models.name;
When I run an explain, this is what I get:
Id 1 1 1
Select Type SIMPLE SIMPLE SIMPLE
Table models autos ads
Type ref ref eq_ref
Possible Keys PRIMARY,manufacturer_id model_id,ad_id PRIMARY,quick_search,ad_status_id
Key manufacturer_id model_id PRIMARY
Key Length 5 4 4
Ref const concept_development.models.id concept_development.autos.ad_id
Rows 70 205 1
Extra Using where; Using temporary; Using filesort Using where Using where
I cannot understand why the query is generating a temporary table / using file-sort - all of the referenced keys are indexes. Been trying to figure this out for a few days now and getting nowhere.
Any help is very much appreciated!
EXPLAIN models:
+---------------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(32) | YES | | NULL | |
| manufacturer_id | int(11) | YES | MUL | NULL | |
| vehicle_category_id | int(11) | NO | MUL | 1 | |
| synonym_names | longtext | YES | | NULL | |
+---------------------+-------------+------+-----+---------+----------------+
SHOW INDEXES FROM models:
+--------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| models | 0 | PRIMARY | 1 | id | A | 2261 | NULL | NULL | | BTREE | |
| models | 1 | manufacturer_id | 1 | manufacturer_id | A | 205 | NULL | NULL | YES | BTREE | |
| models | 1 | vehicle_category_id | 1 | vehicle_category_id | A | 7 | NULL | NULL | | BTREE | |
+--------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
MODEL TABLE STATUS:
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| models | MyISAM | 10 | Dynamic | 2261 | 26 | 61000 | 281474976710655 | 84992 | 0 | 2751 | 2010-09-28 18:42:45 | 2010-09-28 18:42:45 | 2010-09-28 18:44:00 | latin1_swedish_ci | NULL | | |
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
EXPLAIN ADS
+------------------+--------------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------------------+------+-----+---------------------+----------------+
| id | int(10) | NO | PRI | NULL | auto_increment |
| fp_urn | int(10) | NO | MUL | 0 | |
| user_id | int(10) | NO | MUL | 0 | |
| ad_status_id | int(3) unsigned | NO | MUL | 1 | |
| style_id | int(10) | NO | | 3 | |
| search_tags | varchar(255) | YES | | NULL | |
| title | varchar(255) | NO | | | |
| description | text | YES | | NULL | |
| currency | enum('EUR','GBP') | NO | | EUR | |
| price | decimal(8,2) | NO | MUL | 0.00 | |
| proposal_type | enum('Offered','Wanted') | NO | | Offered | |
| category_id | int(10) | YES | | 0 | |
| contact | varchar(50) | NO | MUL | | |
| area_id | int(10) | NO | | 0 | |
| origin_id | int(10) | NO | | 0 | |
| reject_reason_id | int(3) | NO | | 0 | |
| date_created | timestamp | NO | | 0000-00-00 00:00:00 | |
| last_modified | timestamp | NO | | CURRENT_TIMESTAMP | |
| pub_start_date | datetime | YES | | 0000-00-00 00:00:00 | |
| pub_end_date | datetime | YES | | 0000-00-00 00:00:00 | |
| bumped_up_date | datetime | YES | | 0000-00-00 00:00:00 | |
| state | smallint(6) | YES | | NULL | |
| eproofed | tinyint(1) | NO | | 0 | |
| is_featured | int(1) | NO | | 0 | |
| num_featured_imp | int(10) | YES | | 0 | |
| num_direct_imp | int(10) | YES | | 0 | |
| is_top_listed | int(1) | NO | | 0 | |
| delta | tinyint(1) | NO | | 0 | |
| ext_ref_id | varchar(50) | YES | | NULL | |
| email_seller | tinyint(1) | YES | | 1 | |
| sort_by | int(10) | YES | | 8 | |
| permalink | varchar(500) | YES | | NULL | |
| external_url | varchar(255) | YES | | NULL | |
+------------------+--------------------------+------+-----+---------------------+----------------+
SHOW TABLE STATUS FROM concept_development WHERE NAME LIKE 'ads';
+------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
| ads | InnoDB | 10 | Compact | 656350 | 232 | 152748032 | 0 | 87736320 | 340787200 | 1148382 | 2010-09-29 09:55:46 | NULL | NULL | utf8_general_ci | NULL | checksum=1 delay_key_write=1 row_format=DYNAMIC | |
+------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
SHOW INDEXES FROM ADS
+-------+------------+-----------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+-----------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| ads | 0 | PRIMARY | 1 | id | A | 521391 | NULL | NULL | | BTREE | |
| ads | 1 | NewIndex1 | 1 | ad_status_id | A | 15 | NULL | NULL | | BTREE | |
| ads | 1 | NewIndex1 | 2 | pub_end_date | A | 260695 | NULL | NULL | YES | BTREE | |
| ads | 1 | NewIndex1 | 3 | category_id | A | 521391 | NULL | NULL | YES | BTREE | |
| ads | 1 | NewIndex1 | 4 | style_id | A | 521391 | NULL | NULL | | BTREE | |
| ads | 1 | NewIndex2 | 1 | user_id | A | 130347 | NULL | NULL | | BTREE | |
| ads | 1 | NewIndex3 | 1 | price | A | 7667 | NULL | NULL | | BTREE | |
| ads | 1 | contact | 1 | contact | A | 260695 | NULL | NULL | | BTREE | |
| ads | 1 | fp_urn | 1 | fp_urn | A | 521391 | NULL | NULL | | BTREE | |
+-------+------------+-----------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
EXPLAIN autos
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+-------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+-------------+----------------+
| id | int(10) | NO | PRI | NULL | auto_increment |
| ad_id | int(10) | YES | MUL | NULL | |
| style_id | int(10) | YES | MUL | NULL | |
| manufacturer_id | int(10) | NO | MUL | NULL | |
| model_id | int(10) | NO | MUL | NULL | |
| registration | varchar(10) | YES | | NULL | |
| year | int(4) | YES | | NULL | |
| fuel_type | enum('Petrol','Diesel') | NO | | Petrol | |
| colour | varchar(75) | YES | | NULL | |
| mileage | varchar(25) | NO | | Not Entered | |
| mileage_units | enum('mls','kms') | NO | | mls | |
| num_doors | varchar(25) | NO | | Not Entered | |
| num_owners | int(2) | YES | | NULL | |
| engine_size | varchar(10) | YES | | NULL | |
| transmission_type | enum('Manual','Automatic') | NO | | Manual | |
| body_type | enum('Saloon','Hatchback') | NO | | Saloon | |
| condition | varchar(75) | NO | | NA | |
| extra_features | text | YES | | NULL | |
| tax_expiry | varchar(7) | YES | | NULL | |
| nct_expiry | varchar(7) | YES | | NULL | |
| variation | text | YES | | NULL | |
| tax_class | enum('Agricultural','Bus') | NO | | Private | |
| co2 | int(9) | YES | | NULL | |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+-----+-------------+----------------+
SHOW TABLE STATUS FROM concept_development WHERE NAME LIKE 'autos'
+-------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
| autos | InnoDB | 10 | Compact | 196168 | 136 | 26804224 | 0 | 26279936 | 340787200 | 485405 | 2010-09-17 22:09:45 | NULL | NULL | utf8_general_ci | NULL | checksum=1 delay_key_write=1 row_format=DYNAMIC | |
+-------+--------+---------+------------+--------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+-------------------------------------------------+---------+
show indexes from autos;
+-------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| autos | 0 | PRIMARY | 1 | id | A | 294937 | NULL | NULL | | BTREE | |
| autos | 1 | ad_id | 1 | ad_id | A | 294937 | NULL | NULL | YES | BTREE | |
| autos | 1 | style_id | 1 | style_id | A | 10 | NULL | NULL | YES | BTREE | |
| autos | 1 | manufacturer_id | 1 | manufacturer_id | A | 194 | NULL | NULL | | BTREE | |
| autos | 1 | model_id | 1 | model_id | A | 830 | NULL | NULL | | BTREE | |
+-------+------------+-----------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
From the MySQL documentation:
Temporary tables can be created under conditions such as these:
* If there is an ORDER BY clause and a different GROUP BY clause, or if the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue, a temporary table is created.
http://dev.mysql.com/doc/refman/5.1/en/internal-temporary-tables.html
Change all the text columns to varchar. If you need to maintain them as "text", you'll need to snowflake the schema and exclude the description tables from this query.
If any of the columns in any of the tables are text or blob, MySQL automatically creates an on-disk temporary table, rather than an in-memory temporary table. The temporary table itself isn't killing you, it's the fact that it's writing it to the disk.
http://dev.mysql.com/doc/refman/5.1/en/internal-temporary-tables.html
Some conditions prevent the use of an in-memory temporary table, in which case the server uses an on-disk table instead:
Presence of a BLOB or TEXT column in the table
You have a index on pub_end_date, but not on pub_start_date and your WHERE clause references both.
It looks like it is not using the pub_end_date index, but this could be because it needs to check pub_start_date as well.
This isn't explaining why but how about you rewrite your query to not use a group by? I think you're just joining on those tables to ensure there exists an ad of interest. So how about:
SELECT `models`.*
FROM `models`
WHERE models.manufacturer_id = 50
AND EXISTS ( SELECT * FROM `autos`
INNER JOIN `ads` ON `ads`.id = `autos`.ad_id
WHERE autos.model_id = models.id
AND ads.ad_status_id = 4
AND ads.pub_start_date < NOW()
AND ads.pub_end_date > NOW()
)
ORDER BY models.name;
The performance issues might be related to the group by, in which case this would improve performance.
Maybe it'd look a bit nicer with and NOW() between ads.pub_start_date and ads.pub_end_date, if you're allowed to do that in mysql (and if it works how you want with the edge cases).
EDIT: Updated with suggestions from Bill Karwin below. Still very slow.
I'm trying to write a query that will find all items on an order that are entered to a warehouse that doesn't have a record for that item in that warehouse. As an example, if item XYZ is entered for warehouse A, but warehouse A doesn't actually carry item XYZ, I want the order item to show up in my report.
I'm able to run the query just fine, but it seems to take forever (50 seconds). It seems to be hanging mainly on the "is null" condition I have in the where clause. If I remove the condition with the "is null" and run it, it executes in about 4.8s. Here's my query:
SELECT
saw_order.Wo,
saw_orderitem.Item,
saw_orderitem.Stock,
saw_order.`Status`,
saw_order.`Date`,
saw_orderitem.Warehouse,
saw_stockbalance.Balno,
saw_stockbalance.Stock
FROM
saw_order
Inner Join saw_orderitem ON saw_order.Wo = saw_orderitem.Wo
Inner Join saw_stock ON saw_orderitem.Stock = saw_stock.Stock
Left Join saw_stockbalance ON saw_orderitem.Stock = saw_stockbalance.Stock
AND saw_orderitem.Warehouse = saw_stockbalance.Warehouse
WHERE
saw_order.`Status` Between 3 and 81 and
saw_stockbalance.Stock Is Null
When I explain the query above, I see:
+----+-------------+------------------+--------+------------------------------+---------+---------+-------------------------------------------------------+-------+-------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+--------+------------------------------+---------+---------+-------------------------------------------------------+-------+-------------------------+
| 1 | SIMPLE | saw_stock | index | PRIMARY | PRIMARY | 17 | NULL | 32793 | Using index |
| 1 | SIMPLE | saw_orderitem | ref | PRIMARY,Stock,StockWarehouse | Stock | 17 | saws.saw_stock.Stock | 68 | |
| 1 | SIMPLE | saw_order | eq_ref | PRIMARY,Status | PRIMARY | 4 | saws.saw_orderitem.Wo | 1 | Using where |
| 1 | SIMPLE | saw_stockbalance | ref | Stock,Warehouse | Stock | 20 | saws.saw_orderitem.Stock,saws.saw_orderitem.Warehouse | 1 | Using where; Not exists |
+----+-------------+------------------+--------+------------------------------+---------+---------+-------------------------------------------------------+-------+-------------------------+
I'm pretty sure I have indexes for all of the fields of the respective tables in my joins, but can't figure out how to rewrite the query to make it go faster.
EDIT: Here are the indexes I have set up on my tables:
mysql> show index from saw_order;
+-----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| saw_order | 0 | PRIMARY | 1 | Wo | A | 553425 | NULL | NULL | | BTREE | |
| saw_order | 1 | Customer | 1 | Customer | A | 14957 | NULL | NULL | | BTREE | |
| saw_order | 1 | Other | 1 | Other | A | 218 | NULL | NULL | | BTREE | |
| saw_order | 1 | Site | 1 | Site | A | 8 | NULL | NULL | | BTREE | |
| saw_order | 1 | Date | 1 | Date | A | 1594 | NULL | NULL | | BTREE | |
| saw_order | 1 | Status | 1 | Status | A | 15 | NULL | NULL | | BTREE | |
+-----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
6 rows in set
mysql> show index from saw_orderitem;
+---------------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| saw_orderitem | 0 | PRIMARY | 1 | Wo | A | NULL | NULL | NULL | | BTREE | |
| saw_orderitem | 0 | PRIMARY | 2 | Item | A | 1842359 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | Stock | 1 | Stock | A | 27093 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | Product | 1 | Product | A | 803 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | GGroup | 1 | GGroup | A | 114 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | ShipVia | 1 | ShipVia | A | 218 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | Warehouse | 1 | Warehouse | A | 9 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | StockWarehouse | 1 | Stock | A | 27093 | NULL | NULL | | BTREE | |
| saw_orderitem | 1 | StockWarehouse | 2 | Warehouse | A | 49793 | NULL | NULL | | BTREE | |
+---------------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
9 rows in set
mysql> show index from saw_stock;
+-----------+------------+-------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+-------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
| saw_stock | 0 | PRIMARY | 1 | Stock | A | 32793 | NULL | NULL | | BTREE | |
| saw_stock | 1 | Class | 1 | Class | A | 655 | NULL | NULL | YES | BTREE | |
| saw_stock | 1 | DateFirstReceived | 1 | DateFirstReceived | A | 2732 | NULL | NULL | | BTREE | |
+-----------+------------+-------------------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+
3 rows in set
mysql> show index from saw_stockbalance;
+------------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+------------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| saw_stockbalance | 0 | PRIMARY | 1 | Balno | A | 146315 | NULL | NULL | | BTREE | |
| saw_stockbalance | 1 | Stock | 1 | Stock | A | 36578 | NULL | NULL | | BTREE | |
| saw_stockbalance | 1 | Stock | 2 | Warehouse | A | 146315 | NULL | NULL | | BTREE | |
| saw_stockbalance | 1 | Warehouse | 1 | Warehouse | A | 11 | NULL | NULL | | BTREE | |
+------------------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
4 rows in set
Any ideas?
I'd try to make it use a covering index. That is, instead of testing if Balno is null, test if one of the columns in your left outer join conditions is null. E.g. Stock or Warehouse.
You should also define an index over the two columns (Stock, Warehouse) in both tables saw_orderitem and saw_stockbalance.
Try:
SELECT t.wo,
soi.item,
soi.stock,
t.Status,
t.Date,
soi.warehouse,
NULL 'balno', --ssb.Balno,
NULL 'stock', --ssb.Stock
FROM SAW_ORDER t
JOIN SAW_ORDERITEM soi ON soi.wo = t.wo
JOIN SAW_STOCK ss ON ss.stock = soi.stock
WHERE t.status BETWEEN 3 AND 81
AND NOT EXISTS (SELECT NULL
FROM SAW_STOCKBALANCE ssb
WHERE ssb.stock != soi.stock
AND ssb.warehouse = soi.warehouse)
The problem with the query is that you are checking for nulls on a LEFT JOIN...