We have a db (9.6) that contains measurement data. The relevant query regards 3 tables:
aufnehmer (i.e. sensor), 5e+2 entries,
zeitpunkt (i.e. point in time), 4e+6 entries
wert (i.e. value), 6e+8 entries
aufnehmer : zeitpunkt = m : n with wert as the mapping table. All relevant columns are indexed.
The following query
select count(*) from wert w
inner join aufnehmer a on w.aufnehmer_id = a.id
inner join zeitpunkt z on z.id = w.zeitpunkt_id
where a.id = 12749
and z.zeitpunkt <= ('2018-05-07')::timestamp without time zone
and z.zeitpunkt >= ('2018-05-01')::timestamp without time zone;
produces this query plan:
Aggregate (cost=3429124.66..3429124.67 rows=1 width=8) (actual time=66.252..66.252 rows=1 loops=1)
-> Nested Loop (cost=571.52..3429084.29 rows=16149 width=0) (actual time=19.051..65.406 rows=15942 loops=1)
-> Index Only Scan using idx_aufnehmer_id on aufnehmer a (cost=0.28..8.29 rows=1 width=4) (actual time=0.007..0.008 rows=1 loops=1)
Index Cond: (id = 12749)
Heap Fetches: 1
-> Nested Loop (cost=571.24..3428914.50 rows=16149 width=4) (actual time=19.040..64.502 rows=15942 loops=1)
-> Bitmap Heap Scan on zeitpunkt z (cost=570.67..22710.60 rows=26755 width=4) (actual time=1.551..3.407 rows=24566 loops=1)
Recheck Cond: ((zeitpunkt <= '2018-05-07 00:00:00'::timestamp without time zone) AND (zeitpunkt >= '2018-05-01 00:00:00'::timestamp without time zone))
Heap Blocks: exact=135
-> Bitmap Index Scan on idx_zeitpunkt_zeitpunkt_order_desc (cost=0.00..563.98 rows=26755 width=0) (actual time=1.527..1.527 rows=24566 loops=1)
Index Cond: ((zeitpunkt <= '2018-05-07 00:00:00'::timestamp without time zone) AND (zeitpunkt >= '2018-05-01 00:00:00'::timestamp without time zone))
-> Index Only Scan using uq1_wert on wert w (cost=0.57..126.94 rows=37 width=8) (actual time=0.002..0.002 rows=1 loops=24566)
Index Cond: ((aufnehmer_id = 12749) AND (zeitpunkt_id = z.id))
Heap Fetches: 15942
Planning time: 0.399 ms
Execution time: 66.339 ms
and takes about a second.
When the end date is augmented by one day and the query is changed to:
... --same as above
and z.zeitpunkt <= ('2018-05-08')::timestamp without time zone
and z.zeitpunkt >= ('2018-05-01')::timestamp without time zone;
the query plan changes to
Aggregate (cost=3711151.24..3711151.25 rows=1 width=8) (actual time=35601.351..35601.351 rows=1 loops=1)
-> Nested Loop (cost=66264.74..3711104.14 rows=18840 width=0) (actual time=35348.705..35600.192 rows=17612 loops=1)
-> Index Only Scan using idx_aufnehmer_id on aufnehmer a (cost=0.28..8.29 rows=1 width=4) (actual time=0.007..0.010 rows=1 loops=1)
Index Cond: (id = 12749)
Heap Fetches: 1
-> Hash Join (cost=66264.47..3710907.45 rows=18840 width=4) (actual time=35348.693..35598.183 rows=17612 loops=1)
Hash Cond: (w.zeitpunkt_id = z.id)
-> Bitmap Heap Scan on wert w (cost=43133.18..3678947.46 rows=2304078 width=8) (actual time=912.086..35145.680 rows=2334815 loops=1)
Recheck Cond: (aufnehmer_id = 12749)
Rows Removed by Index Recheck: 205191912
Heap Blocks: exact=504397 lossy=1316875
-> Bitmap Index Scan on idx_wert_aufnehmer_id (cost=0.00..42557.16 rows=2304078 width=0) (actual time=744.144..744.144 rows=2334815 loops=1)
Index Cond: (aufnehmer_id = 12749)
-> Hash (cost=22741.12..22741.12 rows=31214 width=4) (actual time=8.909..8.909 rows=27675 loops=1)
Buckets: 32768 Batches: 1 Memory Usage: 1229kB
-> Bitmap Heap Scan on zeitpunkt z (cost=664.37..22741.12 rows=31214 width=4) (actual time=1.822..5.600 rows=27675 loops=1)
Recheck Cond: ((zeitpunkt <= '2018-05-08 00:00:00'::timestamp without time zone) AND (zeitpunkt >= '2018-05-01 00:00:00'::timestamp without time zone))
Heap Blocks: exact=152
-> Bitmap Index Scan on idx_zeitpunkt_zeitpunkt_order_desc (cost=0.00..656.57 rows=31214 width=0) (actual time=1.798..1.798 rows=27675 loops=1)
Index Cond: ((zeitpunkt <= '2018-05-08 00:00:00'::timestamp without time zone) AND (zeitpunkt >= '2018-05-01 00:00:00'::timestamp without time zone))
Planning time: 0.404 ms
Execution time: 35608.286 ms
and the execution takes about 1000 times longer.
So it seems that the query planner switches to joining aufnehmer and wert first which takes much longer than joining zeitpunkt and wert first.
Any idea if it can be forced to keep the first execution plan? We have already increased work_mem but it did not make any difference.
Related
EXPLAIN ANALYZE SELECT "alerts"."id",
"alerts"."created_at",
't1'::text AS src_table
FROM "alerts"
INNER JOIN "devices"
ON "devices"."id" = "alerts"."device_id"
INNER JOIN "sites"
ON "sites"."id" = "devices"."site_id"
WHERE "sites"."cloud_id" = 111
AND "alerts"."created_at" >= '2019-08-30'
ORDER BY "created_at" DESC limit 9;
Limit (cost=1.15..36021.60 rows=9 width=16) (actual time=30.505..29495.765 rows=9 loops=1)
-> Nested Loop (cost=1.15..232132.92 rows=58 width=16) (actual time=30.504..29495.755 rows=9 loops=1)
-> Nested Loop (cost=0.86..213766.42 rows=57231 width=24) (actual time=0.029..29086.323 rows=88858 loops=1)
-> Index Scan Backward using alerts_created_at_index on alerts (cost=0.43..85542.16 rows=57231 width=24) (actual time=0.014..88.137 rows=88858 loops=1)
Index Cond: (created_at >= '2019-08-30 00:00:00'::timestamp without time zone)
-> Index Scan using devices_pkey on devices (cost=0.43..2.23 rows=1 width=16) (actual time=0.016..0.325 rows=1 loops=88858)
Index Cond: (id = alerts.device_id)
-> Index Scan using sites_pkey on sites (cost=0.29..0.31 rows=1 width=8) (actual time=0.004..0.004 rows=0 loops=88858)
Index Cond: (id = devices.site_id)
Filter: (cloud_id = 7231)
Rows Removed by Filter: 1
Total runtime: 29495.816 ms
Now we change to LIMIT 10:
EXPLAIN ANALYZE SELECT "alerts"."id",
"alerts"."created_at",
't1'::text AS src_table
FROM "alerts"
INNER JOIN "devices"
ON "devices"."id" = "alerts"."device_id"
INNER JOIN "sites"
ON "sites"."id" = "devices"."site_id"
WHERE "sites"."cloud_id" = 111
AND "alerts"."created_at" >= '2019-08-30'
ORDER BY "created_at" DESC limit 10;
Limit (cost=39521.79..39521.81 rows=10 width=16) (actual time=1.557..1.559 rows=10 loops=1)
-> Sort (cost=39521.79..39521.93 rows=58 width=16) (actual time=1.555..1.555 rows=10 loops=1)
Sort Key: alerts.created_at
Sort Method: quicksort Memory: 25kB
-> Nested Loop (cost=5.24..39520.53 rows=58 width=16) (actual time=0.150..1.543 rows=11 loops=1)
-> Nested Loop (cost=4.81..16030.12 rows=2212 width=8) (actual time=0.137..0.643 rows=31 loops=1)
-> Index Scan using sites_cloud_id_index on sites (cost=0.29..64.53 rows=31 width=8) (actual time=0.014..0.057 rows=23 loops=1)
Index Cond: (cloud_id = 7231)
-> Bitmap Heap Scan on devices (cost=4.52..512.32 rows=270 width=16) (actual time=0.020..0.025 rows=1 loops=23)
Recheck Cond: (site_id = sites.id)
-> Bitmap Index Scan on devices_site_id_index (cost=0.00..4.46 rows=270 width=0) (actual time=0.006..0.006 rows=9 loops=23)
Index Cond: (site_id = sites.id)
-> Index Scan using alerts_device_id_index on alerts (cost=0.43..10.59 rows=3 width=24) (actual time=0.024..0.028 rows=0 loops=31)
Index Cond: (device_id = devices.id)
Filter: (created_at >= '2019-08-30 00:00:00'::timestamp without time zone)
Rows Removed by Filter: 12
Total runtime: 1.603 ms
alerts table has millions of records, other tables are counted in thousands.
I can already optimize the query by simply not using limit < 10. What I don't understand is why the LIMIT affects the performance. Perhaps there's a better way than hardcoding this magic number "10".
The number of result rows affects the PostgreSQL optimizer, because plans that return the first few rows quickly are not necessarily plans that return the whole result as fast as possible.
In your case, PostgreSQL thinks that for small values of LIMIT, it will be faster by scanning the alerts table in the order of the ORDER BY clause using an index and just join the other tables using a nested loop until it has found 9 rows.
The benefit of such a strategy is that it doesn't have to calculate the complete result of the join, then sort it and throw away all but the first few result rows.
The danger is that it takes longer than expected to find the 9 matching rows, and this is what hits you:
Index Scan Backward using alerts_created_at_index on alerts (cost=0.43..85542.16 rows=57231 width=24) (actual time=0.014..88.137 rows=88858 loops=1)
So PostgreSQL has to process 88858 rows and use a nested loop join (which is inefficient if it has to loop often) until it finds 9 result rows. This may be because it underestimates the selectivity of the conditions, or because the many matching rows all happen to have low created_at.
The number 10 just happens to be the cut-off point where PostgreSQL thinks it will no longer be more efficient to use that strategy, it is a value that will change as the data in the database change.
You can avoid using that plan altogether by using an ORDER BY clause that does not match the index:
ORDER BY (created_at + INTERVAL '0 days') DESC
I have a SQL view (CREATE OR REPLACE VIEW table_view AS ...) that has 20 columns, with a mixture of text, integer, boolean and integer array fields. In total, there are nine columns that are integer arrays.
This view works very well the overwhelming majority of the time. However, when a row shows up with ~50 total values within the integer array, the query slows to a crawl. These columns are not being queried against, they are simply just part of the resultset, but the magnitude of the query goes from ~250ms to ~7s if any rows are returned that contain the high number of integer array values.
Here is an example query:
SELECT
"table_view"."id", -- integer
"table_view"."start", -- date with tz
"table_view"."end", -- date with tz
"table_view"."a_id", -- integer
"table_view"."c_id", -- integer
"table_view"."generator_id", -- integer
"table_view"."ci_id", -- integer
"table_view"."kind", -- integer
"table_view"."hidden", -- boolean
"table_view"."is_done", -- integer
"table_view"."title", -- text
"table_view"."c_type", -- integer
"table_view"."ci_title", -- integer
"table_view"."status", -- text
"table_view"."cs", -- integer array
"table_view"."cts", -- integer array
"table_view"."ts", -- integer array
"table_view"."tas", -- integer array
"table_view"."bs", -- integer array
"table_view"."ms", -- integer array
"table_view"."pcs", -- integer array
"table_view"."pts", -- integer array
"table_view"."ks" -- integer array
FROM
"table_view"
WHERE (
"table_view"."a_id" = 3289
AND (
"table_view"."cs" && ARRAY[28890, 21166, 28891, 29581, 29583, 22378, 22380, 22733, 28924, 28925, 28926, 28927, 28478, 41014]::integer[]
OR "table_view"."ms" && ARRAY[11125]::integer[]
) AND NOT (
"table_view"."hidden" = true
AND NOT (
"table_view"."ms" && ARRAY[11125]::integer[]
)
) AND "table_view"."end" >= '2019-02-03T00:00:00+00:00'::timestamptz
AND "table_view"."start" < '2019-04-15T00:00:00+00:00'::timestamptz
);
If none of the rows have a sum of integer array values ~> 50, the query time is ~50ms to 250ms.
Time: 50.417 ms
However, if one or more of the rows have a sum of integer array values <~ 50, the query time is ~6.5s to 7s.
Time: 6737.154 ms
I have obfuscated some of the columns names from this query, but that shouldn't impact the debugging of this issue. I'm beating myself up over this. Nothing I am researching online mentioned anything about column length being an issue in the resulting query.
Here is the output of the EXPLAIN ANALYZE for the "bad" query:
Subquery Scan on table_view (cost=173.16..173.29 rows=1 width=436) (actual time=648.470..800.577 rows=60 loops=1)
-> GroupAggregate (cost=173.16..173.28 rows=1 width=103) (actual time=648.468..800.552 rows=60 loops=1)
Group Key: dhci.id, wfciwf.id, dutm.account_id
Filter: (((array_append('{}'::integer[], dhci.cid) && '{28890,21166,28891,29581,29583,22378,22380,22733,28924,28925,28926,28927,28478,41014}'::integer[]) OR (array_agg(DISTIN
CT dhamti.member_id) && '{11125}'::integer[])) AND ((NOT dhci.hidden) OR (array_agg(DISTINCT dhamti.member_id) && '{11125}'::integer[])))
-> Sort (cost=173.16..173.17 rows=1 width=103) (actual time=647.727..685.155 rows=172000 loops=1)
Sort Key: dhci.id, wfciwf.id, dutm.account_id
Sort Method: external merge Disk: 16392kB
-> Nested Loop Left Join (cost=68.45..173.15 rows=1 width=103) (actual time=1.033..478.844 rows=172000 loops=1)
-> Nested Loop Left Join (cost=68.16..171.71 rows=1 width=99) (actual time=1.017..308.168 rows=172000 loops=1)
-> Nested Loop Left Join (cost=67.87..170.20 rows=1 width=95) (actual time=0.997..126.331 rows=172000 loops=1)
-> Nested Loop Left Join (cost=67.58..168.77 rows=1 width=91) (actual time=0.978..26.478 rows=33400 loops=1)
-> Nested Loop Left Join (cost=67.16..165.68 rows=1 width=87) (actual time=0.960..4.997 rows=3700 loops=1)
-> Nested Loop Left Join (cost=66.73..162.71 rows=1 width=83) (actual time=0.949..2.572 rows=410 loops=1)
-> Nested Loop Left Join (cost=66.44..161.59 rows=1 width=79) (actual time=0.939..2.121 rows=60 loops=1)
-> Nested Loop (cost=66.02..154.64 rows=1 width=75) (actual time=0.926..1.818 rows=60 loops=1)
-> Nested Loop (cost=65.59..148.07 rows=1 width=66) (actual time=0.902..1.446 rows=60 loops=1)
-> Index Scan using dutm_8a089c2a on dutm (cost=0.29..9.40 rows=2 width=8) (actual time=0.017..0.028 rows=9 loops=1)
Index Cond: (account_id = 3289)
-> Bitmap Heap Scan on dhci (cost=65.31..69.33 rows=1 width=66) (actual time=0.137..0.148 rows=7 loops=9)
Recheck Cond: ((owner_member_id = dutm.id) AND (deadline IS NOT NULL) AND (deadline >= '2019-02-03 00:00:00+00'::timestamp with time zone) AND (deadline < '2019-04-15 00:00:00+00'::timestamp with time zone))
Heap Blocks: exact=36
-> BitmapAnd (cost=65.31..65.31 rows=1 width=0) (actual time=0.135..0.135 rows=0 loops=9)
-> Bitmap Index Scan on dhci_9adb17cb (cost=0.00..5.79 rows=182 width=0) (actual time=0.036..0.036 rows=167 loops=9)
Index Cond: (owner_member_id = dutm.id)
-> Bitmap Index Scan on dhci_deadline_ba8bd3addbfe7dc_uniq (cost=0.00..58.66 rows=2419 width=0) (actual time=0.263..0.263 rows=1727 loops=3)
Index Cond: ((deadline IS NOT NULL) AND (deadline >= '2019-02-03 00:00:00+00'::timestamp with time zone) AND (deadline < '2019-04-15 00:00:00+00'::timestamp with time zone))
-> Index Scan using wfciwf_pkey on wfciwf (cost=0.42..6.55 rows=1 width=13) (actual time=0.005..0.005 rows=1 loops=60)
Index Cond: (id = dhci.workflow_state_id)
Filter: (((current_status)::text <> 'killed'::text) AND ((current_status)::text <> 'parked'::text))
-> Index Scan using dhamti_d7bbcb82 on dhamti (cost=0.42..6.93 rows=2 width=8) (actual time=0.003..0.004 rows=1 loops=60)
Index Cond: (content_item_id = dhci.id)
-> Index Scan using dhcibs_6123fe8a on dhcibs (cost=0.29..1.10 rows=2 width=8) (actual time=0.003..0.005 rows=7 loops=60)
Index Cond: (ciid = dhci.id)
-> Index Scan using dhcit_6123fe8a on dhcit (cost=0.42..2.95 rows=2 width=8) (actual time=0.002..0.004 rows=9 loops=410)
Index Cond: (ciid = dhci.id)
-> Index Scan using ghcita_6123fe8a on ghcita (cost=0.42..3.07 rows=2 width=8) (actual time=0.002..0.003 rows=9 loops=3700)
Index Cond: (ciid = dhci.id)
-> Index Scan using dhcipc_6123fe8a on dhcipc (cost=0.29..1.41 rows=2 width=8) (actual time=0.001..0.002 rows=5 loops=33400)
Index Cond: (ciid = dhci.id)
-> Index Scan using dhcipt_6123fe8a on dhcipt (cost=0.29..1.49 rows=2 width=8) (actual time=0.001..0.001 rows=0 loops=172000)
Index Cond: (ciid = dhci.id)
-> Index Scan using dhcik_6123fe8a on dhcik (cost=0.29..1.41 rows=3 width=8) (actual time=0.001..0.001 rows=0 loops=172000)
Index Cond: (ciid = dhci.id)
Planning time: 8.219 ms
Execution time: 804.161 ms
(45 rows)
I have a weird case of query execution performance here. The query has a date values in the WHERE clause, and the speed of executing varies by values of the date.
Actually:
for the dates from the range of the last 30 days, execution takes around 3 min
for the dates before the range of the last 30 days, execution takes a few seconds
The query is listed below, with the date in the last 30 days range:
select
sk2_.code as col_0_0_,
bra4_.code as col_1_0_,
st0_.quantity as col_2_0_,
bat1_.forecast as col_3_0_
from
TBL_st st0_,
TBL_bat bat1_,
TBL_sk sk2_,
TBL_bra bra4_
where
st0_.batc_id=bat1_.id
and bat1_.sku_id=sk2_.id
and bat1_.bran_id=bra4_.id
and not (exists (select
1
from
TBL_st st6_,
TBL_bat bat7_,
TBL_sk sk10_
where
st6_.batc_id=bat7_.id
and bat7_.sku_id=sk10_.id
and bat7_.bran_id=bat1_.bran_id
and sk10_.code=sk2_.code
and st6_.date>st0_.date
and sk10_.acco_id=1
and st6_.date>='2017-04-20'
and st6_.date<='2017-04-30'))
and sk2_.acco_id=1
and st0_.date>='2017-04-20'
and st0_.date<='2017-04-30'
and here is the plan for the query with the date in the last 30 days range:
Nested Loop (cost=289.06..19764.03 rows=1 width=430) (actual time=3482.062..326049.246 rows=249 loops=1)
-> Nested Loop Anti Join (cost=288.91..19763.86 rows=1 width=433) (actual time=3482.023..326048.023 rows=249 loops=1)
Join Filter: ((st6_.date > st0_.date) AND ((sk10_.code)::text = (sk2_.code)::text))
Rows Removed by Join Filter: 210558
-> Nested Loop (cost=286.43..13719.38 rows=1 width=441) (actual time=4.648..2212.042 rows=2474 loops=1)
-> Nested Loop (cost=286.00..6871.33 rows=13335 width=436) (actual time=4.262..657.823 rows=666738 loops=1)
-> Index Scan using uk_TBL_sk0_account_code on TBL_sk sk2_ (cost=0.14..12.53 rows=1 width=426) (actual time=1.036..1.084 rows=50 loops=1)
Index Cond: (acco_id = 1)
-> Bitmap Heap Scan on TBL_bat bat1_ (cost=285.86..6707.27 rows=15153 width=26) (actual time=3.675..11.308 rows=13335 loops=50)
Recheck Cond: (sku_id = sk2_.id)
Heap Blocks: exact=241295
-> Bitmap Index Scan on ix_al_batc_sku_id (cost=0.00..282.07 rows=15153 width=0) (actual time=3.026..3.026 rows=13335 loops=50)
Index Cond: (sku_id = sk2_.id)
-> Index Scan using ix_al_stle_batc_id on TBL_st st0_ (cost=0.42..0.50 rows=1 width=21) (actual time=0.002..0.002 rows=0 loops=666738)
Index Cond: (batc_id = bat1_.id)
Filter: ((date >= '2017-04-20 00:00:00'::timestamp without time zone) AND (date <= '2017-04-30 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 1
-> Nested Loop (cost=2.49..3023.47 rows=1 width=434) (actual time=111.345..130.883 rows=86 loops=2474)
-> Hash Join (cost=2.06..2045.18 rows=1905 width=434) (actual time=0.010..28.028 rows=54853 loops=2474)
Hash Cond: (bat7_.sku_id = sk10_.id)
-> Index Scan using ix_al_batc_bran_id on TBL_bat bat7_ (cost=0.42..1667.31 rows=95248 width=24) (actual time=0.009..11.045 rows=54853 loops=2474)
Index Cond: (bran_id = bat1_.bran_id)
-> Hash (cost=1.63..1.63 rows=1 width=426) (actual time=0.026..0.026 rows=50 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 11kB
-> Seq Scan on TBL_sk sk10_ (cost=0.00..1.63 rows=1 width=426) (actual time=0.007..0.015 rows=50 loops=1)
Filter: (acco_id = 1)
-> Index Scan using ix_al_stle_batc_id on TBL_st st6_ (cost=0.42..0.50 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=135706217)
Index Cond: (batc_id = bat7_.id)
Filter: ((date >= '2017-04-20 00:00:00'::timestamp without time zone) AND (date <= '2017-04-30 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 1
-> Index Scan using TBL_bra_pk on TBL_bra bra4_ (cost=0.14..0.16 rows=1 width=13) (actual time=0.003..0.003 rows=1 loops=249)
Index Cond: (id = bat1_.bran_id)
Planning time: 8.108 ms
Execution time: 326049.583 ms
Here is the same query with the date before the last 30 days range:
select
sk2_.code as col_0_0_,
bra4_.code as col_1_0_,
st0_.quantity as col_2_0_,
bat1_.forecast as col_3_0_
from
TBL_st st0_,
TBL_bat bat1_,
TBL_sk sk2_,
TBL_bra bra4_
where
st0_.batc_id=bat1_.id
and bat1_.sku_id=sk2_.id
and bat1_.bran_id=bra4_.id
and not (exists (select
1
from
TBL_st st6_,
TBL_bat bat7_,
TBL_sk sk10_
where
st6_.batc_id=bat7_.id
and bat7_.sku_id=sk10_.id
and bat7_.bran_id=bat1_.bran_id
and sk10_.code=sk2_.code
and st6_.date>st0_.date
and sk10_.acco_id=1
and st6_.date>='2017-01-20'
and st6_.date<='2017-01-30'))
and sk2_.acco_id=1
and st0_.date>='2017-01-20'
and st0_.date<='2017-01-30'
and here is the plan for the query with the date before the last 30 days range:
Hash Join (cost=576.33..27443.95 rows=48 width=430) (actual time=132.732..3894.554 rows=250 loops=1)
Hash Cond: (bat1_.bran_id = bra4_.id)
-> Merge Anti Join (cost=572.85..27439.82 rows=48 width=433) (actual time=132.679..3894.287 rows=250 loops=1)
Merge Cond: ((sk2_.code)::text = (sk10_.code)::text)
Join Filter: ((st6_.date > st0_.date) AND (bat7_.bran_id = bat1_.bran_id))
Rows Removed by Join Filter: 84521
-> Nested Loop (cost=286.43..13719.38 rows=48 width=441) (actual time=26.105..1893.523 rows=2491 loops=1)
-> Nested Loop (cost=286.00..6871.33 rows=13335 width=436) (actual time=1.159..445.683 rows=666738 loops=1)
-> Index Scan using uk_TBL_sk0_account_code on TBL_sk sk2_ (cost=0.14..12.53 rows=1 width=426) (actual time=0.035..0.084 rows=50 loops=1)
Index Cond: (acco_id = 1)
-> Bitmap Heap Scan on TBL_bat bat1_ (cost=285.86..6707.27 rows=15153 width=26) (actual time=1.741..7.148 rows=13335 loops=50)
Recheck Cond: (sku_id = sk2_.id)
Heap Blocks: exact=241295
-> Bitmap Index Scan on ix_al_batc_sku_id (cost=0.00..282.07 rows=15153 width=0) (actual time=1.119..1.119 rows=13335 loops=50)
Index Cond: (sku_id = sk2_.id)
-> Index Scan using ix_al_stle_batc_id on TBL_st st0_ (cost=0.42..0.50 rows=1 width=21) (actual time=0.002..0.002 rows=0 loops=666738)
Index Cond: (batc_id = bat1_.id)
Filter: ((date >= '2017-01-20 00:00:00'::timestamp without time zone) AND (date <= '2017-01-30 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 1
-> Materialize (cost=286.43..13719.50 rows=48 width=434) (actual time=15.584..1986.953 rows=84560 loops=1)
-> Nested Loop (cost=286.43..13719.38 rows=48 width=434) (actual time=15.577..1983.384 rows=2491 loops=1)
-> Nested Loop (cost=286.00..6871.33 rows=13335 width=434) (actual time=0.843..482.864 rows=666738 loops=1)
-> Index Scan using uk_TBL_sk0_account_code on TBL_sk sk10_ (cost=0.14..12.53 rows=1 width=426) (actual time=0.005..0.052 rows=50 loops=1)
Index Cond: (acco_id = 1)
-> Bitmap Heap Scan on TBL_bat bat7_ (cost=285.86..6707.27 rows=15153 width=24) (actual time=2.051..7.902 rows=13335 loops=50)
Recheck Cond: (sku_id = sk10_.id)
Heap Blocks: exact=241295
-> Bitmap Index Scan on ix_al_batc_sku_id (cost=0.00..282.07 rows=15153 width=0) (actual time=1.424..1.424 rows=13335 loops=50)
Index Cond: (sku_id = sk10_.id)
-> Index Scan using ix_al_stle_batc_id on TBL_st st6_ (cost=0.42..0.50 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=666738)
Index Cond: (batc_id = bat7_.id)
Filter: ((date >= '2017-01-20 00:00:00'::timestamp without time zone) AND (date <= '2017-01-30 00:00:00'::timestamp without time zone))
Rows Removed by Filter: 1
-> Hash (cost=2.10..2.10 rows=110 width=13) (actual time=0.033..0.033 rows=110 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 14kB
-> Seq Scan on TBL_bra bra4_ (cost=0.00..2.10 rows=110 width=13) (actual time=0.004..0.013 rows=110 loops=1)
Planning time: 14.542 ms
Execution time: 3894.793 ms
Does anyone have an idea why does this happens.
Did anyone had an experience with anything similar?
Thank you very much.
Kind regards, Petar
I am not sure, but I had a similar case a while ago(On ORACLE but i guess it is not important).
in my case the difference originated at the difference between the amount of data, meaning: if you have 1% of data from the past 30 days, it uses the indexs. when you need "older" data (the rest 99% of the data) it decides to not use the index and to do a full scan(in the form of nested loop and not hash join).
If you sure that the data distribution is ok then maybe try collecting statistics(worked for me at the time). eventually you can start to analyze every peace of this query and to see what part exactly is the bottleneck and work from there.
BTree indexes can have some issues with dates, especially if you're removing old data from the table (ie, deleting everything older than 90 days). It can cause the tables to get lopsided, with all of the rows being down one branch of the tree. Even without removing old dates, if there are many more "new" rows than "old" rows, it can still happen.
But I don't see your query plans using an index on st0_.date, so I don't think that's the issue. If you can afford a table lock on st0_, you can test this theory by running a REINDEX operation on any indexes that contain st0_.date.
Instead, I think you just have a lot more rows that match the 2017-01-20 to 2017-01-30 range vs. the 2017-04-20 to 2017-04-30 range. The first doubly indented Nested Loop is the same in both queries, so I'll ignore it. The second doubly indended stanza is different, and much more expensive in the slow query:
-> Materialize (cost=286.43..13719.50 rows=48 width=434) (actual time=15.584..1986.953 rows=84560 loops=1)
-> Nested Loop (cost=286.43..13719.38 rows=48 width=434) (actual time=15.577..1983.384 rows=2491 loops=1)
-> Nested Loop (cost=286.00..6871.33 rows=13335 width=434) (actual time=0.843..482.864 rows=666738
vs
-> Nested Loop (cost=2.49..3023.47 rows=1 width=434) (actual time=111.345..130.883 rows=86 loops=2474)
-> Hash Join (cost=2.06..2045.18 rows=1905 width=434) (actual time=0.010..28.028 rows=54853 loops=2474)
Materialize can be an expensive operation that doesn't necessarily scale with the estimated cost. Take a look at https://www.postgresql.org/docs/10/static/using-explain.html , and search for "Materialize". Also note that the estimated number of rows is much higher in the slow version.
I'm not 100% sure, but I believe that tweaking the "work_mem" parameter can have some effect in this area (https://www.postgresql.org/docs/9.4/static/runtime-config-resource.html#GUC-WORK-MEM). To test this theory, you can change that value per session using
SET LOCAL work_mem = '8MB';
I'm trying to speed up what seems to me to be quite a slow PostgreSQL query in Django (150ms)
This is the query:
SELECT ••• FROM "predictions_prediction"
INNER JOIN "minute_in_time_minute"
ON ( "predictions_prediction"."minute_id" = "minute_in_time_minute"."id" )
WHERE ("minute_in_time_minute"."datetime" >= '2014-08-21 13:12:00+00:00'
AND "predictions_prediction"."location_id" = 1
AND "minute_in_time_minute"."datetime" < '2014-08-24 13:12:00+00:00'
AND "predictions_prediction"."tide_level" >= 3.0)
ORDER BY "minute_in_time_minute"."datetime" ASC
Here's the result of the PostgreSQL EXPLAIN:
Sort (cost=17731.45..17739.78 rows=3331 width=32) (actual time=151.755..151.901 rows=3515 loops=1)
Sort Key: minute_in_time_minute.datetime
Sort Method: quicksort Memory: 371kB
-> Hash Join (cost=3187.44..17536.56 rows=3331 width=32) (actual time=96.757..150.693 rows=3515 loops=1)
Hash Cond: (predictions_prediction.minute_id = minute_in_time_minute.id)
-> Seq Scan on predictions_prediction (cost=0.00..11232.00 rows=411175 width=20) (actual time=0.017..88.063 rows=410125 loops=1)
Filter: ((tide_level >= 3::double precision) AND (location_id = 1))
Rows Removed by Filter: 115475
-> Hash (cost=3134.21..3134.21 rows=4258 width=12) (actual time=9.221..9.221 rows=4320 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 203kB
-> Bitmap Heap Scan on minute_in_time_minute (cost=92.07..3134.21 rows=4258 width=12) (actual time=1.147..8.220 rows=4320 loops=1)
Recheck Cond: ((datetime >= '2014-08-21 13:18:00+00'::timestamp with time zone) AND (datetime < '2014-08-24 13:18:00+00'::timestamp with time zone))
-> Bitmap Index Scan on minute_in_time_minute_datetime_key (cost=0.00..91.00 rows=4258 width=0) (actual time=0.851..0.851 rows=4320 loops=1)
Index Cond: ((datetime >= '2014-08-21 13:18:00+00'::timestamp with time zone) AND (datetime < '2014-08-24 13:18:00+00'::timestamp with time zone))
I've tried visualising it in an external tool (http://explain.depesz.com/s/CWW) which shows that the start of the problem is the Seq Scan on predictions_prediction
What I've tried so far:
Add an index on predictions_prediction.tide_level
Add a composite index on tide_level and location on predictions.prediction
But neither had any effect as far as I could see.
Can someone please help me interpret the query plan?
Thanks
Try creating the following composite index on predictions_prediction
(minute_id, location_id, tide_level)
150ms for this type of query (a join, several where conditions with inequalities and an order by) is actually relatively normal, so you might not be able to speed it up much more.
This question already has answers here:
Keep PostgreSQL from sometimes choosing a bad query plan
(6 answers)
Closed 8 years ago.
I am doing a query to select records that lie in a certain geographic region and I am doing some joins and couple of filtering as well.
This is my query:
SELECT "events".* FROM "events" INNER JOIN "albums" ON "albums"."event_id" = "events"."id" INNER JOIN "photos" ON "photos"."album_id" = "albums"."id" WHERE "events"."deleted_at" IS NULL AND "albums"."deleted_at" IS NULL AND "photos"."deleted_at" IS NULL AND (events.latitude BETWEEN -44.197088742316055 AND -23.22003941183816 AND events.longitude BETWEEN 133.226480859375 AND 165.570230859375) GROUP BY events.id HAVING count(albums.id) > 0 ORDER BY start_date DESC
I have the following indexes:
Events:
"events_pkey" PRIMARY KEY, btree (id)
"index_events_on_deleted_at" btree (deleted_at)
"index_events_on_latitude_and_longitude" btree (latitude, longitude)
Albums:
"albums_pkey" PRIMARY KEY, btree (id)
"index_albums_on_deleted_at" btree (deleted_at)
"index_albums_on_event_id" btree (event_id)
Photos:
"photos_pkey" PRIMARY KEY, btree (id)
"index_photos_on_album_id" btree (album_id)
"index_photos_on_deleted_at" btree (deleted_at)
Doing an EXPLAIN ANALYZE results in this and I dont see any usage of my indexes which. I am not sure how to force it to use the indexes. Can anyone help me optimize this?
Sort (cost=4057.46..4057.84 rows=150 width=668) (actual time=556.114..556.187 rows=76 loops=1)
Sort Key: events.start_date
Sort Method: quicksort Memory: 78kB
-> HashAggregate (cost=4050.16..4052.04 rows=150 width=668) (actual time=555.667..555.783 rows=76 loops=1)
Filter: (count(albums.id) > 0)
-> Hash Join (cost=76.14..3946.54 rows=20724 width=668) (actual time=3.675..467.578 rows=48050 loops=1)
Hash Cond: (photos.album_id = albums.id)
-> Seq Scan on photos (cost=0.00..3441.87 rows=59013 width=4) (actual time=0.008..169.206 rows=60599 loops=1)
Filter: (deleted_at IS NULL)
-> Hash (cost=74.10..74.10 rows=163 width=668) (actual time=3.633..3.633 rows=318 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 176kB
-> Hash Join (cost=49.80..74.10 rows=163 width=668) (actual time=1.195..2.519 rows=318 loops=1)
Hash Cond: (albums.event_id = events.id)
-> Seq Scan on albums (cost=0.00..21.47 rows=321 width=8) (actual time=0.011..0.458 rows=321 loops=1)
Filter: (deleted_at IS NULL)
-> Hash (cost=47.92..47.92 rows=150 width=664) (actual time=1.151..1.151 rows=195 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 126kB
-> Seq Scan on events (cost=0.00..47.92 rows=150 width=664) (actual time=0.007..0.488 rows=195 loops=1)
Filter: ((deleted_at IS NULL) AND (latitude >= (-44.1970887423161)::double precision) AND (latitude <= (-23.2200394118382)::double precision) AND (longitude >= 133.226480859375::double precision) AND (longitude <= 165.570230859375::double precision))
Total runtime: 556.459 ms
Thanks!!
EDIT: Thanks for the links. I have tried disabling seqscan. Now my plan is:
Sort (cost=5565.73..5566.10 rows=150 width=46) (actual time=451.208..451.290 rows=76 loops=1)
Sort Key: (date(events.start_date))
Sort Method: quicksort Memory: 31kB
-> GroupAggregate (cost=0.00..5560.31 rows=150 width=46) (actual time=2.990..450.850 rows=76 loops=1)
Filter: (count(albums.id) > 0)
-> Nested Loop (cost=0.00..5454.44 rows=20724 width=46) (actual time=0.077..278.319 rows=48050 loops=1)
-> Merge Join (cost=0.00..205.35 rows=163 width=46) (actual time=0.051..2.856 rows=318 loops=1)
Merge Cond: (events.id = albums.event_id)
-> Index Scan using events_pkey on events (cost=0.00..118.72 rows=150 width=42) (actual time=0.024..0.792 rows=195 loops=1)
Filter: ((deleted_at IS NULL) AND (latitude >= (-44.1970887423161)::double precision) AND (latitude <= (-23.2200394118382)::double precision)
AND (longitude >= 133.226480859375::double precision) AND (longitude <= 165.570230859375::double precision))
-> Index Scan using index_albums_on_event_id on albums (cost=0.00..83.83 rows=321 width=8) (actual time=0.017..0.832 rows=321 loops=1)
Filter: (deleted_at IS NULL)
-> Index Scan using index_photos_on_album_id on photos (cost=0.00..30.27 rows=155 width=4) (actual time=0.010..0.409 rows=151 loops=318)
Index Cond: (album_id = albums.id)
Filter: (deleted_at IS NULL)
Total runtime: 451.562 ms
Still Indexes are not being fully used especially for the latitude and long conditions. Do I have my indexes setup correctly?
EDIT: After looking at answers at http://stackoverflow.com/questions/8228326/how-can-i-avoid-postgresql-sometimes-choosing-a-bad-query-plan-for-one-of-two-ne, I assumed that its behaving like this because my query was returning all records, then I updated the conditions, and the new query plan is:
Sort (cost=786.18..786.22 rows=19 width=668) (actual time=3.754..3.755 rows=2 loops=1)
Sort Key: events.start_date
Sort Method: quicksort Memory: 25kB
-> HashAggregate (cost=785.54..785.77 rows=19 width=668) (actual time=3.700..3.703 rows=2 loops=1)
Filter: (count(albums.id) > 0)
-> Nested Loop (cost=48.39..765.51 rows=2670 width=668) (actual time=1.116..2.968 rows=543 loops=1)
-> Hash Join (cost=48.39..89.25 rows=21 width=668) (actual time=1.093..1.128 rows=3 loops=1)
Hash Cond: (events.id = albums.event_id)
-> Bitmap Heap Scan on events (cost=9.42..49.44 rows=19 width=664) (actual time=0.061..0.080 rows=9 loops=1)
Recheck Cond: ((latitude >= (-33.7474111086624)::double precision) AND (latitude <= (-33.581678187556)::double precision) AND (longitude >= 151.193933862305::double precision) AND (longitude <= 151.44661940918::double precision))
Filter: (deleted_at IS NULL)
-> Bitmap Index Scan on index_events_on_latitude_and_longitude (cost=0.00..9.42 rows=28 width=0) (actual time=0.050..0.050 rows=9 loops=1)
Index Cond: ((latitude >= (-33.7474111086624)::double precision) AND (latitude <= (-33.581678187556)::double precision) AND (longitude >= 151.193933862305::double precision) AND (longitude <= 151.44661940918::double precision))
-> Hash (cost=34.95..34.95 rows=321 width=8) (actual time=0.992..0.992 rows=321 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 13kB
-> Bitmap Heap Scan on albums (cost=14.74..34.95 rows=321 width=8) (actual time=0.069..0.570 rows=321 loops=1)
Recheck Cond: (deleted_at IS NULL)
-> Bitmap Index Scan on index_albums_on_deleted_at (cost=0.00..14.66 rows=321 width=0) (actual time=0.056..0.056 rows=321 loops=1)
Index Cond: (deleted_at IS NULL)
-> Index Scan using index_photos_on_album_id on photos (cost=0.00..30.27 rows=155 width=4) (actual time=0.014..0.273 rows=181 loops=3)
Index Cond: (album_id = albums.id)
Filter: (deleted_at IS NULL)
Total runtime: 3.958 ms
And the time is verrry less!!
Any suggestions?
This is mostly because you did not set a LIMIT clause on your query. Without LIMIT you always requests all data from your tables, so looking at their indexes too would be insufficient.
SQLFiddle 1, 2 vs. 3, 4
Also note, that FOREIGN KEY does not add indexes (but UNIQUE & PRIMARY KEY constraint does). So you might want to add indexes for albums.event_id & photos.album_id.
SQLFiddle 3 vs. 4
Sorting can use an index, if present. At your query this means an index on events.start_date.