time-dependent postgres query speed - sql

Question

I have a situation where the select query could be done in 3 seconds or more than 1 hours still not finish (I could not wait that long and killed it). I believe it may have something to do with the automatic statistics collection behavior of postgres server. I have a 3 table join one of them has over 70 million rows.
-- tmp_variant_filtered has about 4000 rows
-- variant_quick > 70 million rows
-- filtered_variant_quick has about 70 k rows
select count(*)
from "tmp_variant_filtered" t join "variant_quick" v on getchrnum(t.seqname)=v.chrom
and t.pos_start=v.pos and t.ref=v.ref
and t.alt=v.alt
join "filtered_variant_quick" f on f.variantid=v.id
where v.samplerun=165
;
-- running the query immediately after tmp_variant_filtered was loaded
-- Query plan that will take > 1 hour and not finish
Aggregate (cost=332.05..332.06 rows=1 width=8)
-> Nested Loop (cost=0.86..332.05 rows=1 width=0)
-> Nested Loop (cost=0.57..323.74 rows=1 width=8)
Join Filter: ((t.pos_start = v.pos) AND ((t.ref)::text = (v.ref)::text) AND ((t.alt)::text = (v.alt)::text) AND (getchrnum(t.seqname) = v.chrom))
-> Seq Scan on tmp_variant_filtered t (cost=0.00..315.00 rows=1 width=1126)
-> Index Scan using variant_quick_samplerun_chrom_pos_ref_alt_key on variant_quick v (cost=0.57..8.47 rows=1 width=20)
Index Cond: (samplerun = 165)
-> Index Only Scan using filtered_variant_quick_pkey on filtered_variant_quick f (cost=0.29..8.31 rows=1 width=8)
Index Cond: (variantid = v.id)
-- running the query a few minutes after tmp_variant_filtered was loaded with copy command
-- query plan that will take less than 5 seconds to finish
Aggregate (cost=425.69..425.70 rows=1 width=8)
-> Nested Loop (cost=8.78..425.68 rows=1 width=0)
-> Hash Join (cost=8.48..417.37 rows=1 width=8)
Hash Cond: ((t.pos_start = v.pos) AND ((t.ref)::text = (v.ref)::text) AND ((t.alt)::text = (v.alt)::text))
Join Filter: (getchrnum(t.seqname) = v.chrom)
-> Seq Scan on tmp_variant_filtered t (cost=0.00..359.06 rows=4406 width=13)
-> Hash (cost=8.47..8.47 rows=1 width=20)
-> Index Scan using variant_quick_samplerun_chrom_pos_ref_alt_key on variant_quick v (cost=0.57..8.47 rows=1 width=20)
Index Cond: (samplerun = 165)
-> Index Only Scan using filtered_variant_quick_pkey on filtered_variant_quick f (cost=0.29..8.31 rows=1 width=8)
Index Cond: (variantid = v.id)
If you run the query immediately after the tmp table got populated, it will give you the plan as shown on top, and the query will take a very long time. If you wait a few minutes, the the plan will be the lower with hash-join. The cost estimate for the upper is less than the lower.
Since the query was embedded in some scripting language, the top plan is used and usually it got finished in a couple of hours. If I do this on a terminal, after I terminated the script, the lower plan would be used, and it usually take a couple of seconds to finish.
I even did an experiment by copying the tmp_variant_filtered table into another table, say 'test'. If I run the query immediately after the copy (manually, there will be a couple of seconds of delay), then I was stuck. Killing the current job, wait for a few minutes, the the same query become blazing fast.
It was long time ago that I was doing query tuning; now I am just starting to pick it up again. I am reading and trying to understand why postgres has such a behavior. Would appreciate the experts to give a hint.

Answer 1

Immediately after inserting the rows into the table, there are no statistics available for column values and their distribution. Thus the optimizer assumes the table is empty. The only sensible strategy to retrieve all rows from an (supposedly) empty table is to do a Seq Scan. You can see this assumption in the execution plan:
Seq Scan on tmp_variant_filtered t (cost=0.00..315.00 rows=1 width=1126)
The rows=1 means that the optimizer expects that only one row will be returned by the Seq Scan. Because it's only one row, the planner chooses a nested loop to do the join - which means the Seq Scan is done once for each row in the other table (you could see that more clearly if your use explain (analyze, verbose) to generate the execution plan)
The statistics are updated in the background by the "autovacuum daemon" if you don't do it manually. That's why after waiting a while, you see a better plan, as the optimizer now know the table isn't empty.
Once the optimizer has better knowledge of the size of the table, it chooses the much more efficient Hash Join to bring the two tables together - which means the Seq Scan is only executed once, rather than multiple times.
It is always recommended to run analyze (or vacuum analyze) on tables where you changed the number of rows significantly if you need a good execution plan immediately after populating the table.
Quote from the manual
Whenever you have significantly altered the distribution of data within a table, running ANALYZE is strongly recommended. This includes bulk loading large amounts of data into the table. Running ANALYZE (or VACUUM ANALYZE) ensures that the planner has up-to-date statistics about the table. With no statistics or obsolete statistics, the planner might make poor decisions during query planning, leading to poor performance on any tables with inaccurate or nonexistent statistics

Answer 2

Regardless the mechanism for this time dependent behavior, I figure out a solution with VACUUM ANALYZE my_table. Not sure it is the cure or just give a little bit time delay. I was using psocopg2 to execute the query and had to avoid the 'cannot vacuum inside a transaction' exception. Here I list the code block you need:
self.conn.commit()
self.conn.set_session(autocommit=True)
self.cursor.execute("vacuum analyze {}".format(one_of_my_tables))
# here you probably should have used sql.SQL("...").format()
# to be more secure, I am using the text composition for example
self.conn.set_session(autocommit=False)
I applied to two of the three tables involved in my join in the question. Maybe apply vacuum analyze to one should be sufficient. As mentioned by Basil, I should have asked the question in the dba group.