Query performance with sorting in postgres - sql

Question

I've a performance issue with a query on a table which has 33m rows. The query should return 6m rows.
I'm trying to achive that the response to the request to begin without any significant delay. It's required for data streaming in my app.
After the start, the data transfer may take longer. The difficult is the query has sorting.
So, I created an Index with fields that are used in the "order by" statement and in the "where" clause.
Example likes that:
CREATE TABLE Table1 (
Id SERIAL PRIMARY KEY,
Field1 INT NOT NULL,
Field2 INT NOT NULL,
Field3 INT NOT NULL,
Field4 VARCHAR(200) NOT NULL,
CreateDate TIMESTAMP,
CloseDate TIMESTAMP NULL
);
CREATE INDEX IX_Table1_SomeIndex ON Table1 (Field2, Field4);
And query likes that:
SELECT * FROM Table1 t
WHERE t.CreateDate >= '2020-01-01' AND t.CreateDate < '2021-01-01'
ORDER BY t.Field2, t.Field4
It leads to the following:
when I add "LIMIT 1000" it retruns result immediately and builds the following plan:
the plan with 'LIMIT'
when I run without "LIMIT" it "thinks" for about a minute and returns data for about 16 minutes. And it builds the following plan:
the plan with 'LIMIT'
Why are plans different?
Could you help me to make souliton for streaming immediately (without LIMIT)?
Thanks!

Answer 1

You will need to use a server side cursor or something similar for this to work. Otherwise it runs the query to completion before returning any results. There is no "streaming" by default. How you do this depends on your client, which you don't mention.
If you simply DECLARE a cursor and then FETCH in chunks, then the setting cursor_tuple_fraction will control whether it chooses the plan with a faster start up cost (like what you get with the LIMIT), or a faster overall run cost (like you get without the LIMIT).

Answer 2

If "when I add LIMIT 1000 it returns result immediately" and you want to avoid latency then I would suggest that you run a slightly modified query many times in a loop with LIMIT 1000. An important benefit would be that there will be no long running transactions.
The query to run many times in a loop should return records starting after the largest value of (field2, field4) from the previous iteration run.
SELECT *
FROM table1 t
WHERE t.CreateDate >= '2020-01-01' AND t.CreateDate < '2021-01-01'
AND (t.field2, t.field4) > (:last_run_largest_f2_value, :last_run_largest_f4_value)
ORDER BY t.field2, t.field4
LIMIT 1000;
last_run_largest_f2_value and last_run_largest_f4_value are parameters. Their values shall come from the last record returned by the previous iteration.
AND (t.field2, t.field4) > (:last_run_largest_f2_value, :last_run_largest_f4_value) shall be omitted in the first iteration.
Important limitation
This is an alternative of OFFSET that will work correctly if (field2, field4) values are unique