how to speed up left join qualify operation in sql - sql

Question

I have a following sql code:
with mytable(stock, datetime, price) as (
select * from values
(1, '2022-12-13 12:31:45.00'::timestamp, 10.0),
(1, '2022-12-13 12:31:45.01'::timestamp, 10.1),
(1, '2022-12-13 12:31:45.02'::timestamp, 10.2),
(1, '2022-12-13 12:31:46.00'::timestamp, 11.0),
(1, '2022-12-13 12:31:46.01'::timestamp, 11.1),
(1, '2022-12-13 12:31:46.02'::timestamp, 11.2),
(1, '2022-12-13 12:31:46.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:47.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:48.00'::timestamp, 11.3)
)
select t1.*
,t2.datetime as next_datetime
,t2.price as next_price
,t3.datetime as next2_datetime
,t3.price as next2_price
,t4.datetime as next3_price
,t4.price as next3_price
from mytable as t1
left join mytable as t2
on t1.stock = t2.stock and timediff(second, t1.datetime, t2.datetime) < 1
left join mytable as t3
on t1.stock = t3.stock and timediff(second, t1.datetime, t3.datetime) < 2
left join mytable as t4
on t1.stock = t4.stock and timediff(second, t1.datetime, t4.datetime) < 3
qualify row_number() over (partition by t1.stock, t1.datetime order by (t2.datetime, t3.datetime, t4.datetime) desc) = 1
ORDER BY 1,2;
With a small table, this works just fine. However, I have a table with at least 100,000 rows and doing a single left join takes about 30 minutes. Is there a better faster way to do the above operation in sql?

Answer 1

There is more extensive refactoring that could be done to this query, and there is also a lot that can be done with data shaping and other techniques on these kinds of time series joins. For now, it's important to reduce the intermediate cardinality in the plan. Specifically, we want to reduce the cardinality of the join, which feeds into the window function and filter. That's currently 3,897.
Step one is to run this after running your sample query to see if the refactor produces identical results:
select hash_agg(*) from table(result_scan(last_query_id()));
This produces 1433824845005768014 when running your sample.
Now on the join, we want to restrict the number of rows that survive the join condition. Since we're looking for rows that happen in the future only, we don't have to join the ones that happen in the past:
with mytable(stock, datetime, price) as (
select * from values
(1, '2022-12-13 12:31:45.00'::timestamp, 10.0),
(1, '2022-12-13 12:31:45.01'::timestamp, 10.1),
(1, '2022-12-13 12:31:45.02'::timestamp, 10.2),
(1, '2022-12-13 12:31:46.00'::timestamp, 11.0),
(1, '2022-12-13 12:31:46.01'::timestamp, 11.1),
(1, '2022-12-13 12:31:46.02'::timestamp, 11.2),
(1, '2022-12-13 12:31:46.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:47.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:48.00'::timestamp, 11.3)
)
select t1.*
,t2.datetime as next_datetime
,t2.price as next_price
,t3.datetime as next2_datetime
,t3.price as next2_price
,t4.datetime as next3_price
,t4.price as next3_price
from mytable as t1
left join mytable as t2
on t1.stock = t2.stock and timediff(second, t1.datetime, t2.datetime) < 1 and t1.datetime <= t2.datetime
left join mytable as t3
on t1.stock = t3.stock and timediff(second, t1.datetime, t3.datetime) < 2 and t1.datetime <= t3.datetime
left join mytable as t4
on t1.stock = t4.stock and timediff(second, t1.datetime, t4.datetime) < 3 and t1.datetime <= t4.datetime
qualify row_number() over (partition by t1.stock, t1.datetime order by (t2.datetime, t3.datetime, t4.datetime) desc) = 1
ORDER BY 1,2;
This reduces the cardinality heading into the window function and qualify filter to 989. That's a 75% reduction of rows heading into the window function. Next, run the check again to make sure the results are identical:
select hash_agg(*) from table(result_scan(last_query_id()));
Edit: A slight change to the method of ensuring only future rows get joined reduced the intermediate cardinality from 989 to 497, an 87% reduction.
Edit 2: This is a JavaScript UDTF approach that does not increase cardinality at all. I tested this on real-world NTSE data. It processed 2.55 billion rows in 12min on a Large warehouse.
create or replace function LAG_BY_TIME(ROW_TIME float, LAGGED_VALUE float, LAG_TIME1 float, LAG_TIME2 float, LAG_TIME3 float)
returns table(LAGGED_TIME1 float, LAGGED_VALUE1 float, LAGGED_TIME2 float, LAGGED_VALUE2 float, LAGGED_TIME3 float, LAGGED_VALUE3 float)
language javascript
as
$$
{
initialize: function (argumentInfo, context) {
this.buffer1 = [];
this.buffer2 = [];
this.buffer3 = [];
},
processRow: function (row, rowWriter, context) {
var shifted;
var laggedTime = new Date(row.ROW_TIME);
this.buffer1.push({laggedTime:laggedTime, laggedValue:row.LAGGED_VALUE});
this.buffer2.push({laggedTime:laggedTime, laggedValue:row.LAGGED_VALUE});
this.buffer3.push({laggedTime:laggedTime, laggedValue:row.LAGGED_VALUE});
var laggedTime1 = new Date(laggedTime.getTime() + row.LAG_TIME1);
var laggedTime2 = new Date(laggedTime.getTime() + row.LAG_TIME2);
var laggedTime3 = new Date(laggedTime.getTime() + row.LAG_TIME3);
do {
if (this.buffer1[0].laggedTime >= laggedTime1) {
this.buffer1.shift();
shifted = true;
} else {
shifted = false;
}
} while (shifted)
do {
if (this.buffer2[0].laggedTime >= laggedTime2) {
this.buffer2.shift();
shifted = true;
} else {
shifted = false;
}
} while (shifted)
do {
if (this.buffer3[0].laggedTime >= laggedTime3) {
this.buffer3.shift();
shifted = true;
} else {
shifted = false;
}
} while (shifted)
rowWriter.writeRow({
LAGGED_TIME1:this.buffer1[0].laggedTime, LAGGED_VALUE1:this.buffer1[0].laggedValue,
LAGGED_TIME2:this.buffer2[0].laggedTime, LAGGED_VALUE2:this.buffer2[0].laggedValue,
LAGGED_TIME3:this.buffer3[0].laggedTime, LAGGED_VALUE3:this.buffer3[0].laggedValue,
});
},
finalize: function (rowWriter, context) {/*...*/},
}
$$;
create or replace function TO_EPOCH(TS timestamp) returns float as
$$ round(datediff(milliseconds, '1970-01-01'::timestamp, TS),0)::float $$;
create or replace function FROM_EPOCH(EPOCH float) returns timestamp as
$$ dateadd(milliseconds, round(EPOCH,0), '1970-01-01'::timestamp) $$;
with mytable(stock, datetime, price) as (
select * from values
(1, '2022-12-13 12:31:45.00'::timestamp, 10.0),
(1, '2022-12-13 12:31:45.01'::timestamp, 10.1),
(1, '2022-12-13 12:31:45.02'::timestamp, 10.2),
(1, '2022-12-13 12:31:46.00'::timestamp, 11.0),
(1, '2022-12-13 12:31:46.01'::timestamp, 11.1),
(1, '2022-12-13 12:31:46.02'::timestamp, 11.2),
(1, '2022-12-13 12:31:46.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:47.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:48.00'::timestamp, 11.3)
)
select STOCK
,DATETIME
,PRICE as PRICE
,from_epoch(LAGGED_TIME1) as NEXT_DATETIME
,LAGGED_VALUE1::number(12,2) as NEXT_PRICE
,from_epoch(LAGGED_TIME2) as NEXT2_DATETIME
,LAGGED_VALUE2::number(12,2) as NEXT2_PRICE
,from_epoch(LAGGED_TIME1) as NEXT3_DATETIME
,LAGGED_VALUE3::number(12,2) as NEXT3_PRICE
from MYTABLE, table(lag_by_time(to_epoch(DATETIME), PRICE::float, 1000::float, 2000::float, 3000::float)
over (partition by stock order by DATETIME desc))
order by STOCK, DATETIME
;

Answer 2

If you quantize your data to the seconds, and pick "winner row" per second, then you can use lag/lead.
Or:
If you need each milli-second entry to be matched to the next n second rows. quantize the data to windows and double the data, then do an equi join on that, so the sliding windows are on very heavly constrained groupings
Or:
Use a user defined table function to provide aggregate/cache values, and build yourself a latch/caching scrolling window.

Answer 3

I run your query and here is a part of the execution plan:
As you can see, your 9 rows became 3897 rows. If each stock has average 10 records/rows in your real table, that means even a single left join will multiply the rows by 10! If there are some stocks that have more records/rows than others, this may cause serious skewness and performance degradation.
The result of your sample query doesn't make any sense to me. Maybe your sample data has some issues because the difference between them is mostly milliseconds but your query looks for seconds. Anyway I suggest you to check LEAD function:
https://docs.snowflake.com/en/sql-reference/functions/lead.html
with mytable(stock, datetime, price) as (
select * from values
(1, '2022-12-13 12:31:45.00'::timestamp, 10.0),
(1, '2022-12-13 12:31:45.01'::timestamp, 10.1),
(1, '2022-12-13 12:31:45.02'::timestamp, 10.2),
(1, '2022-12-13 12:31:46.00'::timestamp, 11.0),
(1, '2022-12-13 12:31:46.01'::timestamp, 11.1),
(1, '2022-12-13 12:31:46.02'::timestamp, 11.2),
(1, '2022-12-13 12:31:46.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:47.03'::timestamp, 11.3),
(1, '2022-12-13 12:31:48.00'::timestamp, 11.3)
)
select t1.*
,LEAD( t1.datetime) over (partition by t1.stock order by t1.datetime ) as next_datetime
,LEAD( t1.price) over (partition by t1.stock order by t1.datetime ) as next_price
,LEAD( t1.datetime, 2) over (partition by t1.stock order by t1.datetime ) as next2_datetime
,LEAD( t1.price, 2) over (partition by t1.stock order by t1.datetime ) as next2_price
,LEAD( t1.datetime, 3) over (partition by t1.stock order by t1.datetime ) as next3_datetime
,LEAD( t1.price, 3) over (partition by t1.stock order by t1.datetime ) as next3_price
from mytable as t1;
ORDER BY 1,2;