SQL aggregation query, non linear cost - sql

Question

I'm doing a complex aggregation of some timeseris GPS data in a postgres 13 + postgis 3 + timescaleDB 2 database. The table I'm looking at has several million entries per day and I want to do an aggregation (one row per day, per gps_id, per group gap ID) for several months.
Lets say that I've created a function to perform the aggregation:
--pseudo code, won't actually work...
CREATE FUNCTION my_agg_func(starttime, endtime)
AS
WITH gps_window AS
(SELECT gps.id,
gps.geom,
gps.time,
-- find where there are 1 hour gaps in data
lag(ais.time) OVER (PARTITION BY gps.id ORDER BY gps.time) <= (gps.time - '01:00:00'::interval) AS time_step,
-- find where there are 0.1 deg gaps in position
st_distance(gps.geom, lag(gps.geom) OVER (PARTITION BY gps.id ORDER BY gps.time)) >= 0.1 AS dist_step
FROM gps
WHERE gps.time BETWEEN starttime AND endtime
), groups AS (
SELECT gps_window.id,
gps_window.geom,
gps_window.time,
count(*) FILTER (WHERE gps_window.time_step) OVER (PARTITION BY gps_window.id ORDER BY gps_window.time) AS time_grp,
count(*) FILTER (WHERE gps_window.dist_step) OVER (PARTITION BY gps_window.id ORDER BY gps_window.time) AS dist_grp
FROM gps_window
--get rid of duplicate points
WHERE gps_window.dist > 0
)
SELECT
gps_id,
date(gps.time),
time_grp,
dist_grp
st_setsrid(st_makeline(gps_window."position" ORDER BY gps_window.event_time), 4326) AS geom,
FROM groups
WHERE gps_time BETWEEN starttime AND endtime
GROUP BY gps.id, date(gps.time), time_grp, dist_grp
where the gap_id functions are checking for sequential gps points from the same gps_id that are too distant from each other, traveled unreasonably fast or the time between messages was too much. The aggregates are basically creating a line from the gps points. The end result of is a bunch of lines where all the points in the line are "reasonable".
To run the aggregation function for 1 day (starttime = '2020-01-01', endtime = '2020-01-02') it takes about 12 secs to complete. If I choose a week of data, it takes 10 minutes. If I choose a month of data it takes 15h+ to complete.
I would expect linear performance since the data is going to be grouped per day anyway but this isn't the case. The obvious way to get around this performance bottleneck would be to run this in a for loop:
for date in date_range(starttime, endtime):
my_agg_func(date, date+1)
I can do this in Python but any ideas how to either get a for loop running in postgres or to alter the aggregation query to be linear?

Answer 1

The aggregation of time intervals (known as COLLAPSE operator in SQL litterature) leads to write complex queries whose execution cost can be exponential or polynomial depending on the method used. The old classical SQL formulations of SNODGRASS or Chris Date are exponential. Recently Itzik Ben Gan, a Microsoft SQL Server MVP write a polynomial form which gives excellent response times, but uses the CROSS APPLY, operator invented by Microsoft and since taken over by Oracle ... The queries are as follows:
WITH
C1 AS (SELECT ITV_ITEM, ITV_DEBUT AS ts, +1 AS genre, NULL AS e,
ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ITV_DEBUT) AS s
FROM T_INTERVAL_ITV
UNION ALL
SELECT ITV_ITEM, ITV_FIN AS ts, -1 AS genre,
ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ITV_FIN) AS e,
NULL AS s
FROM T_INTERVAL_ITV),
C2 AS (SELECT C1.*, ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ts, genre DESC)
AS se
FROM C1),
C3 AS (SELECT ITV_ITEM, ts,
FLOOR((ROW_NUMBER() OVER(PARTITION BY ITV_ITEM ORDER BY ts) - 1) / 2 + 1)
AS grpnum
FROM C2
WHERE COALESCE(s - (se - s) - 1, (se - e) - e) = 0),
C4 AS (SELECT ITV_ITEM, MIN(ts) AS ITV_DEBUT, max(ts) AS ITV_FIN
FROM C3
GROUP BY ITV_ITEM, grpnum)
SELECT A.ITV_ITEM, A.ITV_DEBUT, A.ITV_FIN
FROM (SELECT DISTINCT ITV_ITEM
FROM T_INTERVAL_ITV) AS U
CROSS APPLY (SELECT *
FROM C4
WHERE ITV_ITEM = U.ITV_ITEM) AS A
ORDER BY ITV_ITEM, ITV_DEBUT, ITV_FIN;
Can you transform this MS SQL Server specific query by using the LATERAL join ? This will help you to have a better execution time.