Get data from multiple tables by two timestamp - sql

Question

PostgreSQL 10.12
I have a table with calculated data grouped by date with hour, e.g.:
hourly_stats
clicks_count | visitors_count | product_id | promoter_id | bundle_id | date_time
------------------------------------------------------------------------------------------
15 | 6 | 123 | 456 | 789 | 2018-11-02 12:00:00
8 | 3 | 123 | 456 | 789 | 2018-11-02 16:00:00
2 | 1 | 123 | 456 | 789 | 2018-11-13 10:00:00
5 | 2 | 123 | 456 | 789 | 2018-11-13 21:00:00
Every new hour I collect statistics for the previous hour and insert it into the table.
In addition, to always display fresh data, I use a materialized view, which stores the calculated data from the beginning of the current hour to the current moment (refreshed every 5 minutes).
The core part of the query is always based on two timestamp values and looks like this:
SELECT *
FROM (
SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM hourly_stats
UNION ALL (
SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM materialized_stats
)
)
WHERE (date_time > start_date AND date_time <= end_date)
This core part is used in multiple really complex queries, which are too slow. For example, it takes more than a 1.5 minute to complete the query (if no row is filtered by start_date and end_date) if table has more than 20 million records in one of the cases.
I decided to add two more table with calculated data grouped by year-month-day:
daily_stats
clicks_count | visitors_count | product_id | promoter_id | bundle_id | date_time
------------------------------------------------------------------------------------------
23 | 9 | 123 | 456 | 789 | 2018-11-02
7 | 3 | 123 | 456 | 789 | 2018-11-13
and by year-month:
monthly_stats
clicks_count | visitors_count | product_id | promoter_id | bundle_id | date_time
------------------------------------------------------------------------------------------
30 | 12 | 123 | 456 | 789 | 2018-11
So, if I have start_date = '2019-01-01 00:00:00' and end_date = '2020-08-12 16:00:00' I will be able to collect data like this
(SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM monthly_stats
WHERE 'monthly_condition')
UNION ALL
(SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM daily_stats
WHERE 'daily_condition')
UNION ALL
(SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM hourly_stats
WHERE 'hourly_condition')
UNION ALL (
SELECT
clicks_count,
visitors_count,
product_id,
promoter_id,
bundle_id,
date_time
FROM materialized_stats
)
Each calculated row is added to the corresponding table only after the base time period (month, day, or hour) is over. So for specific set of product_id | promoter_id | bundle_id I should get:
19 rows from monthly_stats +
11 rows from daily_stats +
16 rows from hourly_stats +
1 row from materialized_stats
Already implemented restrictions (on a application layer):
max end_date value may be equal to the end of the current day
start_date is always less than end_date
start_date and end_date values ​​are specified with an hour precision
Question: how to implement these 'monthly_condition', 'daily_condition' and 'hourly_condition' above? They should be based on the start_date and end_date parts, but I quite don't understand how to do this.
Thanks for any help.

Answer 1

This is an interesting problem. I had to solve this once before for SQL Server. PostgreSQL makes it much easier. Everything down to the fullness cte has been tested. The allstats cte is a best guess since I do not have your tables or data.
with invars as (
select '2016-08-15 12:35:00'::timestamptz as start_date,
'2020-08-12 19:00:00'::timestamptz as end_date
), days as (
select c.dhour,
tstzrange(
date_trunc('hour', i.start_date),
date_trunc('hour', i.end_date), '[)') as qrange
from invars i
cross join lateral generate_series(
date_trunc('hour', i.start_date),
date_trunc('hour', i.end_date),
interval '1 hour'
) as c(dhour)
), calendar as (
select dhour,
date_trunc('day', dhour) as dday,
date_trunc('month', dhour) as dmonth,
qrange
from days
), fullness as (
select dhour, dday, dmonth, qrange,
qrange #> tstzrange(dday, dday + interval '1 day', '[)') as full_day,
qrange #> tstzrange(dmonth, dmonth + interval '1 month', '[)') as full_month
from calendar
), allstats as (
select clicks_count, visitors_count, product_id, promoter_id, bundle_id
from monthly_stats
where date_time in (select distinct to_char(dmonth, 'YYYY-MM')
from fullness where full_month)
union all
select clicks_count, visitors_count, product_id, promoter_id, bundle_id
from daily_stats
where date_time in (select distinct to_char(dday, 'YYYY-MM-DD')
from fullness where full_day and not full_month)
union all
select clicks_count, visitors_count, product_id, promoter_id, bundle_id
from hourly_stats
where date_time in (select dhour from fullness
where not full_day and not full_month
and dhour < date_trunc(hour, now()))
union all
select clicks_count, visitors_count, product_id, promoter_id, bundle_id
from materialized_stats
)
select * from allstats;
I think your problem description leaves off the fact that the start_date can begin in the middle of a month or even a day. This query covers that.