30-day rolling/moving sum when current date is missing - sql

Question

I have a table (view_of_referred_events) which stores the number of visitors for a given page.
date country_id referral product_id visitors
2016-04-01 216 pl 113759 1
2016-04-03 216 pl 113759 1
2016-04-06 216 pl 113759 13
2016-04-07 216 pl 113759 10
I want to compute the 30-day rolling/moving sum for this product, even for those days which are missing. So the end result should be something like the following:
date country_id referral product_id cumulative_visitors
2016-04-01 216 pl 113759 1
2016-04-02 216 pl 113759 1
2016-04-03 216 pl 113759 2
2016-04-04 216 pl 113759 2
2016-04-05 216 pl 113759 2
2016-04-06 216 pl 113759 15
2016-04-07 216 pl 113759 25
Now, this is a simplistic representation, because I have tens of different country_id, referral and product_id. I can't pre-create a table with all possible combinations of {date, country_id, referral and product_id} because this would become untreatable considering the size of the table. I don't also want to have a row in the final table if that specific {date, country_id, referral and product_id} didn't exist before.
I was thinking if there was an easy way to tell Impala to use the value of the previous row (the previous day) if in view_of_referred_events there are no visitors for that day.
I wrote this query, where list_of_dates is a table with a list of days from April 1st to April 7th.
select
t.`date`,
t.country_id,
t.referral,
t.product_id,
sum(visitors) over (partition by t.country_id, t.referral, t.product_id order by t.`date`
rows between 30 preceding and current row) as cumulative_sum_visitors
from (
selec
d.`date`,
re.country_id,
re.referral,
re.product_id,
sum(visitors) as visitors
from list_of_dates d
left outer join view_of_referred_events re on d.`date` = re.`date`
and re.referral = "pl"
and re.product_id = "113759"
and re.country_id = "216"
group by d.`date`, re.country_id, re.referral, re.product_id
) t
order by t.`date` asc;
This returns something similar to what I want, but not exactly that.
date country_id referral product_id cumulative_visitors
2016-04-01 216 pl 113759 1
2016-04-02 NULL NULL NULL NULL
2016-04-03 216 pl 113759 2
2016-04-04 NULL NULL NULL NULL
2016-04-05 NULL NULL NULL NULL
2016-04-06 216 pl 113759 15
2016-04-07 216 pl 113759 25

Answer 1

I have added another sub query to get the value from the last row in the partition. I am not sure what version of hive/impala you are using, last_value(column_name, ignore null values true/false) is the syntax.
I assume you are trying to find the cumulative counts over a 30 days (month), I recommend using month field to group the rows. The month could come either from your dimension table list_of_dates or just substr(date, 1, 7) and get the cumulative counts of visitors over ..rows unbounded preceding and current row.
query:
select
`date`,
country_id,
referral,
product_id,
sum(visitors) over (partition by country_id, referral, product_id order by `date`
rows between 30 preceding and current row) as cumulative_sum_visitors
from (select
t.`date`,
-- get the last not null value from the partition window w for country_id, referral & product_id
last_value(t.country_id, true) over w as country_id,
last_value(t.referral, true) over w as referral
last_value(t.product_id, true) over w as product_id
if(visitors = null, 0, visitors) as visitors
from (
select
d.`date`,
re.country_id,
re.referral,
re.product_id,
sum(visitors) as visitors
from list_of_dates d
left outer join view_of_referred_events re on d.`date` = re.`date`
and re.referral = "pl"
and re.product_id = "113759"
and re.country_id = "216"
group by d.`date`, re.country_id, re.referral, re.product_id
) t
window w as (partition by t.country_id, t.referral, t.product_id order by t.`date`
rows between unbounded preceding and unbounded following)) t1
order by `date` asc;

Answer 2

I'm not sure how goo the performance will be, but you can do this by aggregating the data twice and adding 30 days for the second aggregation and negating the count.
Something like this:
with t as (
select d.`date`, re.country_id, re.referral, re.product_id,
sum(visitors) as visitors
from list_of_dates d left outer join
view_of_referred_events re
on d.`date` = re.`date` and
re.referral = 'pl' and
re.product_id = 113759 and
re.country_id = 216
group by d.`date`, re.country_id, re.referral, re.product_id
)
select date, country_id, referral, product_id,
sum(sum(visitors)) over (partition by country_id, referral, product_id order by date) as visitors
from ((select date, country_id, referral, product_id, visitors
from t
) union all
(select date_add(date, 30), country_id, referral, product_id, -visitors
from t
)
) tt
group by date, country_id, referral, product_id;