Complete datetime list despite missing values for analysis - sql

Question

I have been a reader of StackOverflow for a long time already. Nearly always, I find my answers here. Great!
But now, I have a problem where I could not find a solution yet:
I have an Oracle table with an ID, a date and a value.
Think of it as a list of outstanding tasks (value) and project (ID). When the number of open tasks of a project changes, the list gets a new entry.
It looks like this:
ID month RemainingValue
1 01/01/2018 1000
1 01/03/2018 800
1 01/04/2018 600
1 01/07/2018 400
2 01/02/2018 700
2 01/03/2018 650
2 01/05/2018 600
3 01/02/2018 50
3 01/08/2018 40
4 01/01/2018 2000
(DateFormat DD/MM/YYYY)
Please note that not every month has a value!
I have to calculate the sum of all open tasks per month.
If there is no value for a month, that means that the number of open tasks has not decreased in that month, so the query should take the previous existing value of this project into account.
I want this result:
month result calculation remark
01/01/2018 3000 =1000 + 2000 ID 1+4
01/02/2018 3750 =1000 + 700 + 50 + 2000 ID 1[value of 01/01/2018]+2+3+4[value of 01/01/2018]
01/03/2018 3500 =800 + 650 + 50 + 2000 ID 1+2+3[value of 01/02/2018]+4[value of 01/01/2018]
What I did already:
I created a list of all months using the CONNECT BY LEVEL functionality, similar to this:
SELECT LEVEL AS NR
, ADD_MONTHS('01-JAN-2018', LEVEL) AS MONAT
FROM DUAL
CONNECT BY LEVEL <= (... SOME.SUBSELECT.TO.GET.THE.NUMBER.OF.LEVELS ...)
Then I can outer join this list of months to the table above based on the date.
The problem is, that the values of tasks of the unfilled months are NULL. But I don't want them to be NULL, I want the previous filled value in this case.
I tried with LAG functions, but without success so far.
I am hoping that there is some functionality in (Oracle) SQL which can do this where I don't know of.
Or maybe it's even simpler and I just don't get it...
The resulting query should also be performant, because the underlying table has millions of rows. So I'd like to avoid slow PL/SQL solutions...
Hope you can help!
Kind Regards,
Nadine

Answer 1

You could use an analytic query to get the latest value for each ID, up to and including that month (relying on the default windowing clause.
This uses your sample data in a CTE, and adds another one to provide your month generation (may not match your desired range of course):
-- first CTE to replictae your data
with my_table(ID, month, RemainingValue) as (
select 1, to_date('01/01/2018', 'DD/MM/YYYY'), 1000 from dual
union all select 1, to_date('01/03/2018', 'DD/MM/YYYY'), 800 from dual
union all select 1, to_date('01/04/2018', 'DD/MM/YYYY'), 600 from dual
union all select 1, to_date('01/07/2018', 'DD/MM/YYYY'), 400 from dual
union all select 2, to_date('01/02/2018', 'DD/MM/YYYY'), 700 from dual
union all select 2, to_date('01/03/2018', 'DD/MM/YYYY'), 650 from dual
union all select 2, to_date('01/05/2018', 'DD/MM/YYYY'), 600 from dual
union all select 3, to_date('01/02/2018', 'DD/MM/YYYY'), 50 from dual
union all select 3, to_date('01/08/2018', 'DD/MM/YYYY'), 40 from dual
union all select 4, to_date('01/01/2018', 'DD/MM/YYYY'), 2000 from dual
),
-- second CTE to generate all months, here based on full range in table
-- use whatever you currently have for this
all_months (month) as (
select add_months(min_month, + level - 1)
from (
select min(month) as min_month, max(month) as max_month from my_table
)
connect by level <= months_between(max_month, min_month) + 1
)
select am.month, mt.id,
max(mt.remainingvalue) keep (dense_rank last order by mt.month) as remainingvalue
from all_months am
left join my_table mt on mt.month <= am.month
group by am.month, mt.id
order by id, month;
which gets
MONTH ID REMAININGVALUE
---------- ---------- --------------
2018-01-01 1 1000
2018-02-01 1 1000
2018-03-01 1 800
2018-04-01 1 600
2018-05-01 1 600
2018-06-01 1 600
2018-07-01 1 400
2018-08-01 1 400
2018-02-01 2 700
2018-03-01 2 650
2018-04-01 2 650
...
And then use that as an inline view or another CTE, summing the values:
-- first CTE to replictae your data
with my_table(ID, month, RemainingValue) as (
select 1, to_date('01/01/2018', 'DD/MM/YYYY'), 1000 from dual
union all select 1, to_date('01/03/2018', 'DD/MM/YYYY'), 800 from dual
union all select 1, to_date('01/04/2018', 'DD/MM/YYYY'), 600 from dual
union all select 1, to_date('01/07/2018', 'DD/MM/YYYY'), 400 from dual
union all select 2, to_date('01/02/2018', 'DD/MM/YYYY'), 700 from dual
union all select 2, to_date('01/03/2018', 'DD/MM/YYYY'), 650 from dual
union all select 2, to_date('01/05/2018', 'DD/MM/YYYY'), 600 from dual
union all select 3, to_date('01/02/2018', 'DD/MM/YYYY'), 50 from dual
union all select 3, to_date('01/08/2018', 'DD/MM/YYYY'), 40 from dual
union all select 4, to_date('01/01/2018', 'DD/MM/YYYY'), 2000 from dual
),
-- second CTE to generate all months, here based on full range in table
-- use whatever you currently have for this
all_months (month) as (
select add_months(min_month, + level - 1)
from (
select min(month) as min_month, max(month) as max_month from my_table
)
connect by level <= months_between(max_month, min_month) + 1
),
-- third CTE to get the latest value for each ID up to that month
inter (month, id, remainingvalue) as (
select am.month, mt.id,
max(mt.remainingvalue) keep (dense_rank last order by mt.month)
from all_months am
left join my_table mt on mt.month <= am.month
group by am.month, mt.id
)
select month, sum(remainingvalue) as result,
listagg(remainingvalue, ' + ') within group (order by id) as calculation
from inter
group by month
order by month;
which gets:
MONTH RESULT CALCULATION
---------- ---------- ------------------------------
2018-01-01 3000 1000 + 2000
2018-02-01 3750 1000 + 700 + 50 + 2000
2018-03-01 3500 800 + 650 + 50 + 2000
2018-04-01 3300 600 + 650 + 50 + 2000
2018-05-01 3250 600 + 600 + 50 + 2000
2018-06-01 3250 600 + 600 + 50 + 2000
2018-07-01 3050 400 + 600 + 50 + 2000
2018-08-01 3040 400 + 600 + 40 + 2000
I assume the calculation and remark columns in your result are just for our benefit to understand the logic; if you do want them then calculation is easy to get as above, and if you want remark too then you just need to identify the month the value comes from too, and add another listagg:
...
-- third CTE to get the latest value for each ID up to that month
inter (month, id, remainingvalue, valuemonth) as (
select am.month, mt.id,
max(mt.remainingvalue) keep (dense_rank last order by mt.month),
max(mt.month)
from all_months am
left join my_table mt on mt.month <= am.month
group by am.month, mt.id
)
select month, sum(remainingvalue) as result,
'= ' || listagg(remainingvalue, ' + ') within group (order by id) as calculation,
'ID ' || listagg(id || case when month != valuemonth then '[' || valuemonth || ']' end, ' + ')
within group (order by id) as remark
from inter
group by month
order by month;
MONTH RESULT CALCULATION REMARK
---------- ---------- ------------------------ -----------------------------------------------------------------
2018-01-01 3000 = 1000 + 2000 ID 1 + 4
2018-02-01 3750 = 1000 + 700 + 50 + 2000 ID 1[2018-01-01] + 2 + 3 + 4[2018-01-01]
2018-03-01 3500 = 800 + 650 + 50 + 2000 ID 1 + 2 + 3[2018-02-01] + 4[2018-01-01]
2018-04-01 3300 = 600 + 650 + 50 + 2000 ID 1 + 2[2018-03-01] + 3[2018-02-01] + 4[2018-01-01]
2018-05-01 3250 = 600 + 600 + 50 + 2000 ID 1[2018-04-01] + 2 + 3[2018-02-01] + 4[2018-01-01]
2018-06-01 3250 = 600 + 600 + 50 + 2000 ID 1[2018-04-01] + 2[2018-05-01] + 3[2018-02-01] + 4[2018-01-01]
2018-07-01 3050 = 400 + 600 + 50 + 2000 ID 1 + 2[2018-05-01] + 3[2018-02-01] + 4[2018-01-01]
2018-08-01 3040 = 400 + 600 + 40 + 2000 ID 1[2018-07-01] + 2[2018-05-01] + 3 + 4[2018-01-01]

Answer 2

You seem to want to sum the most recent value for each project before a given month.
The following gets the remaining value for each id for each month:
with months as (
SELECT LEVEL AS NR, ADD_MONTHS(DATE '2018-01-01', LEVEL) AS MONTH
FROM DUAL
CONNECT BY LEVEL <= (... SOME.SUBSELECT.TO.GET.THE.NUMBER.OF.LEVELS ...)
)
select m.month, i.id,
(select max(t.remainingvalue) keep (dense_rank first order by month desc)
from t
where t.id = i.id and t.month <= m.month
) as remainingvalue
from months m cross join
(select distinct id from t) i;
Now let's just summarize this:
with months as (
SELECT LEVEL AS NR, ADD_MONTHS(DATE '2018-01-01', LEVEL) AS MONTH
FROM DUAL
CONNECT BY LEVEL <= (... SOME.SUBSELECT.TO.GET.THE.NUMBER.OF.LEVELS ...)
)
select month, sum(remainingvalue)
from (select m.month, i.id,
(select max(t.remainingvalue) keep (dense_rank first order by month desc)
from t
where t.id = i.id and t.month <= m.month
) as remaining_value
from months m cross join
(select distinct id from t) i
) mi
group by month;