Insert missing dates into existing table - sql

Question

I have a query that finds missing dates from a table.
The query is:
;WITH NullGaps AS
(
SELECT
ROW_NUMBER() OVER (ORDER BY ChannelName, ReadingDate) AS ID,
SerialNumber, ReadingDate, ChannelName, uid
FROM
[UriData]
)
SELECT
(DATEDIFF(MINUTE, g1.ReadingDate , g2.ReadingDate) / 15) -1 AS 'MissingCount',
g1.ReadingDate AS 'FromDate', g2.ReadingDate AS 'ToDate'
FROM
NullGaps g1
INNER JOIN
NullGaps g2 ON g1.ID = (g2.ID - 1)
WHERE
DATEADD(MINUTE, 15, g1.ReadingDate) < g2.ReadingDate
The output is:
--------------------------------------------------------------
| MissingCount | FromDate | ToDate |
--------------------------------------------------------------
| 2 | 2018-09-20 14:30:00 | 2018-09-20 15:15:00 |
| 1 | 2018-09-20 15:30:00 | 2018-09-20 16:00:00 |
| 1 | 2018-09-20 20:30:00 | 2018-09-20 21:00:00 |
--------------------------------------------------------------
The output is the number of datetimes that are missing from the FromDate to the ToDate (which both exist). For example, in the first row of the output (above), the times I want to create and insert will be '2018-09-20 14:45:00' and '2018-09-20 15:00:00' (they are all 15-minute intervals)
I need to understand, how I now create the new dates and insert them into an existing table. I can create one date, but I can't create dates where there are multiple missing values between two times.
TIA
SQL Fiddle

Answer 1

If you also want to find the missing datetimes at the start and the end of a date?
Then comparing to generated datetimes should be a valiable method.
Such dates can be generated via a Recursive CTE.
Then you can join your data to the Recursive CTE and select those that are missing.
Or use a NOT EXISTS.
For example:
WITH RCTE AS
(
select [SerialNumber], [ChannelName], 0 as Lvl, cast(cast([ReadingDate] as date) as datetime) as ReadingDate
from [UriData]
group by SerialNumber, [ChannelName], cast([ReadingDate] as date)
union all
select [SerialNumber], [ChannelName], Lvl + 1, DATEADD(MINUTE,15,[ReadingDate])
from RCTE
where cast([ReadingDate] as date) = cast(DATEADD(MINUTE,15,[ReadingDate]) as date)
)
SELECT [SerialNumber], [ChannelName], [ReadingDate] AS FromDate
FROM RCTE r
WHERE NOT EXISTS
(
select 1
from [UriData] t
where t.[SerialNumber] = r.[SerialNumber]
and t.[ChannelName] = r.[ChannelName]
and t.[ReadingDate] = r.[ReadingDate]
);
A test can be found here
And here's another query that takes a different approuch :
WITH CTE AS
(
SELECT SerialNumber, ChannelName, ReadingDate,
LAG(ReadingDate) OVER (PARTITION BY SerialNumber, ChannelName ORDER BY ReadingDate) AS prevReadingDate
FROM [UriData]
)
, RCTE AS
(
select SerialNumber, ChannelName, 0 as Lvl,
prevReadingDate AS ReadingDate,
prevReadingDate AS MinReadingDate,
ReadingDate AS MaxReadingDate
from CTE
where DATEDIFF(MINUTE, prevReadingDate, ReadingDate) > 15
union all
select SerialNumber, ChannelName, Lvl + 1,
DATEADD(MINUTE,15,ReadingDate),
MinReadingDate,
MaxReadingDate
from RCTE
where ReadingDate < DATEADD(MINUTE,-15,MaxReadingDate)
)
select SerialNumber, ChannelName,
ReadingDate AS FromDate,
DATEADD(MINUTE,15,ReadingDate) AS ToDate,
dense_rank() over (partition by SerialNumber, ChannelName order by MinReadingDate) as GapRank,
(DATEDIFF(MINUTE, MinReadingDate, MaxReadingDate) / 15) AS TotalMissingQuarterGaps
from RCTE
where Lvl > 0 AND MinReadingDate < MaxReadingDate
ORDER BY SerialNumber, ChannelName, MinReadingDate;
You can test that one here

Answer 2

I don't understand your query for calculating missing values. Your question doesn't have sample data or explain the logic. I'm pretty sure that lag() would be much simpler.
But given your query (or any other), one method to expand out the data is to use a recursive CTE:
with missing as (<your query here>)
cte as (
select dateadd(minute, 15, fromdate) as dte, missingcount - 1 as missingcount
from missing
union all
select dateadd(minute, 15, dte), missingcount - 1
from cte
where missingcount > 0
)
select *
from cte;
If you have more than 100 missing times in one row, then add option (maxrecursion 0) to the end of the query.

Answer 3

Based on the information shared with me, I did the following which does what I need.
The first part is to find the date ranges that are missing by finding the from and to dates that have missing dates between them, then insert them into a table for auditing, but it will hold the missing dates I am looking for:
;WITH NullGaps AS(
SELECT ROW_NUMBER() OVER (ORDER BY ChannelName, ReadingDate) AS ID,SerialNumber, ReadingDate, ChannelName, uid
FROM [Staging].[UriData]
)
INSERT INTO [Staging].[MissingDates]
SELECT (DATEDIFF(MINUTE, g1.ReadingDate , g2.ReadingDate) / 15) -1 AS 'MissingCount',
g1.ChannelName,
g1.SerialNumber,
g1.ReadingDate AS FromDate,
g2.ReadingDate AS ToDate
FROM NullGaps g1
INNER JOIN NullGaps g2
ON g1.ID = (g2.ID - 1)
WHERE DATEADD(MINUTE, 15, g1.ReadingDate) < g2.ReadingDate
AND g1.ChannelName IN (SELECT ChannelName FROM staging.ActiveChannels)
AND NOT EXISTS(
SELECT 1 FROM [Staging].[MissingDates] m
WHERE m.Channel = g1.ChannelName
AND m.Serial = g1.SerialNumber
AND m.FromDate = g1.ReadingDate
AND m.ToDate = g2.ReadingDate
)
Now that I have the ranges to look for, I can now create the missing dates and insert them into the table that holds real data.
;WITH MissingDateTime AS(
SELECT DATEADD(MINUTE, 15, FromDate) AS dte, MissingCount -1 AS MissingCount, Serial, Channel
FROM [Staging].[MissingDates]
UNION ALL
SELECT DATEADD(MINUTE, 15, dte), MissingCount - 1, Serial, Channel
FROM MissingDateTime
WHERE MissingCount > 0
) -- END CTE
INSERT INTO [Staging].[UriData]
SELECT NEWID(), Serial, Channel, '999', '0', dte, CURRENT_TIMESTAMP, 0,1,0 FROM MissingDateTime m
WHERE NOT EXISTS(
SELECT 1 FROM [Staging].[UriData] u
WHERE u.ChannelName = m.Channel
AND u.SerialNumber = m.Serial
AND u.ReadingDate = m.dte
) -- END SELECT
I am sure you can offer improvements to this. This solution finds only the missing dates and allows me to back fill my data table with only the missing dates. I can also change the intervals later should other devices need to be used for different intervals. I have put the queries in two sperarate SPROC's so I can control both apects, being: one for auditing and one for back filling.