Latest value for each timeperiod, person and category - sql

Question

I have tried browsing the problems & answers in this forum, but neither of them fit's my case sufficiently.
I have some people reporting in their status for 2 categories, which looks like this:
TimeStamp | PersonID | Category | Value
2015-07-02 01:25:00 | 2303 | CatA | 8.2
2015-07-02 01:25:00 | 2303 | CatB | 10.1
2015-07-02 03:35:00 | 2303 | CatA | 8.0
2015-07-02 03:35:00 | 2303 | CatB | 9.9
2015-07-02 02:30:00 | 4307 | CatA | 8.7
2015-07-02 02:30:00 | 4307 | CatB | 12.7
.
.
.
2015-07-31 22:15:00 | 9011 | CatA | 7.9
2015-07-31 22:15:00 | 9011 | CatB | 8.9
Some people report status several times per hour, but others only a couple of times per day.
I need to produce an an output, which shows latest know status for each day, for each hour of the day, for each person and category. This should look like this:
Date |Hour| Person | Category | Value
2015-07-02 | 1 | 2307 | CatA | Null
2015-07-02 | 1 | 2307 | CatB | Null
2015-07-02 | 2 | 2307 | CatA | 8.2
2015-07-02 | 2 | 2307 | CatB | 10.2
2015-07-02 | 3 | 2307 | CatA | 8.2
2015-07-02 | 3 | 2307 | CatB | 10.2
2015-07-02 | 4 | 2307 | CatA | 8.0
2015-07-02 | 4 | 2307 | CatB | 9.9
.
.
.
2015-07-31 | 23 | 9011 | CatA | 7.9
2015-07-31 | 23 | 9011 | CatB | 8.9
The first row(s) for each person and category will probably be null as there will be no known values as this is "beginning of time"
I have tried using a sub query like this:
SELECT Date
,hour
,Person
,Category
,(SELECT TOP 1 status FROM readings WHERE (readings.Date<=structure.Date) AND readings.Hour<=structure.hour)....and so forth.... order by TimeStamp DESC
FROM structure
This works - except in terms of performance because I need to do this for a month, for 2.000 persons for 2 categories and that means that the sub query must run (30*24*2000*2=2,880,000) times, and given the fact that table containing the readings also contains hundreds of thousands of readings, this don't work.
I have also tried messing round with row_number(), but have not succeed in this.
Any suggestions?
Edit (19-10-2015 15:34): In my query example above I am referring to a "structure" table. This is actually just (for the time being) a view, with the following SQL:
SELECT Calendar.CalendarDay, Hours.Hour, Persons.Person, Categories.Category
FROM Calendar CROSS JOIN Hours CROSS JOIN Persons CROSS JOIN Categories
This in order to produce a table containing a row for each day, for each hour for each person and each category. This table then contains (30*24*2000*2=2,880,000) rows.
For each of these rows, I need to locate the latest status from the readings table. So for each Day, for each hour, for each person and each category I need to read the latest available status from the readings table.

Answer 1

Let me guess.
Based on the task "to produce an output, which shows latest know status for each day, for each hour of the day, for each person and category" you need to take three steps:
(1) Find latest records for every hour;
(2) Get a table of all date and hours to show;
(3) Multiply that date-hours-table by persons and categories and left join the result with latest-records-for-every-hour.
-- Test data
declare #t table ([Timestamp] datetime2(0), PersonId int, Category varchar(4), Value decimal(3,1));
insert into #t values
('2015-07-02 01:25:00', 2303, 'CatA', 8.2 ),
('2015-07-02 01:45:00', 2303, 'CatA', 9.9 ),
('2015-07-02 01:25:00', 2303, 'CatB', 10.1 ),
('2015-07-02 03:35:00', 2303, 'CatA', 8.0 ),
('2015-07-02 03:35:00', 2303, 'CatB', 9.9 ),
('2015-07-02 02:30:00', 4307, 'CatA', 8.7 ),
('2015-07-02 02:30:00', 4307, 'CatB', 12.7 );
-- Latest records for every hour
declare #Latest table (
[Date] date,
[Hour] tinyint,
PersonId int,
Category varchar(4),
Value decimal(3,1)
primary key ([Date], [Hour], PersonId, Category)
);
insert into #Latest
select top 1 with ties
[Date] = cast([Timestamp] as date),
[Hour] = datepart(hour, [Timestamp]),
PersonId ,
Category ,
Value
from
#t
order by
row_number() over(partition by cast([Timestamp] as date), datepart(hour, [Timestamp]), PersonId, Category order by [Timestamp] desc);
-- Date-hours table
declare #FromDateTime datetime2(0);
declare #ToDateTime datetime2(0);
select #FromDateTime = min([Timestamp]), #ToDateTime = max([Timestamp]) from #t;
declare #DateDiff int = datediff(day, #FromDateTime, #ToDateTime);
declare #FromDate date = cast(#FromDateTime as date);
declare #FromHour int = datepart(hour, #FromDateTime);
declare #ToHour int = datepart(hour, #ToDateTime);
declare #DayHours table ([Date] date, [Hour] tinyint, primary key clustered ([Date], [Hour]) );
with N as
(
select n from (values (1),(2),(3),(4),(5),(6),(7),(8),(9),(10)) t(n)
),
D as (
select
row_number() over(order by (select 1))-1 as d
from
N n1, N n2, N n3
),
H as (
select top 24
row_number() over(order by (select 1)) - 1 as h
from
N n1, N n2
)
insert into #DayHours
select dateadd(day, d, #FromDate), h
from
D, h
where
#FromHour <= (d * 100 + h)
and (d * 100 + h) <= (#DateDiff * 100 + #ToHour);
-- #PersonsIds & #Categories tables (just an imitation of the real tables)
declare #PersonsIds table (Id int primary key);
declare #Categories table (Category varchar(4) primary key);
insert into #PersonsIds select distinct PersonId from #t;
insert into #Categories select distinct Category from #t;
-- The result
select
dh.[Date],
dh.[Hour],
PersonId = p.Id,
c.Category,
l.Value
from
#PersonsIds p cross join #Categories c cross join #DayHours dh
left join #Latest l on l.[Date] = dh.[Date] and l.[Hour] = dh.[Hour] and l.PersonId = p.Id and l.Category = c.Category
order by
[Date], [Hour], PersonId, Category;
Edit (1):
OK.
In order to bring over the previous values to empty spaces,
let's replace the last select statement with this one:
select top 1 with ties
dh.[Date],
dh.[Hour],
PersonId = p.Id,
c.Category,
l.Value
from
#PersonsIds p cross join #Categories c cross join #DayHours dh
left join #Latest l
on (l.[Date] = dh.[Date] and l.[Hour] <= dh.[Hour] or l.[Date] < dh.[Date])
and l.PersonId = p.Id and l.Category = c.Category
order by
row_number()
over (partition by dh.[Date], dh.[Hour], p.Id, c.Category
order by l.[Date] desc, l.[Hour] desc);
Edit (2):
Let's try to collect the Cartesian product in temporary table with clustered index: PersonId, Category, [Date], [Hour].
And then update the table dragging non-changed values:
declare #Result table (
[Date] date,
[Hour] tinyint,
PersonId int,
Category varchar(4),
Value decimal(3,1)
primary key (PersonId, Category, [Date], [Hour]) -- Important !!!
)
insert into #Result
select
dh.[Date],
dh.[Hour],
PersonId = p.Id,
c.Category,
l.Value
from
#PersonsIds p cross join #Categories c cross join #DayHours dh
left join #Latest l on l.[Date] = dh.[Date] and l.[Hour] = dh.[Hour] and l.PersonId = p.Id and l.Category = c.Category
order by
[Date], [Hour], PersonId, Category;
declare #PersonId int;
declare #Category varchar(4);
declare #Value decimal(3,1);
update #Result set
#Value = Value = isnull(Value, case when #PersonId = PersonId and #Category = Category then #Value end),
#PersonId = PersonId,
#Category = Category;
For yet better performance consider changing table variables with temporary tables and applying indexes in accordance with query plan recommendations.

Answer 2

If i got it correctly ..it should give you desired result.
select st.Date,
case when hour =1 then NULL
else hour
end as hour
,st.Person,st.Category,
(select status from reading qualify row_number() over (partition by personid
order by status desc)=1)
from structure;

Answer 3

You can achieve this in SQL, but it will be quite slow, because for every person, category, day and hour you will have to look for the latest entry for the person and category until then. Just think of the process: Pick a record in your big table, find all statuses until then, order them and find the latest thus and pick its value. And this will be done for every record in your big table.
You might be better of to simply retrieve all data with a program written in a programming language and collect the data with a control-break algorithm.
However, let's see how it's done in SQL.
One problem is SQL Server's poor date/time functions. We want to compare date plus hour, which would be easiest with strings in 'yyyymmddhh' format, e.g. '2015101923' < '2015102001'. In your big table you have date and hour and in your status table you have datetimes. Let's see how we can get the desired strings:
convert(varchar(8), bigtable.calendarday, 112) +
right('0' + convert(varchar(2), bigtable.hour), 2)
and
convert(varchar(8), status.timestamp, 112) +
right('0' + convert(varchar(2), datepart(hour, status.timestamp)), 2)
As this is - along with person and category - our key criterion to find records, you may want to have it as computed columns and add indexes (person + category + dayhourkey) in both tables.
You'd select from your big table and get the status value in a subquery. In order to get the latest matching record, you'd order by timestamp and limit to 1 record.
select
personid,
calendarday,
hour,
category,
(
select value
from status s
where s.personid = b.personid
and s.category = b.category
and convert(varchar(8), s.timestamp, 112) + right('0' + convert(varchar(2), datepart(hour, s.timestamp)), 2) <=
convert(varchar(8), b.calendarday, 112) + right('0' + convert(varchar(2), b.hour), 2)
order by s.timestamp desc limit 1
) as value
from bigtable b;