find max logons in an interval - sql

Question

I have a table with timestamps and states for people.
|:--------------------------------------------------------------:|
| user_id | state | start_time | end_time |
|:--------------------------------------------------------------:|
| 4711 | 1 | 2013-10-30 09:01:23 | 2013-10-30 17:12:03 |
| 4712 | 1 | 2013-10-30 07:01:23 | 2013-10-30 18:12:03 |
| 4713 | 1 | 2013-10-30 08:01:23 | 2013-10-30 16:12:03 |
| 4714 | 1 | 2013-10-30 09:01:24 | 2013-10-30 17:02:03 |
My challenge is, to find out how many users are
MAX(logged on) AND AVG(logged on) in same time per Interval. I think that I get out when I can see, how many users are simultaneously logged in per second.
|:-------------------------------------:|
| timestamp | state | userid |
|:-------------------------------------:|
| 1383123683 | 1 | 4711 |
| 1383123684 | 1 | 4711 |
| 1383123684 | 1 | 4712 |
| 1383123685 | 1 | 4711 |
| 1383123685 | 1 | 4712 |
| ... | ... | ... |
By the way, one intervals is a quarter of an hour.
The Data comes via INSERT INTO so my idea was to crate a trigger and wrote in a helper table one row for each second (UNIX timestamp) between start and end adding the state_id.
At the end, it must be possible to group over the seconds and count over the datasets to find out, how many rows are exist in one second. For the AVG I have not yet a formula :-). It's a question of time, you know.
But I'm not sure, if my idea was a good one, because i fear that my plan needs a lot of performance and space.
The better idea will be, to wrote just the start-time and end-time, but i loosing the possibility to grouping over the seconds.
How I can manage that without thousands of rows in my database?

Answer 1

Here can be several solutions, I want to describe one, and I hope you can use/adapt/extent it for your particular needs (NOTE: I'm using mysql dialect, for ms sql it can be a little bit different syntax, but approach will work):
1 create new table, with structure like:
create table changelog (
changetime datetime,
changevalue int,
totalsum int,
primary key (changetime)
);
2 insert basic data:
insert into changelog
select changet, sum(cnts), 0
from
(
select start_time as changet, 1 as cnts from testlog
union all
select end_time as changet, -1 from testlog
) as q
group by changet;
3 update totalsum colum:
update changelog as a set totalsum = ifnull((select sum(changevalue) from (select changet, sum(cnts) as changevalue, 0
from
(
select start_time as changet, 1 as cnts from testlog
union all
select end_time as changet, -1 from testlog
) as q
group by changet) as b where b.changet<=a.changetime),0);
NOTE: for ms sql you can try with syntax, you will be able to do these insert/update as one query
4 after this you will have (based on data from question):
2013-10-30 07:01:23 1 1
2013-10-30 08:01:23 1 2
2013-10-30 09:01:23 1 3
2013-10-30 09:01:24 1 4
2013-10-30 16:12:03 -1 3
2013-10-30 17:02:03 -1 2
2013-10-30 17:12:03 -1 1
2013-10-30 18:12:03 -1 0
As you see, max logged in already here, but here is one problem, imagine that you need to select data for range: 08:00-08:01, there are no data in table, so query like this will not work:
SELECT max(totalsum)
FROM changelog
where changetime between cast(#startrange as datetime) and cast(#endrange as datetime)
but you can change it to:
SELECT max(totalsum)
from
(
select max(totalsum) as totalsum FROM changelog
where changetime between cast(#startrange as datetime) and cast(#endrange as datetime)
union all
select totalsum from changelog where changetime=(select max(changetime) from changelog where changetime<cast(#startrange as datetime))
) as q;
so, basically saying - in addition to your range you need to fetch last row before period starts, to find out how many users was at the moment of range start
5 now, you want to calculate average. Average is kinda tricky function, depending on what you understand - there can be different results, average user's per second or average workload
here is the difference:
100 users logged in at 09:00
98 users logged out at 09:01
1 user logged out at 09:02
Selection range: 09:00 - 09:59 (inclusive)
average per minute will be sum of all logged in users in each minute and divided by 60
(100 + 2 + 1 + 57*1)/60 = 2.6(6) user per minute
but average workload can be calculated as (max(logged_users)+min(logged_users)) / 2
(100 + 1)/2 = 50.5 users, this is average simultaneous users logged in system
another average can be calculated via SQL avg (sum(values)/count(values)), which will give us
(100+98+1)/3 = 66.3(3) - another average workload in persons
first formula says to us that it only 2.65 user at the same time, but second shows "holy #*&####, it is 50.5 users at the same time"
another example:
100 users logged in at 09:00
99 users logged out at 09:58
1 user logged out at 09:59
Selection range: 09:00 - 09:59 (inclusive)
first formula will give you (100*58 + 2 + 1)/60 = 96.71(6) users, second will continue to give 50.5, 3rd one still 66.3(3)
what average suits best for you?
To calculate 1st average you need to create stored procedure which will get data for each minute/seconds of period and summarize it, after divide
To calculate 2nd variant: just select min/max and divide by 2
3rd variant: use avg instead of max
Note #1: of course all this approaches are quite slow with huge traffic, so I suggest you to prepare some "pre-calculated" tables with data which can be fetched fast (for example you can have data for each hour like: YYYY-MM-DD HH loggedInatStart, min, avg, median, max, loggedInatEnd)
Note #2: sometimes median average is more interesting for statistical purposes, to obtain it you will: for each minute calculate how many users was logged in, select distinct values, select middle from this list (for my examples this will give us 2 and 2), or select all values, select middle one (for my example it will give us 1 and 99)