Counting New Unique Values in Growing Time Window - sql

Question

I have a large table of users (as a guid), some associated values, and a time stamp of when each row was inserted. A user might be associated with many rows in this table.
guid | <other columns> | insertdate
I want to count for each month: how many unique new users were inserted. It's easy to do manually:
select count(distinct guid)
from table
where insertdate >= '20060201' and insertdate < '20060301'
and guid not in (select guid from table where
insertdate >= '20060101' and insertdate < '20060201')
How could this be done for each successive month in sql?
I thought to use a rank function to associate clearly each guid with a month:
select guid,
,dense_rank() over ( order by datepart(YYYY, insertdate),
datepart(m, t.TransactionDateTime)) as MonthRank
from table
and then iterate upon each rank value:
declare #no_times int
declare #counter int = 1
set #no_times = select count(distinct concat(datepart(year, t.TransactionDateTime),
datepart(month, t.TransactionDateTime))) from table
while #no_times > 0 do
(
select count(*), #counter
where guid not in (select guid from table where rank = #counter)
and rank = #int + 1
#counter += 1
#no_times -= 1
union all
)
end
I know this strategy is probably the wrong way to go about things.
Ideally, I would like a result set to look like this:
MonthRank | NoNewUsers
I would be extremely interested and grateful if a sql wizard could point me in the right direction.

Answer 1

SELECT
DATEPART(year,t.insertdate) AS YearNum
,DATEPART(mm,t.insertdate) as MonthNum
,COUNT(DISTINCT guid) AS NoNewUsers
,DENSE_RANK() OVER (ORDER BY COUNT(DISTINCT t.guid) DESC) AS MonthRank
FROM
table t
LEFT JOIN table t2
ON t.guid = t2.guid
AND t.insertdate > t2.insertdate
WHERE
t2.guid IS NULL
GROUP BY
DATEPART(year,t.insertdate)
,DATEPART(mm,t.insertdate)
Use a left join to see if the table ever existed as a prior insert date and if they didn't then count them using aggregation like you normally would. If you want to add a rank to see which month(s) have the highest number of new users then you can use your DENSE_RANK() function but because you are already grouping by want you want you do not need a partition clause.

Answer 2

If you want the first time that a guid entered, then your query doesn't exactly work. You can get the first time with two aggregations:
select year(first_insertdate), month(first_insertdate), count(*)
from (select t.guid, min(insertdate) as first_insertdate
from t
group by t.guid
) t
group by year(first_insertdate), month(first_insertdate)
order by year(first_insertdate), month(first_insertdate);
If you are looking for counting guids each time they skip a month, then you can use lag():
select year(insertdate), month(insertdate), count(*)
from (select t.*,
lag(insertdate) over (partition by guid order by insertdate) as prev_insertdate
from t
) t
where prev_insertdate is null or
datediff(month, prev_insertdate, insertdate) >= 2
group by year(insertdate), month(insertdate)
order by year(insertdate), month(insertdate);

Answer 3

I solved it with the terrible while loop, then a friend helped me to solve it more efficiently in another way.
The loop version:
--ranked by month
select t.TransactionID
,t.BuyerUserID
,concat(datepart(year, t.InsertDate), datepart(month,
t.InsertDate)) MonthRankName
,dense_rank() over ( order by datepart(YYYY, t.InsertDate),
datepart(m, t.InsertDate)) as MonthRank
into #ranked
from table t;
--iteratate
declare #counter int = 1
declare #no_times int
select #no_times = count(distinct concat(datepart(year, t.InsertDate),
datepart(month, t.InsertDate))) from table t;
select count(distinct r.guid) as NewUnique, r.Monthrank into #results
from #ranked r
where r.MonthRank = 1 group by r.MonthRank;
while #no_times > 1
begin
insert into #results
select count(distinct rt.guid) as NewUnique, #counter + 1 as MonthRank
from #ranked r
where rt.guid not in
(
select rt2.guid from #ranked rt2
where rt2.MonthRank = #counter
)
and rt.MonthRank = #counter + 1
set #counter = #counter+1
set #no_times = #no_times-1
end
select * from #results r
This turned out to run pretty slowly (as you might expect)
What turned out to be faster by a factor of 10 was this method:
select t.guid,
cast (concat(datepart(year, min(t.InsertDate)),
case when datepart(month, min(t.InsertDate)) < 10 then
'0'+cast( datepart(month, min(t.InsertDate)) as varchar(10))
else cast (datepart(month, min(t.InsertDate)) as varchar(10)) end
) as int) as MonthRankName
into #NewUnique
from table t
group by t.guid;
select count(1) as NewUniques, t.MonthRankName from #NewUnique t
group by t.MonthRankName
order by t.MonthRankName
Simply identifying the very first month each guid appears, then counting the number of these occurring each month. With a bit of a hack to get YearMonth formatted nicely (this seems to be more efficient than format([date], 'yyyyMM') but need to experiment more on that.