return a result even if there is no data for that day/ week/ month - sql

I have an MSSQL database and I want to get a value for each day/ week/ month in separate queries.
I got this working just fine except for intervals where there is no data, it wont return anything. And since im putting this in a graph, I want it to display a 0 or a NULL at least instead of jumping days or weeks etc.
I dont know if it will be different for each query but here is my daily query:
select CAST(Placements.CreatedOn AS DATE) AS
date,SUM(Placements.CommissionPerc * (Placements.PlacementFee / 100)) AS value
from [placements]
where [Placements].[CreatedOn] >= '2018-06-07' and [Placements].[CreatedOn] < '2018-06-12'
group by CAST(Placements.CreatedOn AS DATE)
order by CAST(Placements.CreatedOn AS DATE) ASC
This returns a result like:
So it returns 0 for when the data is actually 0 but when its missing, theres nothing like for days 9, 10 and 12
How can i fix this? thanks

Using a recursive CTE you can generate a list of dates.
Which can then be used to LEFT JOIN your table.
Example:
WITH DATES2018 AS
(
SELECT CAST('2018-01-01' AS DATE) AS [date]
UNION ALL
SELECT DATEADD(day, 1, [date])
FROM DATES2018
WHERE [date] < CAST('2018-12-31' AS DATE)
)
SELECT
d.[Date],
SUM(p.CommissionPerc * (p.PlacementFee / 100.0)) AS [value]
FROM DATES2018 AS d
LEFT JOIN [Placements] AS p ON CAST(p.CreatedOn AS DATE) = d.[Date]
WHERE d.[Date] BETWEEN '2018-06-07' AND '2018-06-11'
GROUP BY d.[Date]
ORDER BY d.[Date] ASC
OPTION (MAXRECURSION 366)
But you could also just add a new permanent table with all dates.
And use that table to left join your table.
Btw, if variables are used for the start and end date then that SQL can be optimized.
DECLARE #StartDate DATE = '2018-06-07';
DECLARE #EndDate DATE = '2018-06-11';
WITH DATES AS
(
SELECT #StartDate AS [date]
UNION ALL
SELECT DATEADD(day, 1, [date])
FROM DATES
WHERE [date] < #EndDate
)
SELECT
d.[Date],
SUM(p.CommissionPerc * (p.PlacementFee / 100.0)) AS [value]
FROM DATES AS d
LEFT JOIN [Placements] AS p
ON p.CreatedOn BETWEEN CAST(#StartDate AS DATETIME) AND CAST(DATEADD(day, 1, #EndDate) AS DATETIME) AND
CAST(p.CreatedOn AS DATE) = d.[Date]
GROUP BY d.[Date]
ORDER BY d.[Date] ASC
OPTION (MAXRECURSION 0)

A permanent calendar table would be best but here's and example that uses a CTE to create the dates needed for a LEFT JOIN. This uses a maximum of 1,000 days but can be extended as needed.
DECLARE
#StartDate date = '2018-06-07'
, #EndDate date = '2018-06-12';
WITH
t10 AS (SELECT n FROM (VALUES(0),(0),(0),(0),(0),(0),(0),(0),(0),(0)) t(n))
,t1k AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 0)) - 1 AS num FROM t10 AS a CROSS JOIN t10 AS b CROSS JOIN t10 AS c)
,calendar AS (SELECT DATEADD(day, num, #StartDate) AS calendar_date
FROM t1k
WHERE num <= DATEDIFF(day, #StartDate, #EndDate)
)
SELECT
calendar.calendar_date AS date
, SUM( COALESCE(Placements.CommissionPerc * (Placements.PlacementFee / 100),0 ) ) AS value
FROM calendar
LEFT JOIN [placements] ON [Placements].[CreatedOn] = calendar.calendar_date
GROUP BY calendar.calendar_date
ORDER BY calendar.calendar_date ASC;

Related

Join Generated Date Sequence

Currently I'm trying to join a date table to a ledger table so I can fill the gaps of the ledger table whenever there are no transactions in certain instances (e.g. there are transactions on March 1st and in March 3rd, but no transaction in March 2nd. And by joining both tables March 2nd would appear in the ledger table but with 0 for the variable we're analyzing.)
The challenge is that I can't create a Date object/table/dimension because I don't have permissions to create tables in the database. Therefore I've been generating a date sequence with this code:
DECLARE #startDate date = CAST('2016-01-01' AS date),
#endDate date = CAST(GETDATE() AS date);
SELECT DATEADD(day, number - 1, #startDate) AS [Date]
FROM (
SELECT ROW_NUMBER() OVER (
ORDER BY n.object_id
)
FROM sys.all_objects n
) S(number)
WHERE number <= DATEDIFF(day, #startDate, #endDate) + 1;
So, is there the possibility to join both tables into the same statement? Let's say the ledger table looks like this:
SELECT
date,cost
FROM ledger
I'd assume it can be done by using a subquery but I don't know how.
Thank you.
There is a very good article by Aaron Bertrand showing several methods for generating a sequence of numbers (or dates) in SQL Server: Generate a set or sequence without loops – part 1.
Try them out and see for yourself which is faster or more convenient to you. (spoiler - Recursive CTE is rather slow)
Once you've picked your preferred method you can wrap it in a CTE (common-table expression).
Here I'll use your method from the question
WITH
CTE_Dates
AS
(
SELECT
DATEADD(day, number - 1, #startDate) AS dt
FROM (
SELECT ROW_NUMBER() OVER (
ORDER BY n.object_id
)
FROM sys.all_objects n
) S(number)
WHERE number <= DATEDIFF(day, #startDate, #endDate) + 1
)
SELECT
...
FROM
CTE_Dates
LEFT JOIN Ledger ON Ledger.dt = CTE_Dates.dt
;
You can use your generated date sequence as a CTE and LEFT JOIN that to your ledger table. For example:
DECLARE #startDate date = CAST('2020-02-01' AS date);
DECLARE #endDate date = CAST(GETDATE() AS date);
WITH dates AS (
SELECT DATEADD(day, number - 1, #startDate) AS [Date]
FROM (
SELECT ROW_NUMBER() OVER (
ORDER BY n.object_id
)
FROM sys.all_objects n
) S(number)
WHERE number <= DATEDIFF(day, #startDate, #endDate) + 1
)
SELECT dates.Date, COALESCE(ledger.cost, 0)
FROM dates
LEFT JOIN (VALUES ('2020-02-02', 14), ('2020-02-05', 10)) AS ledger([Date], [cost]) ON dates.Date = ledger.Date
Output:
Date cost
2020-02-01 0
2020-02-02 14
2020-02-03 0
2020-02-04 0
2020-02-05 10
2020-02-06 0
Demo on dbfiddle

How to loop through dates, 30 times to find historical information at points in time?

I am trying to find out the transaction details of customers at different points in time in the past. I came up with a query but I have to change the date every single time in my declare statement. Is there a way to loop through the dates to get data back x amount of days in the past?
DECLARE #StartDate AS Date
SET #StartDate = '2018-10-30'
SELECT u.member_id AS Member_id
,CAST(MAX(pe.executed_time) AS Date) AS Max_Date
,DATEDIFF(dd,#StartDate,MAX(pe.executed_time))*-1+1 AS R
,COUNT(*) AS F
,SUM(pi.Price) AS M
FROM [user] AS u
LEFT JOIN purchase_entry AS pe
ON u.member_id=pe.member_id
LEFT JOIN purchase_item AS pi
ON pe.order_id=pi.order_id
WHERE 1=1
AND pe.[status] = 'purchase'
AND pe.executed_time <= #StartDate -- Change the declared Date to have historical information.
GROUP BY u.id, u.member_id
ORDER BY Recency
I need a way to have my query to loop through the #StartDate for 30 days in the past for instance. For ex: #StartDate = '2018-11-30' then #StartDate '2018-11-29' , and so on ...
You can list the dates that you want:
WITH dates as (
SELECT v.*
FROM (VALUES (CONVERT(DATE, '2018-11-30')), (CONVERT(DATE, '2018-11-29'))
) v(dte)
)
SELECT dates.dte, u.member_id, . . .
FROM dates CROSS JOIN
[user] u JOIN
purchase_entry pe
ON u.member_id = pe.member_id LEFT JOIN
purchase_item pi
ON pe.order_id = pi.order_id
WHERE pe.[status] = 'purchase' AND
pe.executed_time <= date.dte -- Change the declared Date to have historical information.
GROUP BY v.dte, u.id, u.member_id
ORDER BY Recency;
For a specific period of dates, you can construct them using a recursive CTE:
with dates as (
select convert(date, '2018-11-30') as dte, 1 as n
union all
select dateadd(day, -1, date.dte), n + 1
from dates
where n < 30
)
. . .
Use a tally table to generate the date and then query:
DECLARE #StartDate AS Date
SET #StartDate = '2018-10-30'
WITH N AS (
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL)) N(N)),
Tally AS(
SELECT TOP 30 ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) -1 AS I
FROM N N1
CROSS JOIN N N2
CROSS JOIN N N3),
Dates AS(
SELECT DATEADD(DAY, I, #StartDate) AS [Date]
FROM Tally
)
SELECT u.member_id AS Member_id
,CAST(MAX(pe.executed_time) AS Date) AS Max_Date
,DATEDIFF(dd,#StartDate,MAX(pe.executed_time))*-1+1 AS R
,COUNT(*) AS F
,SUM(pi.Price) AS M
FROM [user] AS u
CROSS JOIN Dates D
LEFT JOIN purchase_entry AS pe
ON u.member_id=pe.member_id
LEFT JOIN purchase_item AS pi
ON pe.order_id=pi.order_id
WHERE pe.[status] = 'purchase'
AND pe.executed_time <= D.[Date]
ORDER BY Recency;

Find missing date as compare to calendar

I am explain problem in short.
select distinct DATE from #Table where DATE >='2016-01-01'
Output :
Date
2016-11-23
2016-11-22
2016-11-21
2016-11-19
2016-11-18
Now i need to find out missing date a compare to our calender dates from year '2016'
i.e. Here date '2016-11-20' is missing.
I want list of missing dates.
Thanks for reading this. Have nice day.
You need to generate dates and you have to find missing ones. Below with recursive cte i have done it
;WITH CTE AS
(
SELECT CONVERT(DATE,'2016-01-01') AS DATE1
UNION ALL
SELECT DATEADD(DD,1,DATE1) FROM CTE WHERE DATE1<'2016-12-31'
)
SELECT DATE1 MISSING_ONE FROM CTE
EXCEPT
SELECT * FROM #TABLE1
option(maxrecursion 0)
Using CTE and get all dates in CTE table then compare with your table.
CREATE TABLE #yourTable(_Values DATE)
INSERT INTO #yourTable(_Values)
SELECT '2016-11-23' UNION ALL
SELECT '2016-11-22' UNION ALL
SELECT '2016-11-21' UNION ALL
SELECT '2016-11-19' UNION ALL
SELECT '2016-11-18'
DECLARE #DATE DATE = '2016-11-01'
;WITH CTEYear (_Date) AS
(
SELECT #DATE
UNION ALL
SELECT DATEADD(DAY,1,_Date)
FROM CTEYear
WHERE _Date < EOMONTH(#DATE,0)
)
SELECT * FROM CTEYear
WHERE NOT EXISTS(SELECT 1 FROM #yourTable WHERE _Date = _Values)
OPTION(maxrecursion 0)
You need to generate the dates and then find the missing ones. A recursive CTE is one way to generate a handful of dates. Another way is to use master..spt_values as a list of numbers:
with n as (
select row_number() over (order by (select null)) - 1 as n
from master..spt_values
),
d as (
select dateadd(day, n.n, cast('2016-01-01' as date)) as dte
from n
where n <= 365
)
select d.date
from d left join
#table t
on d.dte = t.date
where t.date is null;
If you are happy enough with ranges of missing dates, you don't need a list of dates at all:
select date, (datediff(day, date, next_date) - 1) as num_missing
from (select t.*, lead(t.date) over (order by t.date) as next_date
from #table t
where t.date >= '2016-01-01'
) t
where next_date <> dateadd(day, 1, date);

SQL calculate date segments within calendar year

What I need is to calculate the missing time periods within the calendar year given a table such as this in SQL:
DatesTable
|ID|DateStart |DateEnd |
1 NULL NULL
2 2015-1-1 2015-12-31
3 2015-3-1 2015-12-31
4 2015-1-1 2015-9-30
5 2015-1-1 2015-3-31
5 2015-6-1 2015-12-31
6 2015-3-1 2015-6-30
6 2015-7-1 2015-10-31
Expected return would be:
1 2015-1-1 2015-12-31
3 2015-1-1 2015-2-28
4 2015-10-1 2015-12-31
5 2015-4-1 2015-5-31
6 2015-1-1 2015-2-28
6 2015-11-1 2015-12-31
It's essentially work blocks. What I need to show is the part of the calendar year which was NOT worked. So for ID = 3, he worked from 3/1 through the rest of the year. But he did not work from 1/1 till 2/28. That's what I'm looking for.
You can do it using LEAD, LAG window functions available from SQL Server 2012+:
;WITH CTE AS (
SELECT ID,
LAG(DateEnd) OVER (PARTITION BY ID ORDER BY DateEnd) AS PrevEnd,
DateStart,
DateEnd,
LEAD(DateStart) OVER (PARTITION BY ID ORDER BY DateEnd) AS NextStart
FROM DatesTable
)
SELECT ID, DateStart, DateEnd
FROM (
-- Get interval right before current [DateStart, DateEnd] interval
SELECT ID,
CASE
WHEN DateStart IS NULL THEN '20150101'
WHEN DateStart > start THEN start
ELSE NULL
END AS DateStart,
CASE
WHEN DateStart IS NULL THEN '20151231'
WHEN DateStart > start THEN DATEADD(d, -1, DateStart)
ELSE NULL
END AS DateEnd
FROM CTE
CROSS APPLY (SELECT COALESCE(DATEADD(d, 1, PrevEnd), '20150101')) x(start)
-- If there is no next interval then get interval right after current
-- [DateStart, DateEnd] interval (up-to end of year)
UNION ALL
SELECT ID, DATEADD(d, 1, DateEnd) AS DateStart, '20151231' AS DateEnd
FROM CTE
WHERE DateStart IS NOT NULl -- Do not re-examine [Null, Null] interval
AND NextStart IS NULL -- There is no next [DateStart, DateEnd] interval
AND DateEnd < '20151231' -- Current [DateStart, DateEnd] interval
-- does not terminate on 31/12/2015
) AS t
WHERE t.DateStart IS NOT NULL
ORDER BY ID, DateStart
The idea behind the above query is simple: for every [DateStart, DateEnd] interval get 'not worked' interval right before it. If there is no interval following the current interval, then also get successive 'not worked' interval (if any).
Also note that I assume that if DateStart is NULL then DateStart is also NULL for the same ID.
Demo here
If your data is not too big, this approach will work. It expands all the days and ids and then re-groups them:
with d as (
select cast('2015-01-01' as date)
union all
select dateadd(day, 1, d)
from d
where d < cast('2015-12-31' as date)
),
td as (
select *
from d cross join
(select distinct id from t) t
where not exists (select 1
from t t2
where d.d between t2.startdate and t2.enddate
)
)
select id, min(d) as startdate, max(d) as enddate
from (select td.*,
dateadd(day, - row_number() over (partition by id order by d), d) as grp
from td
) td
group by id, grp
order by id, grp;
An alternative method relies on cumulative sums and similar functionality that is much easier to expression in SQL Server 2012+.
Somewhat simpler approach I think.
Basically create a list of dates for all work block ranges (A). Then create a list of dates for the whole year for each ID (B). Then remove the A from B. Compile the remaining list of dates into date ranges for each ID.
DECLARE #startdate DATETIME, #enddate DATETIME
SET #startdate = '2015-01-01'
SET #enddate = '2015-12-31'
--Build date ranges from remaining date list
;WITH dateRange(ID, dates, Grouping)
AS
(
SELECT dt1.id, dt1.Dates, dt1.Dates + row_number() over (order by dt1.id asc, dt1.Dates desc) AS Grouping
FROM
(
--Remove (A) from (B)
SELECT distinct dt.ID, tmp.Dates FROM DatesTable dt
CROSS APPLY
(
--GET (B) here
SELECT DATEADD(DAY, number, #startdate) [Dates]
FROM master..spt_values
WHERE type = 'P' AND DATEADD(DAY, number, #startdate) <= #enddate
) tmp
left join
(
--GET (A) here
SELECT DISTINCT T.Id,
D.Dates
FROM DatesTable AS T
INNER JOIN master..spt_values as N on N.number between 0 and datediff(day, T.DateStart, T.DateEnd)
CROSS APPLY (select dateadd(day, N.number, T.DateStart)) as D(Dates)
WHERE N.type ='P'
) dr
ON dr.Id = dt.Id and dr.Dates = tmp.Dates
WHERE dr.id is null
) dt1
)
SELECT ID, CAST(MIN(Dates) AS DATE) DateStart, CAST(MAX(Dates) AS DATE) DateEnd
FROM dateRange
GROUP BY ID, Grouping
ORDER BY ID
Heres the code:
http://sqlfiddle.com/#!3/f3615/1
I hope this helps!

How to get latest record weekly within an SQL statement

I'm trying to export the latest records from SQL Server 2005 database once weekly. This is my table:
agent_name date ID
ALEX 2015-05-25 13
ALEX 2015-05-22 13
ALICE 2015-05-24 10
ALICE 2015-05-26 10
How to create output table should like this:
agent_name date ID
ALEX 2015-05-25 13
ALICE 2015-05-26 10
My SQL script:
SELECT a.agent_name,
a.date,
a.ID
FROM Payment a
INNER JOIN agentmaster b ON a.ID = b.ID2
WHERE b.agent ='Y'
AND a.date >= DATEADD(day, -7, GETDATE())
If you want latest rows for each ID and agent_name use this:
SELECT a.agent_name,
MAX(a.[date]) [date],
a.ID
FROM Payment a
GROUP BY
a.agent_name,
a.ID
If you want to have latest rows in each week use this:
SELECT a.agent_name,
a.date,
a.ID
FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY DATEPART(WEEK, [date]) ORDER BY [date] DESC) seq
FROM Payment ) a
WHERE
seq = 1
If you want latest rows for each ID and agent_name at each week use this:
;WITH p AS (
/* add your query here */
)
SELECT a.agent_name,
a.date,
a.ID
FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY agent_name, ID, DATEPART(WEEK, [date]) ORDER BY [date] DESC) seq
FROM p ) a
WHERE
seq = 1
Try This
SELECT ID, Agent_Name, MAX(Date) Max_Date
FROM AgentMaster
GROUP BY Agent_Name, ID
If you want to add the date filter then add the where condition
date >= DATEADD(day,-7, GETDATE())
This will return records from the first day of the current week to the last day of the current week
SELECT a.agent_name
,a.date
,a.ID
FROM Payment a INNER JOIN agentmaster b
ON a.ID = b.ID2
WHERE b.agent ='Y' AND
(a.date BETWEEN
DATEADD(wk, DATEDIFF(wk, 0, GetDate()),0) AND
DATEADD(wk, DATEDIFF(wk, 0, GETDATE()),6))
In this solution, I make a table in a CTE and defines the first day of week within a period. And then make a JOIN between the first day of weeks and our actual query
Sample data
DECLARE #tbl TABLE(agent_name VARCHAR(10) ,[date] date, Id INT)
INSERT #tbl
SELECT 'ALEX' , '2015-05-25' , 13 UNION
SELECT 'ALEX' , '2015-05-22' , 13 UNION
SELECT 'ALICE' , '2015-05-24' , 10 UNION
SELECT 'ALICE' , '2015-05-26' , 10
SELECT * FROM #tbl
Query
-- This period can be changed
DECLARE #StartDate date = '01/01/2015'
DECLARE #EndDate date = '01/01/2016'
;WITH Numbers (Number)
AS
(
SELECT number
FROM master..spt_values
WHERE [type] = 'P'
)
,firstDayOfWeek (fWDate)
AS
(
SELECT DATEADD(DAY, n.Number, #StartDate)
FROM Numbers AS n
WHERE n.Number <= DATEDIFF(DAY, #StartDate, #EndDate)
AND DATEPART(WEEKDAY, DATEADD(DAY, n.Number, #StartDate)) = 1 --Defines first day of week
)
SELECT t.agent_name, Max(t.date) AS [date], t.Id
FROM #tbl AS t
INNER JOIN firstDayOfWeek AS fw ON t.date >= fw.fWDate
AND t.[date] < DATEADD(DAY, 7, fw.fWDate)
GROUP BY t.Id, t.agent_name