Related
I have a merge statement that builds my SCD type 2 table each night. This table must house all historical changes made in the source system and create a new row with the date from/date to columns populated along with the "islatest" flag. I have come across an issue today that I am not really sure how to handle.
There looks to have been multiple changes to the source table within a 24 hour period.
ID Code PAN EnterDate Cost Created
16155 1012401593331 ENRD 2015-11-05 7706.3 2021-08-17 14:34
16155 1012401593331 ENRD 2015-11-05 8584.4 2021-08-17 16:33
I use a basic merge statement to identify my changes however what would be the best approach to ensure all changes get picked up correctly? The above is giving me an error as it's trying to insert/update multiple rows with the same value
DECLARE #DateNow DATETIME = Getdate()
IF Object_id('tempdb..#meteridinsert') IS NOT NULL
DROP TABLE #meteridinsert;
CREATE TABLE #meteridinsert
(
meterid INT,
change VARCHAR(10)
);
MERGE
INTO [DIM].[Meters] AS target
using stg_meters AS source
ON target.[ID] = source.[ID]
AND target.latest=1
WHEN matched THEN
UPDATE
SET target.islatest = 0,
target.todate = #Datenow
WHEN NOT matched BY target THEN
INSERT
(
id,
code,
pan,
enterdate,
cost,
created,
[FromDate] ,
[ToDate] ,
[IsLatest]
)
VALUES
(
source.id,
source.code ,
source.pan ,
source.enterdate ,
source.cost ,
source.created ,
#Datenow ,
NULL ,
1
)
output source.id,
$action
INTO #meteridinsert;INSERT INTO [DIM].[Meters]
(
[id] ,
[code] ,
[pan] ,
[enterdate] ,
[cost] ,
[created] ,
[FromDate] ,
[ToDate] ,
[IsLatest]
)
SELECT ([id] ,[code] ,[pan] ,[enterdate] ,[cost] ,[created] , #DateNow ,NULL ,1 FROM stg_meters a
INNER JOIN #meteridinsert cid
ON a.id = cid.meterid
AND cid.change = 'UPDATE'
Maybe you can do it using merge statement, but I would prefer to use typicall update and insert approach in order to make it easier to understand (also I am not sure that merge allows you to use the same source record for update and insert...)
First of all I create the table dimscd2 to represent your dimension table
create table dimscd2
(naturalkey int, descr varchar(100), startdate datetime, enddate datetime)
And then I insert some records...
insert into dimscd2 values
(1,'A','2019-01-12 00:00:00.000', '2020-01-01 00:00:00.000'),
(1,'B','2020-01-01 00:00:00.000', NULL)
As you can see, the "current" is the one with descr='B' because it has an enddate NULL (I do recommend you to use surrogate keys for each record... This is just an incremental key for each record of your dimension, and the fact table must be linked with this surrogate key in order to reflect the status of the fact in the moment when happened).
Then, I have created some dummy data to represent the source data with the changes for the same natural key
-- new data (src_data)
select 1 as naturalkey,'C' as descr, cast('2020-01-02 00:00:00.000' as datetime) as dt into src_data
union all
select 1 as naturalkey,'D' as descr, cast('2020-01-03 00:00:00.000' as datetime) as dt
After that, I have created a temp table (##tmp) with this query to set the enddate for each record:
-- tmp table
select naturalkey, descr, dt,
lead(dt,1,0) over (partition by naturalkey order by dt) enddate,
row_number() over (partition by naturalkey order by dt) rn
into ##tmp
from src_data
The LEAD function takes the next start date for the same natural key, ordered by date (dt).
The ROW_NUMBER marks with 1 the oldest record in the source data for the natural key in the dimension.
Then, I proceed to close the "current" record using update
update d
set enddate = t.dt
from dimscd2 d
join ##tmp t
on d.naturalkey = t.naturalkey
and d.enddate is null
and t.rn = 1
And finally I add the new source data to the dimension with insert
insert into dimscd2
select naturalkey, descr, dt,
case enddate when '1900-00-00' then null else enddate end
from ##tmp
Final result is obtained with the query:
select * from dimscd2
You can test on this db<>fiddle
For example I have three table where say DataTable1, DataTable2 and DataTable3
and need to filter it from DataRange table, every time I have used NOT exist as shown below,
Is there a better way to write this.
Temp table to hold some daterange which is used for fiter:
Declare #DateRangeTable as Table(
StartDate datetime,
EndDate datetime
)
Some temp table which will hold data on which we need to apply date range filter
INSERT INTO #DateRangeTable values
('07/01/2020','07/04/2020'),
('07/06/2020','07/08/2020');
/*Table 1 which will hold some data*/
Declare #DataTable1 as Table(
Id numeric,
Date datetime
)
INSERT INTO #DataTable1 values
(1,'07/09/2020'),
(2,'07/06/2020');
Declare #DataTable2 as Table(
Id numeric,
Date datetime
)
INSERT INTO #DataTable2 values
(1,'07/10/2020'),
(2,'07/06/2020');
Declare #DataTable3 as Table(
Id numeric,
Date datetime
)
INSERT INTO #DataTable3 values
(1,'07/11/2020'),
(2,'07/06/2020');
Now I want to filter data based on DateRange table, here I need some optimized way so that i don't have to use not exists mutiple times, In real senario, I have mutiple tables where I have to filter based on the daterange table.
Select * from #DataTable1
where NOT EXISTS(
Select 1 from #DateRangeTable
where [Date] between StartDate and EndDate
)
Select * from #DataTable2
where NOT EXISTS(
Select 1 from #DateRangeTable
where [Date] between StartDate and EndDate
)
Select * from #DataTable3
where NOT EXISTS(
Select 1 from #DateRangeTable
where [Date] between StartDate and EndDate
)
Instead of using NOT EXISTS you could join the date range table:
SELECT dt.*
FROM #DataTable1 dt
LEFT JOIN #DateRangeTable dr ON dt.[Date] BETWEEN dr.StartDate and dr.EndDate
WHERE dr.StartDate IS NULL
It may perform better on large tables but you would have to compare the execution plans and make sure you have indexes on the date columns.
I would write the same query... but if you can change table structure I would try to improve performance adding two columns to specify the month as an integer (I suppose is the first couple of figures).
Obviously you have to test with your data and compare the timings.
Declare #DateRangeTable as Table(
StartDate datetime,
EndDate datetime,
StartMonth tinyint,
EndMonth tinyint
)
INSERT INTO #DateRangeTable values
('07/01/2020','07/04/2020', 7, 7),
('07/06/2020','07/08/2020', 7, 7),
('07/25/2020','08/02/2020', 7, 8); // (another record with different months)
Now your queries can use the new column to try to reduce comparisons (is a tinyint, sql server can partition records if you define a secondary index for StartMonth and EndMonth):
Select * from #DataTable1
where NOT EXISTS(
Select 1 from #DateRangeTable
where (DATEPART('month', [Date]) between StartMonth and EndMonth)
and ([Date] between StartDate and EndDate)
)
I'm trying to understand on how to properly execute this query into this table.
The output is supposed to only loop based on count table, but will display all the dates within the parameter of the table row.
I separated the database for APPLE, ORANGE, and MANGO because I need to mimic my database structure. The data should be called from different database, and then process the looping of Date Range and Row Count.
I got the idea from this question, and used some of the codes to try to replicate it :
SQL how to convert row with date range to many rows with each date
This is the expected output.
This is the query that I'm trying to fix
DECLARE #dbApple TABLE
(
FromDate VARCHAR(30) NOT NULL,
ToDate VARCHAR(30) NOT NULL,
Name VARCHAR(30) NOT NULL,
Count VARCHAR(30) NOT NULL
)
INSERT INTO #dbApple (FromDate, ToDate, Name, Count)
VALUES ('2019-10-05', '2019-10-09', 'APPLE', '3');
DECLARE #dbOrange TABLE
(
FromDate VARCHAR(30) NOT NULL,
ToDate VARCHAR(30) NOT NULL,
Name VARCHAR(30) NOT NULL,
Count VARCHAR(30) NOT NULL
)
INSERT INTO #dbOrange (FromDate, ToDate, Name, Count)
VALUES ('2019-10-10', '2019-10-14', 'ORANGE', '2');
DECLARE #dbMango TABLE
(
FromDate VARCHAR(30) NOT NULL,
ToDate VARCHAR(30) NOT NULL,
Name VARCHAR(30) NOT NULL,
Count VARCHAR(30) NOT NULL
)
INSERT INTO #dbMango (FromDate, ToDate, Name, Count)
VALUES ('2019-10-15', '2019-10-19', 'MANGO', '4');
(SELECT
CONVERT(DATE, CONVERT(DATE, DATEADD(D, v.number, FromDate))) AS Date,
DB.Name,
CASE
WHEN ROW_NUMBER() OVER(PARTITION BY Count, FromDate, ToDate ORDER BY Count) = Count
THEN Count
ELSE NULL
END AS Count
FROM
#dbApple DB
JOIN
MASTER..spt_values v ON v.TYPE = 'P'
AND v.number BETWEEN 0 AND DATEDIFF(D, FromDate, ToDate))
UNION
(SELECT
CONVERT(DATE, DATEADD(D, v.number, FromDate)) AS Date,
DB.Name,
CASE
WHEN ROW_NUMBER() OVER(PARTITION BY Count, FromDate, ToDate ORDER BY Count) = Count
THEN Count
ELSE NULL
END AS Count
FROM
#dbOrange DB
JOIN
MASTER..spt_values v ON v.TYPE = 'P'
AND v.number BETWEEN 0 AND DATEDIFF(D, FromDate, ToDate))
UNION
(SELECT
CONVERT(DATE, DATEADD(D, v.number, FromDate)) AS Date,
DB.Name,
CASE
WHEN ROW_NUMBER() OVER(PARTITION BY Count, FromDate, ToDate ORDER BY Count) = Count
THEN Count
ELSE NULL
END AS Count
FROM
#dbMango DB
JOIN
MASTER..spt_values v ON v.TYPE = 'P'
AND v.number BETWEEN 0 AND DATEDIFF(D, FromDate, ToDate))
This is the output:
Tried using CASE WITHIN CASE but no luck.
declare #dbApple TABLE(
FromDate varchar(30) NOT NULL,
ToDate varchar(30) NOT NULL,
Name varchar(30) NOT NULL,
Count varchar(30) NOT NULL
)
INSERT INTO #dbApple
(FromDate,ToDate,Name,Count) VALUES ('2019-10-05','2019-10-09','APPLE','3');
(SELECT
CONVERT(date,CONVERT(date,DATEADD(D,VAL.NUMBER,FromDate))) AS Date,
DB.Name,
CASE WHEN CONVERT(date,CONVERT(date,DATEADD(D,VAL.NUMBER,FromDate))) BETWEEN
CONVERT(date,CONVERT(date,DATEADD(D,VAL.NUMBER,FromDate))) AND
CONVERT(date,CONVERT(date,DATEADD(D,VAL.NUMBER,ToDate)))
THEN CASE WHEN ROW_NUMBER()
OVER(PARTITION BY Count,FromDate,ToDate
ORDER BY Count) = Count
THEN Count
ELSE '1' END
ELSE NULL END AS Count
FROM
#dbApple DB
JOIN MASTER..SPT_VALUES VAL on VAL.TYPE='P'
AND VAL.NUMBER BETWEEN 0 AND DATEDIFF(D, FromDate, ToDate))
Output using CASE WITHIN CASE.
How about solving this recursively?
First put the fruit in a basket, then peel them.
WITH BASKET AS
(
SELECT FromDate, ToDate, Name, Count
FROM #dbApple
UNION ALL
SELECT FromDate, ToDate, Name, Count
FROM #dbOrange
UNION ALL
SELECT FromDate, ToDate, Name, Count
FROM #dbMango
),
PEELED AS
(
SELECT
FromDate as [Date], 1 as Lvl,
FromDate, ToDate, Name, Count
FROM BASKET
UNION ALL
SELECT
DATEADD(day,1,[Date]), Lvl +1,
FromDate, ToDate, Name, Count
FROM PEELED p
WHERE [Date] < ToDate
)
SELECT [Date], [Name],
CASE WHEN Lvl <= Count THEN 1 END AS [Count]
FROM PEELED
ORDER BY [Date];
A test on rextester here
Using a tally, you can achieve this very easily:
CREATE TABLE dbo.YourTable (Fruit varchar(20),
Quantity int);
INSERT INTO dbo.YourTable
VALUES ('Apple',3),
('Orange',2),
('Mango',4);
GO
DECLARE #StartDate date = '20190101',
#EndDate date = '20190110';
--Going to us a tally, incase there can be large date ranges
WITH N AS(
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL),(NULL))N(N)),
Tally AS(
SELECT TOP(DATEDIFF(DAY, #StartDate, #EndDate)+1) ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS I
FROM N N1, N N2, N N3) --1000 rows
SELECT DATEADD(DAY, T.I-1, #StartDate) aS [Date],
YT.Fruit,
CASE WHEN T.I <= YT.Quantity THEN 1 END AS [Count]
FROM dbo.YourTable YT
CROSS JOIN Tally T
ORDER BY Fruit,
[Date];
GO
DROP TABLE dbo.YourTable
I have a table
create table temp1
(
GroupId int,
DateCol Date,
ValueCol int,
)
insert into temp1 values(1,'2016-1-1',10);
insert into temp1 values(1,'2017-1-1',20);
insert into temp1 values(1,'2018-1-1',30);
insert into temp1 values(2,'2016-5-1',101);
insert into temp1 values(2,'2017-5-1',102);
insert into temp1 values(2,'2018-5-1',103);
I want to get a date range (min to next to min) grouped by GroupId column:
ValueCol DateCol DateColNextToMin
111 (min Date) (next to Min Date)
GroupId ValueCol DateCol DateColNextToMin
1 10 2016-1-1 2017-1-1
2 101 2016-5-1 2017-5-1
How can I do this?
I tried to get thorough windows function , pivot etc, but there seems to be no such elegant solution.
try this,
DECLARE #table Table
(
GroupId int,
DateCol Date,
ValueCol int
)
INSERT INTO #table
VALUES
(111, '01-Aug-2016', 0)
,(111, '02-Aug-2016', 0)
,(111, '03-Aug-2016', 0)
,(111, '04-Aug-2016', 0)
,(112, '05-Aug-2016', 0)
,(112, '06-Aug-2016', 0)
,(112, '07-Aug-2016', 0)
,(112, '08-Aug-2016', 0)
;WITH CTE AS
(
SELECT *, ROW_NUMBER() OVER (PARTITION BY GroupId ORDER BY DateCol ASC) AS RowNo
FROM #table
)
SELECT c1.GroupId, c1.ValueCol, c1.DateCol, c2.DateCol
FROM CTE c1
INNER JOIN CTE c2 ON c1.GroupId = c2.GroupId
AND c1.RowNo = 1
AND c2.RowNo = 2
How do I remove duplicates from a table that is set up in the following way?
unique_ID | worker_ID | date | type_ID
A worker can have multiple type_ID's associated with them and I want to remove any duplicate types. If there is a duplicate, I want to remove the type with the most recent entry.
A textbook candidate for the window function row_number():
;WITH x AS (
SELECT unique_ID
,row_number() OVER (PARTITION BY worker_ID,type_ID ORDER BY date) AS rn
FROM tbl
)
DELETE FROM tbl
FROM x
WHERE tbl.unique_ID = x.unique_ID
AND x.rn > 1
This also takes care of the situation where a set of dupes on (worker_ID,type_ID) shares the same date.
See the simplified demo on data.SE.
Update with simpler version
Turns out, this can be simplified: In SQL Server you can delete from the CTE directly:
;WITH x AS (
SELECT unique_ID
,row_number() OVER (PARTITION BY worker_ID,type_ID ORDER BY date) AS rn
FROM tbl
)
DELETE x
WHERE rn > 1
delete from table t
where exists ( select 1 from table t2
where t2.worker_id = t.worker_id
and t2.type_id = t.type_id
and t2.date < t.date )
HTH
DELETE FROM #t WHERE unique_Id IN
(
SELECT unique_Id FROM
(
SELECT unique_Id
,Type_Id
,ROW_NUMBER() OVER (PARTITION BY worker_Id, type_Id ORDER BY date) AS rn
FROM #t
) Q
WHERE rn > 1
)
And to test...
DECLARE #t TABLE
(
unique_ID INT IDENTITY,
worker_ID INT,
date DATETIME,
type_ID INT
)
INSERT INTO #t VALUES (1, DATEADD(DAY, 1, GETDATE()), 1)
INSERT INTO #t VALUES (1, GETDATE(), 1)
INSERT INTO #t VALUES (2, GETDATE(), 1)
INSERT INTO #t VALUES (1, DATEADD(DAY, 2, GETDATE()), 1)
INSERT INTO #t VALUES (1, DATEADD(DAY, 3, GETDATE()), 2)
SELECT * FROM #t
DELETE FROM #t WHERE unique_Id IN
(
SELECT unique_Id FROM
(
SELECT unique_Id
,Type_Id
,ROW_NUMBER() OVER (PARTITION BY worker_Id, type_Id ORDER BY date) AS rn
FROM #t
) Q
WHERE rn > 1
)
SELECT * FROM #t
you may use this query
delete from worker where unique_id in (
select max(unique_id) from worker group by worker_ID , type_ID having count(type_id)>1)
here i am assuming worker as your table name