(SQL Merge) I am getting duplicates in the table - sql

Question

We have a daily stream where we are getting the list of customers using various products.
I am trying to create a table for the customers where we can track their changes, and at the same time, we can get a distinct list of customers.
The stream contains thousands of records each day. That was the reason we thought we should move from SCD Type 1 to SCD Type 2.
We want to implement this procedure so that it will run each day and get the records from the last day and compare them to the whole table. If the customer has any change, it will mark that row as 0 and get the new row and mark it as 1.
But in this process, I am getting the new records, but I am also getting duplicate data when I am running the stored procedure.
Please guide.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
Create PROC [dbo].[sp_UpdateCustomerInfoHistory] AS BEGIN
SET
NOCOUNT ON --Truncate Table [dbo].[CustomerInfoHistory];
DECLARE #TODAY DATE = GETDATE();
DECLARE #YESTERDAY DATE = GETDATE() - 1;
WITH CTE AS (
SELECT
DISTINCT(a.CustomerId) AS CustomerId,
ISNULL(b.[CustomerName], a.[CustomerName]) AS CustomerName,
ISNULL(b.[CurrentDefaultDomain], a.[CustomerName]) AS CurrentDefaultDomain,
ISNULL(b.[CustomerCountryCode], 'Unknown') AS CustomerCountryCode,
ISNULL(b.[HasC], 0) AS HasC,
ISNULL(b.[HasG], 0) AS HasG,
ISNULL(b.[IsV], 0) AS IsV,
ISNULL(
ISNULL(b.[CustomerCreatedDate], a.[ProductCreatedTimeUtc]),
#TODAY
) AS CustomerCreatedDate,
ISNULL(b.[CustomerState], 'Active') AS CustomerState,
ISNULL(b.[CustomerType], 'RegularCustomer') AS CustomerType,
ISNULL(b.[DataCenterProduct], 'Unknown') AS DataCenterProduct,
ISNULL(b.[DataCenterModel], 'Unknown') AS DataCenterModel,
ISNULL(b.[IsTestCustomer], 0) AS IsTestCustomer,
ISNULL(b.[CommunicationLanguage], 'Unknown') AS CommunicationLanguage,
ISNULL(b.[IsInternal], 0) AS IsInternal,
ISNULL(b.[IndustryName], 'N/A') AS IndustryName,
ISNULL(c.MappingID, 0) AS MappingID
FROM
[dbo].[ProductDetails] AS a
LEFT JOIN [Common].[vwdimCustomer_Staging] AS b ON a.CustomerId = b.CustomerId
LEFT JOIN [Common].[vwmapCustomerMappingID_Staging] AS c ON b.CustomerId = c.CustomerId
WHERE a.[TIMESTAMP] = #YESTERDAY
), CTE1 AS (
Select *, BINARY_CHECKSUM(
CustomerId,
CustomerName,
IsTestCustomer,
IsInternal
) AS MKEY
from CTE)
MERGE INTO [dbo].[CustomerInfoHistory] AS T USING CTE1 AS S ON T.[MKEY] = S.[MKEY]
WHEN MATCHED
AND [Current_Flag] = 1
AND T.CustomerName <> S.CustomerName THEN
UPDATE
SET
T.Current_Flag = 0,
T.End_date = #YESTERDAY
WHEN NOT MATCHED BY TARGET THEN
INSERT
(
CustomerId,
CustomerName,
CurrentDefaultDomain,
CustomerCountryCode,
HasC,
HasG,
IsV,
CustomerCreatedDate,
CustomerState,
CustomerType,
DataCenterProduct,
DataCenterModel,
IsTestCustomer,
CommunicationLanguage,
IsInternal,
IndustryName,
MappingID,
Eff_Date,
End_Date,
Current_Flag,
MKEY,
RefreshedDate
)
VALUES
(
S.CustomerId,
S.CustomerName,
S.CurrentDefaultDomain,
S.CustomerCountryCode,
S.HasC,
S.HasG,
S.IsV,
S.CustomerCreatedDate,
S.CustomerState,
S.CustomerType,
S.DataCenterProduct,
S.DataCenterModel,
S.IsTestCustomer,
S.CommunicationLanguage,
S.IsInternal,
S.IndustryName,
S.MappingID,
#YESTERDAY,
'12/31/2099',
1,
S.MKEY,
#TODAY
);
END

Answer 1

I think you can use MERGE in Azure Synapse. It will insert new rows or update old rows based on the primary key value.
For example:
Create table:
CREATE TABLE dbo.CustomerInfoHistory (
CustomerId int NOT NULL,
CustomerName nvarchar(254) NOT NULL,
CurrentDefaultDomain nvarchar(max) NULL
);
GO
ALTER TABLE dbo.CustomerInfoHistory ADD CONSTRAINT PK__kruserpr__6E092EE804688C07 PRIMARY KEY (CustomerId, CustomerName);
GO
Create a Table-valued parameter named dbo.CustomerInfoHistory_type, it will be used in my stored procedure:
create TYPE dbo.CustomerInfoHistory_type AS TABLE(
CustomerId int NOT NULL,
CustomerName nvarchar(254) NOT NULL,
CurrentDefaultDomain nvarchar(max)
)
GO
Create a Stored procedure, it will merge the same records and insert new records based on the primary key:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
create PROCEDURE [dbo].[spUpsertCustomerInfoHistory]
#profile dbo.CustomerInfoHistory_type READONLY
AS
BEGIN
MERGE dbo.CustomerInfoHistory AS target_sqldb
USING #profile AS source_tblstg
ON (target_sqldb.CustomerId = source_tblstg.CustomerId and target_sqldb.CustomerName = source_tblstg.CustomerName )
WHEN MATCHED THEN
UPDATE SET
CurrentDefaultDomain = source_tblstg.CurrentDefaultDomain
WHEN NOT MATCHED THEN
INSERT (
CustomerId,
CustomerName,
CurrentDefaultDomain
)
VALUES (
source_tblstg.CustomerId,
source_tblstg.CustomerName,
source_tblstg.CurrentDefaultDomain
);
END
GO
After that, we can execute the stored procedure by following code:
DECLARE #profileVar AS dbo.CustomerInfoHistory_type;
/* Add data to the table variable. */
INSERT INTO #profileVar (CustomerId, CustomerName, CurrentDefaultDomain) values (1, 'tom','wednesday');
exec [dbo].[spUpsertCustomerInfoHistory] #profileVar
That's all.