Automatic Deletion of Duplicate SQL Records based on date or count - sql

Question

After importing Excel data from a LightSwitch application into a holding table in SQL Server I end up with duplicate records. I need a way to remove the duplicates that can either be executed from LightSwitch or something that will automatically run in SQL after/during insert. I thought about a trigger, but I'm not sure it's the best solution.
The duplicates will be something like this
DocName|DocUser|DocType|DocDate|
test user1 word 10/12/2012
test user1 word 10/12/2012
test2 user2 word 10/11/2012
test2 user2 word 10/12/2012
In the case of the first set of duplicates either record can be deleted so I have one record.
However in the second case the record with the date of 10/11/2012 would need to be deleted.
I'm not apposed to a Stored Procedure if it can be executed by from LightSwitch. I know it can be done with a series of queries but I'm not sure how that could be executed from LightSwitch?

Answer 1

I have no experience with Lightswitch so apologies if any of this is not relevant, but speaking from the SQL side a stored procedure you could use to delete the duplicates is:
CREATE PROCEDURE dbo.DeleteDuplicatesFromT
AS
BEGIN
WITH CTE AS
( SELECT DocName,
DocUser,
DocType,
DocDate,
[RowNumber] = ROW_NUMBER() OVER(PARTITION BY DocName, DocUser, DocType ORDER BY DocDate DESC)
FROM T
)
DELETE CTE
WHERE RowNumber > 1;
END
Example on SQLFiddle
HOWEVER
I'd advise managing this before/during the insert stage, you can either do this in the application code, and ensure only unique records are passed to the table to begin with, or use a procedure again to perform the insert. To do the latter you will first need to create a TYPE to handle your new records:
CREATE TYPE dbo.TableTypeParameter AS TABLE
( DocName VARCHAR(5),
DocUser VARCHAR(5),
DocType VARCHAR(4),
DocDate DATETIME
);
You can fill this in your client code (using System.Data.DataTable) and pass this as a parameter to your stored procedure:
CREATE PROCEDURE dbo.BulkInsert #NewRecords dbo.TableTypeParameter READONLY
AS
BEGIN
WITH NewRecords AS
( SELECT DocName, DocType, DocUser, DocDate = MAX(DocDate)
FROM #NewRecords
GROUP BY DocName, DocType, DocUser
)
MERGE INTO T
USING NewRecords nr
ON T.DocName = nr.DocName
AND T.DocType = nr.DocType
AND T.DocUser = nr.DocUser
WHEN MATCHED AND nr.DocDate > T.DocDate THEN UPDATE
SET DocDate = nr.DocDate
WHEN NOT MATCHED THEN INSERT (DocName, DocUser, DocType, DocDate)
VALUES (nr.DocName, nr.DocUser, nr.DocType, nr.DocDate);
END;
EDIT
The procedure to insert can fairly easily be turned into a trigger if this is what is required:
CREATE TRIGGER dbo.T_InsteadOfInsert
ON T
INSTEAD OF INSERT
AS
BEGIN
WITH NewRecords AS
( SELECT DocName, DocType, DocUser, DocDate = MAX(DocDate)
FROM inserted
GROUP BY DocName, DocType, DocUser
)
MERGE INTO T
USING NewRecords nr
ON T.DocName = nr.DocName
AND T.DocType = nr.DocType
AND T.DocUser = nr.DocUser
WHEN MATCHED AND nr.DocDate > T.DocDate THEN UPDATE
SET DocDate = nr.DocDate
WHEN NOT MATCHED THEN INSERT (DocName, DocUser, DocType, DocDate)
VALUES (nr.DocName, nr.DocUser, nr.DocType, nr.DocDate);
END

Answer 2

A user on the MSDN forums had a similar problem using the Excel Importer. He wrote a few functions to check and validate his tables to prevent entry of and remove existing duplicates. Take a look: Automatic Validation Error Handling.