Select duplicate rows based on time difference and occurrence count - sql

Question

I have a table like this :
As you can see, some records with the same farsi_pelak field have been added(detected) more than 1 time within a few seconds.
That's happened because of some application bug which has been fixed.
Now I need to select and then delete duplicate rows which have been added at the same time (+- few seconds)
And this is my query :
SELECT TOP 100 PERCENT
y.id, y.farsi_pelak , y.detection_date_p , y.detection_time
FROM dbo._tbl_detection y
INNER JOIN
(SELECT TOP 100 PERCENT
farsi_pelak , detection_date_p
FROM dbo._tbl_detection WHERE camera_id = 2
GROUP BY farsi_pelak , detection_date_p
HAVING COUNT(farsi_pelak)>1) dt
ON
y.farsi_pelak=dt.farsi_pelak AND y.detection_date_p =dt.detection_date_p
ORDER BY farsi_pelak , detection_date_p DESC
But I can't calculate the time difference because my detection_time field should not be grouped by.

Answer 1

If you use SQL Server 2012 or later, you can use LAG function to get the values from the "previous" row.
Then calculate the difference between adjacent timestamps and find those rows where this difference is small.
WITH
CTE
AS
(
SELECT
id
,farsi_pelak
,detection_date_p
,detection_time
,LAG(detection_time) OVER (PARTITION BY farsi_pelak
ORDER BY detection_date_p, detection_time) AS prev_detection_time
FROM dbo._tbl_detection
)
,CTE_Diff
AS
(
SELECT
id
,farsi_pelak
,detection_date_p
,detection_time
,prev_detection_time
,DATEDIFF(second, prev_detection_time, detection_time) AS diff
FROM CTE
)
SELECT
id
,farsi_pelak
,detection_date_p
,detection_time
,prev_detection_time
,diff
FROM CTE_Diff
WHERE
diff <= 10
;
When you run this query and verify that it returns only rows that you want to delete, you can change the last SELECT to DELETE:
WITH
CTE
AS
(
SELECT
id
,farsi_pelak
,detection_date_p
,detection_time
,LAG(detection_time) OVER (PARTITION BY farsi_pelak
ORDER BY detection_date_p, detection_time) AS prev_detection_time
FROM dbo._tbl_detection
)
,CTE_Diff
AS
(
SELECT
id
,farsi_pelak
,detection_date_p
,detection_time
,prev_detection_time
,DATEDIFF(second, prev_detection_time, detection_time) AS diff
FROM CTE
)
DELETE
FROM CTE_Diff
WHERE
diff <= 10
;

Answer 2

I guess you need rownumber to check time as below keeping the earliest time data and discarding the rest detection time for rownums greater than 1
Select y.id, y.farsi_pelak ,
y.detection_date_p , y.detection_time,
row_number() over (partition by
y.farsi_pelak,
y.detection_date_p order by
y.detection_time) rn
from ( the above query) where rn>1