How to remove duplicates while sorting by unique datetime - sql

Question

I am working with a fairly bad data source, the column that has the information I need is within a varchar(max) and is delimited. However, the data can be duplicated across multiple rows so I am trying to remove these duplicates.
This can be done by trimming the column I am interested in, as when I repeat occurs the "ID" gets re-appended to the end of the column. Then I am taking a distinct of that to, which I then concatenate the results; it isn't pretty.
Example data and the query I currently use SQL Fiddle
Data Table
| id | callID | callDateTime | history |
|----|--------|-----------------------------|-------------------------------------|
| 1 | 1 | 2021-01-01 10:00:00.0000000 | Amount: 10, Ref:123, ID:123 |
| 2 | 1 | 2021-01-01 10:01:00.0000000 | Amount: 10, Ref:123, ID:123, ID:123 |
| 3 | 2 | 2021-01-01 11:00:00.0000000 | Amount:12.44, Ref:SIS, ID:124 |
| 4 | 2 | 2021-01-01 11:02:00.0000000 | Amount:11.22, Ref:Dad, ID:124 |
| 5 | 2 | 2021-01-01 11:01:00.0000000 | Amount:11.22, Ref:Mum, ID:124 |
| 6 | 3 | 2021-01-01 12:00:00.0000000 | Amount:11, ID:125 |
Query
select CallID, Concat([1],',', [2],',',[3])
from
(
select CallID, historyEdit, ROW_NUMBER() over (partition by callID order by callID) as rowNum
from
(
select distinct callID,
substring(history, 0, charindex(', ID:',history)) historyEdit
from test
) a
)b
PIVOT(max(historyEdit) for rowNum IN ([1],[2],[3])) piv
Result
| CallID | |
|--------|-------------------------------------------------------------------|
| 1 | Amount: 10, Ref:123,, |
| 2 | Amount:11.22, Ref:Dad,Amount:11.22, Ref:Mum,Amount:12.44, Ref:SIS |
| 3 | Amount:11,, |
The issue is that I need to ensure the concatenate part is doing so in the order of when the events occurred. In the above you will see that CallID 2 is in the wrong order as Information 3 is coming before Information 2, I did try to sort the base table by callDateTime first and then run the query, however it does seem to yield somewhat random results. Sometimes it will be in the correct order, other times it won't be. I assume this is because I am not specifying any order by clause in the query.
Including the callDateTime in the results then causes the distinct not return the unqiue data rows as the callDateTime is still unique to that duplicated row of data
I am using SQL Server v12
Desired Result
| CallID | |
|--------|-------------------------------------------------------------------|
| 1 | Amount: 10, Ref:123,, |
| 2 | Amount:12.44, Ref:SiS,Amount:11.22, Ref:Mum,Amount:11.22, Ref:Dad |
| 3 | Amount:11,, |

Answer 1

If I understand correctly, you want to break apart the history and recombine -- without duplicates -- for each callid. If so, you can use string_split() and string_agg():
select callid, string_agg(value, ', ')
from (select distinct t.callid, s.value
from test t cross apply
(select trim(s.value) as value
from string_split(t.history, ',') s
) s
) st
group by callid;
Here is a db<>fiddle.

Answer 2

You could use TOP clause inside the select to order the records before pivoting the results if you are sure of the number of records like below:
select callID, historyEdit
from
(
select distinct top 100000 callID, callDateTime,
substring(history, 0, charindex(', ID:',history)) historyEdit
from test
order by callDateTime
)t
Please see the results here.

Answer 3

One way to calculate the strings could be to CROSS APPLY using an ordinal splitter to separate the 'history' column into components which can be enumerated. The result is very close to what was provided in the question. Maybe the provided expected results aren't accurately representative? Something like this
Ordinal splitter described here
CREATE FUNCTION [dbo].[DelimitedSplit8K_LEAD]
--===== Define I/O parameters
(#pString VARCHAR(8000), #pDelimiter CHAR(1))
RETURNS TABLE WITH SCHEMABINDING AS
RETURN
WITH E1(N) AS (
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1
), --10E+1 or 10 rows
E2(N) AS (SELECT 1 FROM E1 a, E1 b), --10E+2 or 100 rows
E4(N) AS (SELECT 1 FROM E2 a, E2 b), --10E+4 or 10,000 rows max
cteTally(N) AS (--==== This provides the "zero base" and limits the number of rows right up front
-- for both a performance gain and prevention of accidental "overruns"
SELECT 0 UNION ALL
SELECT TOP (DATALENGTH(ISNULL(#pString,1))) ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM E4
),
cteStart(N1) AS (--==== This returns N+1 (starting position of each "element" just once for each delimiter)
SELECT t.N+1
FROM cteTally t
WHERE (SUBSTRING(#pString,t.N,1) = #pDelimiter OR t.N = 0)
)
--===== Do the actual split. The ISNULL/NULLIF combo handles the length for the final element when no delimiter is found.
SELECT ItemNumber = ROW_NUMBER() OVER(ORDER BY s.N1),
Item = SUBSTRING(#pString,s.N1,ISNULL(NULLIF((LEAD(s.N1,1,1) OVER (ORDER BY s.N1) - 1),0)-s.N1,8000))
FROM cteStart s
;
query
with
unq_cte as (
select distinct callID
from #test),
exp_cte as (
select callID, callDateTime , dl.*,
row_number() over (partition by callID, dl.Item order by callDateTime) as rn
from #test t
cross apply dbo.DelimitedSplit8K_LEAD(t.history, ',') dl)
select t.callID,
stuff((select ',' + case when rn>1 then '' else Item end
from exp_cte tt
where t.callID = tt.callID
and ltrim(rtrim(Item)) not like 'ID%'
order by tt.callDateTime, tt.ItemNumber for xml path('')), 1, 1, '') [value1]
from unq_cte t
group by t.callID;
callID value1
1 Amount: 10, Ref:123,,
2 Amount:12.44, Ref:SIS,Amount:11.22, Ref:Mum,, Ref:Dad
3 Amount:11