How to exclude duplicate values / NULL values SQL - sql

Question

I hope that someone from you will be able to help me as I got stuck with a (silly) condition logic.
I am joining 2 temp tables.
The target is to see the Jobs, Tracking numbers, etc. (all actions) in one row always for one Consignee (Consignee Ref). In other words, to show all actions/data per Consignee Ref.
The issue I got is duplacated values. The problem is that the Original Job (620X) field can have assigned Tracking_1 but it can be also null. Also the Second Job (629X) filed can have but do not need to have assigned the Tracking number/value. It can be also the case that both Tracking_1 and Tracking are NULL for one Consignee Ref.
When I would exclude Jobs where Tracking_1 or Tracking is NULL, then I will loose the Jobs that have no Tracking Numbers at all. So I used the code below, but then I am getting duplicates.
SELECT * FROM #Temp t1
INNER JOIN #Temp2 t2 on t1.[Consignee Ref] = t2.[Consignee Ref]
WHERE t1.[Tracking_1] is not null
ORDER BY [Original Job (620X)] asc
How can I make it that I will get rid of the unnecessary duplicates (an example marked in red)?
And at the same time not to loose those jobs that have no Tracking Numbers?
I hope it makes sense.
Thank you very much in advance for any advise!
This is the entire code:
IF OBJECT_ID('tempdb..#Temp') IS NOT NULL
DROP TABLE #Temp;
SELECT distinct
cne.[Consignee Ref],
s.[Trial AWB] as [Original Job (620X)],
rl.[CarrierReference] as [Tracking_1],
CAST(st.[Sched Collection date] as date) AS [Collection Date],
CAST(st.[Act Del Date] as date) AS [Delivery Date],
s.[Clientaccountcode] as [Account Code]
into #Temp
FROM MKN_Reporting.dbo.shipment AS s WITH (NOLOCK)
LEFT JOIN MKN_Reporting.dbo.[Lookup Month By JN Tb] AS b WITH (NOLOCK) ON s.[id] = b.[JobId]
LEFT JOIN MKN_Reporting.dbo.[Lookup Branch currency Tb] AS c WITH (NOLOCK) ON b.BranchPrefix = c.[Branch Prefix]
LEFT JOIN MKN_Reporting.dbo.[Lookup Client group Tb] AS g WITH (NOLOCK) ON s.[Clientaccountcode] = g.[Customer A/c]
LEFT JOIN MKN_Reporting.dbo.[Chargeto] AS cg WITH (NOLOCK) ON s.[id] = cg.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Shipper] AS sh WITH (NOLOCK) ON s.[id] = sh.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Cnee] AS cne WITH (NOLOCK) ON s.[id] = cne.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Lookup Country & Region Tb] AS r WITH (NOLOCK) ON sh.[Shipper Country code] = r.[Country Code]
LEFT JOIN MKN_Reporting.dbo.[Lookup Country & Region Tb] AS reg WITH (NOLOCK) ON cne.[Consignee Country code] = reg.[Country Code]
INNER JOIN MKN_Reporting.dbo.[Status] st ON COALESCE(s.ParentId, s.id) = st.Jobid
LEFT JOIN [CARRIERS_CHARGES] AS cc ON s.id = cc.JobId
LEFT JOIN [RouteLegs] as rl on cc.RouteLegId = rl.id
WHERE [Clientaccountcode] in ('US429', 'MI1091')
--AND rl.[CarrierReference] is not null
-------------------------------------------
IF OBJECT_ID('tempdb..#Temp2') IS NOT NULL
DROP TABLE #Temp2;
SELECT distinct
cne.[Consignee Ref],
s.[Study Number] as [Study_],
s.[Site Number] as [Site No_],
s.[Trial AWB] as [Second (629X)],
rl.[CarrierReference] as [Tracking],
CAST(st.[Sched Collection date] as date) AS [Collection Date],
CAST(st.[Act Del Date] as date) AS [Delivery Date],
s.[Clientaccountcode] as [Account Code]
into #Temp2
FROM MKN_Reporting.dbo.shipment AS s WITH (NOLOCK)
LEFT JOIN MKN_Reporting.dbo.[Lookup Month By JN Tb] AS b WITH (NOLOCK) ON s.[id] = b.[JobId]
LEFT JOIN MKN_Reporting.dbo.[Lookup Branch currency Tb] AS c WITH (NOLOCK) ON b.BranchPrefix = c.[Branch Prefix]
LEFT JOIN MKN_Reporting.dbo.[Lookup Client group Tb] AS g WITH (NOLOCK) ON s.[Clientaccountcode] = g.[Customer A/c]
LEFT JOIN MKN_Reporting.dbo.[Chargeto] AS cg WITH (NOLOCK) ON s.[id] = cg.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Shipper] AS sh WITH (NOLOCK) ON s.[id] = sh.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Cnee] AS cne WITH (NOLOCK) ON s.[id] = cne.[jobid]
LEFT JOIN MKN_Reporting.dbo.[Lookup Country & Region Tb] AS r WITH (NOLOCK) ON sh.[Shipper Country code] = r.[Country Code]
LEFT JOIN MKN_Reporting.dbo.[Lookup Country & Region Tb] AS reg WITH (NOLOCK) ON cne.[Consignee Country code] = reg.[Country Code]
INNER JOIN MKN_Reporting.dbo.[Status] st ON COALESCE(s.ParentId, s.id) = st.Jobid
LEFT JOIN [CARRIERS_CHARGES] AS cc ON s.id = cc.JobId
left JOIN [RouteLegs] as rl on cc.RouteLegId = rl.id
WHERE [Clientaccountcode] in ('US1598')
--AND rl.[CarrierReference] is not null
---------------------------------------------
SELECT * FROM #Temp t1
INNER JOIN #Temp2 t2 on t1.[Consignee Ref] = t2.[Consignee Ref]
WHERE t1.[Tracking_1] is not null
ORDER BY [Original Job (620X)] asc

Answer 1

I guess OP is looking for some kind of field aggregation for T2.
With SQL Server 2017+, like in PostgreSQL, you can now use STRING_AGG (same syntax than the Oracle LISTAGG.. "Why did they not use the same keyword...").
STRING_AGG will ignore null values from T2 and requires grouping if not alone. You have to add the other T1 and T2 required fields if any, but take care for T2 that it can imply duplicates and may need aggregation too...
Here is an example with your provided TSQL :
SELECT t1.[Consignee Ref], t1.[Original Job (620X)], t1.[Tracking_1],
STRING_AGG(t2.[Tracking],',') WITHIN GROUP (ORDER BY t2.[Tracking] ASC) Tracking_T2
FROM #Temp t1
INNER JOIN #Temp2 t2 on t1.[Consignee Ref] = t2.[Consignee Ref]
WHERE t1.[Tracking_1] is not null
GROUP BY t1.[Consignee Ref], t1.[Original Job (620X)], t1.[Tracking_1]
ORDER BY t1.[Original Job (620X)] asc
EDIT: For previous SQL Server release (before 2017), you can achieve string aggregation with XML query :
EDIT2: Adding EXISTS clause to convey the INNER JOIN from the initial query.
SELECT t1.[Consignee Ref], t1.[Original Job (620X)], t1.[Tracking_1],
STUFF((SELECT ', ' + t2.[Tracking]
FROM #Temp2 t2
WHERE t1.[Consignee Ref] = t2.[Consignee Ref]
ORDER BY t2.[Tracking]
FOR XML PATH('')), 1, 1, N'') Tracking_T2
FROM #Temp t1
WHERE t1.[Tracking_1] is not null
AND EXISTS (SELECT DISTINCT 1 FROM #Temp2 t2 WHERE t2.[Consignee Ref] = t1.[Consignee Ref])
GROUP BY t1.[Consignee Ref], t1.[Original Job (620X)], t1.[Tracking_1]
ORDER BY t1.[Original Job (620X)] asc