Find missing values within groups of data using SQL - sql

Question

Recently I need to generate a report based on the fact:
TableA has the following 2 columns UserID and DocumentType
I have been provided a list of 'mandatory' document types: Type1, Type2, Type3 and I need to return every UserID that doesn't have all three of these types along with the types they are missing.
For example if TableA contains the following rows
12 Type1
12 Type2
12 Type4
13 Type1
13 Type2
13 Type3
14 Type1
15 Type6
15 Type7
15 Type8
Then ideally the output would be something like:
12 Type3
14 Type2, Type3
15 Type1, Type2, Type3
Ideally, the query to generate the results should be able to handle up to tens of millions of records.
We recently implemented a solution to a similar question (which is a bit more complicated than this) with SQL server 2012. It takes 3 and a half minutes to get the full report among multiple tables with around 4 million records in total. We wonder whether there are better ideas which can do this faster.
Please feel free to share your ideas which can solve this problem.
Thank you! :)

Answer 1

DEMO
WITH
base ([DocumentType]) as (
SELECT 'Type1' UNION ALL
SELECT 'Type2' UNION ALL
SELECT 'Type3'
),
users as (
SELECT DISTINCT [userID]
FROM Table1 t
),
pairs as (
SELECT *
FROM users, base
)
SELECT p.userID, p.[DocumentType], t.[DocumentType]
FROM pairs p
LEFT JOIN Table1 t
ON p.[DocumentType] = t.[DocumentType]
AND p.[userID] = t.[userID]
WHERE t.[DocumentType] IS NULL
OUTPUT

Answer 2

i think you need something like this
select x2.*
(select *
from (select distinct UserID from [table])x
cross join
(select 'type1' DocumentType union
select 'type2' union
select 'type3' ) y
) x2
left join [table] y2
on y2.UserID = x2.UserID
and y2.DocumentType = x2.DocumentType
where y2.DocumentType is null
order by x2.UserID

Answer 3

here is the FOR XML path concatenation method:
CREATE TABLE TableA (ID INT, TypeCol CHAR(5));
INSERT INTO TableA (ID,TypeCol) VALUES (12,'Type1')
,(12,'Type2')
,(12,'Type4')
,(13,'Type1')
,(13,'Type2')
,(13,'Type3')
,(14,'Type1')
,(15,'Type6')
,(15,'Type7')
,(15,'Type8')
;WITH cteRequiredTypes AS (
SELECT 'type1' as TypeCol
UNION ALL
SELECT 'type2'
UNION ALL
SELECT 'type3'
)
, cteTableAIds AS (
SELECT DISTINCT Id
FROM
TableA
)
, cteMissingTypes AS (
SELECT
i.ID
,r.TypeCol
FROm
cteRequiredTypes r
CROSS JOIN cteTableAIds i
LEFT JOIN TableA a
ON r.TypeCol = a.TypeCol
AND i.ID = a.ID
WHERE
a.ID IS NULL
)
SELECT
DISTINCT a.ID
,STUFF(
(SELECT ',' + TypeCol
FROM
cteMissingTypes t
WHERE t.ID = a.ID
FOR XML PATH(''))
,1,1,'')
FROM
cteMissingTypes a
I believe for checking a large dataset a conditional aggregation query will probably be better performance.
Conditional Aggregation
SELECT
ID
,CASE WHEN SUM(IIF(TypeCol = 'type1',1,0)) = 0 THEN 'type1' ELSE '' END as Type1
,CASE WHEN SUM(IIF(TypeCol = 'type2',1,0)) = 0 THEN 'type2' ELSE '' END as Type2
,CASE WHEN SUM(IIF(TypeCol = 'type3',1,0)) = 0 THEN 'type3' ELSE '' END as Type3
,STUFF(
REPLACE (
REPLACE (
+ ',' + CASE WHEN SUM(IIF(TypeCol = 'type1',1,0)) = 0 THEN 'type1' ELSE '' END
+ ',' + CASE WHEN SUM(IIF(TypeCol = 'type2',1,0)) = 0 THEN 'type2' ELSE '' END
+ ',' + CASE WHEN SUM(IIF(TypeCol = 'type3',1,0)) = 0 THEN 'type3' ELSE '' END
,',,,',',,')
,',,',',')
,1,1,'') as MissingTypeList
FROM
TableA
GROUP BY
ID
HAVING
SUM(IIF(TypeCol = 'type1',1,0)) = 0
OR SUM(IIF(TypeCol = 'type2',1,0)) = 0
OR SUM(IIF(TypeCol = 'type3',1,0)) = 0

Answer 4

The easy way
SELECT DISTINCT UserID
FROM your_table
EXCEPT
(
SELECT UserID
FROM your_table
WHERE DocumentType = ('Type1')
UNION
SELECT UserID
FROM your_table
WHERE DocumentType = ('Type2')
UNION
SELECT UserID
FROM your_table
WHERE DocumentType = ('Type3')
)
Can't be more exact in the rules than this -- so your compiler will look at indexes etc and optimize.
Unless there are parts of the problem you are not telling us.