Extracting rows from fact table which have missing data, using metadata table - sql

Question

I have a situation where I have:
A fact table with an id column which is NOT unique but is never null. This fact also has a lot of other dimensions (columns) which may be with a default value -1 (which logically means null)
Example:
id | Dimension1 | Dimension2 | Dimension3
1 Value -1 Value
1 -1 -1 Value
2 -1 Value Value
A metadata table that has the same dimensions as the fact table. Each row in this table represents an unique id from the fact table. Rest of the columns are populated with either null or 1, where 1 means that this dimension is a required dimension in the fact table for this id.
Example:
id | Dimension1 | Dimension2 | Dimension3
1 1 1
2 1 1
My goal is to get ONLY the rows from the fact table that are missing required information according to the metadata table. So from the examples above I would get only the row with id = 1 where Dimension1 = -1, since metadata table says for id = 1 dimensions 1 and 3 are required.
Is there an easy way of doing this?
I have made a very complicated query where there is join between these two tables and a case checks between all dimensions (I have more than 100 of them). Then these checks assign a -1 if dimension is missing in fact but is required, and there is an outer query that would sum these for all rows and only pick up rows with negative sum.
It does not work to 100% and I think its way too complicated to run on a real big fact table, so I'm open to ideas.
edit: Dynamic SQL is not allowed :(

Answer 1

I would suggest using a cte and an except query... in this solution, you will have to add the cases as well, but the join seems far more simple to me and you don't need to sum up any dummy values...
DECLARE #t TABLE(
id int, Dimension1 int, Dimension2 int, Dimension3 int
)
DECLARE #tMeta TABLE(
id int, Dimension1 int, Dimension2 int, Dimension3 int
)
INSERT INTO #t VALUES (1, 123, -1, 345), (1, -1, -1, 246), (2, -1, 567, 987)
INSERT INTO #tMeta VALUES (1, 1, NULL, 1), (2, NULL, 1, 1)
;WITH cte AS(
SELECT id,
CASE WHEN Dimension1 = -1 THEN NULL ELSE 1 END Dimension1,
CASE WHEN Dimension2 = -1 THEN NULL ELSE 1 END Dimension2,
CASE WHEN Dimension3 = -1 THEN NULL ELSE 1 END Dimension3
FROM #t
EXCEPT
SELECT *
FROM #tMeta
EXCEPT
SELECT id, ISNULL(Dimension1,1), ISNULL(Dimension2,1), ISNULL(Dimension3,1)
FROM #tMeta
)
SELECT t.*
FROM #t t
JOIN cte c ON t.id = c.id
AND CASE WHEN t.Dimension1 = -1 THEN -1 ELSE 1 END = ISNULL(c.Dimension1, -1)
AND CASE WHEN t.Dimension2 = -1 THEN -1 ELSE 1 END = ISNULL(c.Dimension2, -1)
AND CASE WHEN t.Dimension3 = -1 THEN -1 ELSE 1 END = ISNULL(c.Dimension3, -1)

Answer 2

You can use UNPIVOT to simplify query also you don't have ROWId in your fact table so the first CTE to make ROW_NUMBER() works as a RowId in the fact table. Then we make unpivoted tables (fact and template table) and join them:
WITH TFBase AS
(
SELECT TF.*, ROW_NUMBER() OVER (ORDER BY ID) as TableRowID FROM TF
),
TFU AS
(
select id,TableRowID,dim,val
from TFBase
unpivot
(
val for dim in (Dimension1, Dimension2, Dimension3)
) u
WHERE U.Val <>-1
)
,
TFT AS
(
select id,dim,val
from TTemplate
unpivot
(
val for dim in (Dimension1, Dimension2, Dimension3)
) u
WHERE Val is NOT NULL
)
SELECT * FROM TFBase WHERE
TableRowID IN
(
SELECT TableRowID FROM TFU
LEFT JOIN TFT ON
(TFU.id=TFT.id) AND (TFU.dim = TFT.dim)
GROUP BY TableRowID, TFU.ID
HAVING COUNT(TFT.Val) <> (SELECT COUNT(*) FROM TFT WHERE ID = TFU.ID)
)