Grouping on some common values - sql

Question

This is a hard problem to explain, but I'm trying to create a SQL query that generates a list of parent groups that contains all groups where at least one group shares a product with another group. But they don't ALL have to share products, as long as one other group does they would be included in the parent group.
So for example: Because group 1 has {101,102,103} and group 5 has {101,104,105} they would be considered both part of the same parent group because they share
product 101 in common. So would group 4 {104}, because it has product 104 in common with group 5 (even though it doesn't have a product id in common with group 1).
Example Data:
group_id product_id
1 101
1 102
1 103
2 101
3 103
4 104
5 101
5 104
5 105
6 105
6 106
6 107
7 110
7 111
Results:
parent_group_id group_id
1 1
1 2
1 3
1 4
1 5
1 6
2 7
There is no real limit to the amount of products that could be listed under a group.
I'm not really sure how to go about tackling this. Perhaps recursion using a CTE?
Ideally I'd like to be able to do this on the fly so that I can find all linked products and query them together as a large set.
Edit:
I based the following solution on Raul's answer below. The change was to the bottomLevel CTE. In their solution, the value of the group_id matters and grouping could be "missed". For example, in dataset below, group 2 would not be seen to have a parent group id of 1 because the groups that links 2 to 1 (5,6 and 8) have group ids larger then 2. My solution is to just use a straightforward self join on product id. This solves that problem, but the performance is brutal (stopped it after 30mins) when I use my testing dataset of 150K rows. In production I could expect millions.
I tried tossing the bottomLevel CTE into a temp table and putting an index on it and that helps a bit with smaller datasets, but still way too slow on the full set.
Am I out of luck here?
CREATE TABLE #products
(
group_id int not null,
product_id int not null
)
INSERT INTO #products
VALUES(1, 101)
,(1, 102)
,(1, 103)
,(2, 110)
,(2, 111)
,(3, 103)
,(4, 104)
,(5, 101)
,(5, 104)
,(5, 105)
,(6, 105)
,(6, 106)
,(6, 107)
,(8,106)
,(8,111)
,(9,201)
,(10,300)
,(11,300)
,(11,301)
CREATE CLUSTERED INDEX cx_prods ON #products (group_id,product_id);
----------------------------------------------------------------
;WITH bottomLevel AS (
SELECT DISTINCT
sp.group_id as parent_group_id
,p.group_id
FROM
#products p
inner JOIN
#products sp
ON
sp.product_id = p.product_id
),
rc AS (
SELECT parent_group_id
, group_id
FROM bottomLevel
UNION ALL
SELECT b.parent_group_id
, r.group_id
FROM rc r
INNER JOIN bottomLevel b
ON r.parent_group_id = b.group_id
AND b.parent_group_id < r.parent_group_id
)
SELECT MIN(parent_group_id) as parent_group_id
, group_id
FROM rc
GROUP BY group_id
ORDER BY group_id
OPTION (MAXRECURSION 32767)
DROP TABLE #products

Answer 1

Marking Raul's answer as accepted because it helped me find the right direction.
But for those who may find this later, here is what I did.
The CTE method I based on Raul's answer worked, but was much too slow for my needs. I explored using the new graph features in SQL Server 2017, but it doesn't support transitive closure yet. So no luck there. But it did provide me with a term to search for : transitive closure clustering. I found the following two articles on doing it in SQL Server.
This one from Davide Mauri:
http://sqlblog.com/blogs/davide_mauri/archive/2017/11/12/lateral-thinking-transitive-closure-clustering-with-sql-server-uda-and-json.aspx
And this one from Itzik Ben-Gan:
http://www.itprotoday.com/microsoft-sql-server/t-sql-puzzle-challenge-grouping-connected-items
Both very helpful in understanding the problem, but I used Ben-Gan's solution 4.
It uses a while loop to unfold the connected nodes and removes processed edges from the temp input table as it runs.
It runs very fast on small to medium sets, and scales well. My test data of 1.2m rows runs in 2 minutes.
Here is my version of it:
First create a table to store the test data:
CREATE TABLE [dbo].[GroupsToProducts](
[group_id] [INT] NOT NULL,
[product_id] [INT] NOT NULL,
CONSTRAINT [PK_GroupsToProducts] PRIMARY KEY CLUSTERED
(
[group_id] ASC,
[product_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
INSERT INTO GroupsToProducts
VALUES(1, 101)
,(1, 102)
,(1, 103)
,(2, 110)
,(2, 111)
,(3, 103)
,(4, 104)
,(5, 101)
,(5, 104)
,(5, 105)
,(6, 105)
,(6, 106)
,(6, 107)
,(8,106)
,(8,111)
,(9,201)
,(10,300)
,(11,300)
,(11,301)
Then run the script to generate the clusters.
CREATE TABLE #group_rels
(
from_group_id int not null,
to_group_id int not null
)
INSERT INTO #group_rels
SELECT
p.group_id AS from_group_id,
sp.group_id AS to_group_id
FROM
GroupsToProducts p
inner JOIN
GroupsToProducts sp
ON
sp.product_id = p.product_id
AND p.group_id < sp.group_id
GROUP BY
p.group_id,
sp.group_id
CREATE UNIQUE CLUSTERED INDEX idx_from_group_id_to_group_id ON #group_rels(from_group_id, to_group_id);
CREATE UNIQUE NONCLUSTERED INDEX idx_to_group_id_from_group_id ON #group_rels(to_group_id, from_group_id);
-------------------------------------------------
CREATE TABLE #G
(
group_id INT NOT NULL,
parent_group_id INT NOT NULL,
lvl INT NOT NULL,
PRIMARY KEY NONCLUSTERED (group_id),
UNIQUE CLUSTERED(lvl, group_id)
);
DECLARE #lvl AS INT = 1, #added AS INT, #from_group_id AS INT, #to_group_id AS INT;
DECLARE #CurIds AS TABLE(id INT NOT NULL);
-- gets the first relationship pair
-- will use the from_group_id as a 'root' group
SELECT TOP (1)
#from_group_id = from_group_id,
#to_group_id = to_group_id
FROM
#group_rels
ORDER BY
from_group_id,
to_group_id;
SET #added = ##ROWCOUNT;
WHILE #added > 0
BEGIN
-- inserts two rows into the output table:
-- a self pairing using from_group_id
-- AND the actual relationship pair
INSERT INTO #G
(group_id, parent_group_id, lvl)
VALUES
(#from_group_id, #from_group_id, #lvl),
(#to_group_id, #from_group_id, #lvl);
-- removes the pair from input table
DELETE FROM #group_rels
WHERE
from_group_id = #from_group_id
AND to_group_id = #to_group_id;
WHILE #added > 0
BEGIN
-- increment the lvl variable so we only look at the most recently inserted data
SET #lvl += 1;
----------------------------------------------------------------------------
-- the same basic chunk of code is done twice
-- once for group_ids in the output table that join against from_group_id and
-- once for group_ids in the output table that join against to_group_id
-- 1 - join the output table against the input table, looking for any groups that join
-- against groups that have already been found to (directly or indirectly) connect to the root group.
-- 2 - store the found group_ids in the #CurIds table variable and delete the relationship from the input table.
-- 3 - insert the group_ids in the output table using #from_group_id (the current root node id) as the parent group id
-- if any rows are added to the output table in either chunk, loop and look for any groups that may connect to them.
------------------------------------------------------------------------------
DELETE FROM #CurIds;
DELETE FROM group_rels
OUTPUT deleted.to_group_id AS id INTO #CurIds(id)
FROM
#G AS G
INNER JOIN #group_rels AS group_rels
ON G.group_id = group_rels.from_group_id
WHERE
lvl = #lvl - 1;
INSERT INTO #G
(group_id, parent_group_id, lvl)
SELECT DISTINCT
id,
#from_group_id AS parent_group_id,
#lvl AS lvl
FROM
#CurIds AS C
WHERE
NOT EXISTS
(
SELECT
*
FROM
#G AS G
WHERE
G.group_id = C.id
);
SET #added = ##ROWCOUNT;
-----------------------------------------------------------------------------------
DELETE FROM #CurIds;
DELETE FROM group_rels
OUTPUT deleted.from_group_id AS id INTO #CurIds(id)
FROM
#G AS G
INNER JOIN #group_rels AS group_rels
ON G.group_id = group_rels.to_group_id
WHERE
lvl = #lvl - 1;
INSERT INTO #G
(group_id, parent_group_id, lvl)
SELECT DISTINCT
id,
#from_group_id AS grp,
#lvl AS lvl
FROM
#CurIds AS C
WHERE
NOT EXISTS
(
SELECT
*
FROM
#G AS G
WHERE
G.group_id = C.id
);
SET #added += ##ROWCOUNT;
END;
------------------------------------------------------------------------------
-- At this point, no new rows were added, so the cluster should be complete.
-- Look for another row in the input table to use as a root group
SELECT TOP (1)
#from_group_id = from_group_id,
#to_group_id = to_group_id
FROM
#group_rels
ORDER BY
from_group_id,
to_group_id;
SET #added = ##ROWCOUNT;
END;
SELECT
parent_group_id,
group_id,
lvl
FROM #G
--ORDER BY
--parent_group_id,
--group_id,
--lvl
-------------------------------------------------
DROP TABLE #G
DROP TABLE #group_rels

Answer 2

Take the following statement as a head start:
CREATE TABLE products
(
group_id int not null,
product_id int not null
)
INSERT INTO products
VALUES(1, 101)
,(1, 102)
,(1, 103)
,(2, 101)
,(3, 103)
,(4, 104)
,(5, 101)
,(5, 104)
,(5, 105)
,(6, 105)
,(6, 106)
,(6, 107)
,(7, 110)
,(7, 111)
;WITH bottomLevel AS (
SELECT ISNULL(MIN(matchedGroup),group_id) as parent_group_id
, group_id
FROM products p
OUTER APPLY (
SELECT MIN(group_id) AS matchedGroup
FROM products sp
WHERE sp.group_id != p.group_id
AND sp.product_id = p.product_id
) oa
GROUP BY p.group_id
),
rc AS (
SELECT parent_group_id
, group_id
FROM bottomLevel
UNION ALL
SELECT b.parent_group_id
, r.group_id
FROM rc r
INNER JOIN bottomLevel b
ON r.parent_group_id = b.group_id
AND b.parent_group_id < r.parent_group_id
)
SELECT MIN(parent_group_id) as parent_group_id
, group_id
FROM rc
GROUP BY group_id
ORDER BY group_id
OPTION (MAXRECURSION 32767)
I first grouped by group_id getting the smallest group_id having matching products and recursively joined the parents that have a minor parent.
Now this solution will probably not cover all exceptions you might encounter in a production, but should help you start somewhere.
Also if you have a large product table, this might run really slow, so consider doing this data matching using C#, Spark or SSIS or any other data manipulation engine.