Calculating overlap between groups - sql

Question

I have a table with two columns of interest, item_id and bucket_id. There are a fixed number of values for bucket_id and I'm okay with listing them out if I need to.
Each item_id can appear multiple times, but each occurrence will have a separate bucket_id value. For example, the item_id of 123 can appear twice in the table, once under bucket_id of A, once under B.
My goal is to determine how much overlap exists between each pair of bucket_id values and display it as an N-by-N matrix.
For example, consider the following small example table:
item_id bucket_id
========= ===========
111 A
111 B
111 C
222 B
222 D
333 A
333 C
444 C
So for this dataset, buckets A and B have one item_id in common, buckets C and D have no items in common, etc.
I would like to get the above table formatted into something like the following:
A B C D
===================================
A 2 1 2 0
B 1 2 1 1
C 2 1 3 0
D 0 1 0 1
In the above table, the intersect of a row and column tells you how many records exist in both bucket_id values. For example, where the A row intersects the C column we have a 2, because there are 2 records that exist in both bucket_id A and C. Because the intersection of X and Y is the same as the intersection of Y and X, the above table is mirrored across the diagonal.
I imagine the query involves a PIVOT, but I can't for the life of me figure out how to get it working.

Answer 1

You can use simple PIVOT:
SELECT t1.bucket_id,
SUM( CASE WHEN t2.bucket_id = 'A' THEN 1 ELSE 0 END ) AS A,
SUM( CASE WHEN t2.bucket_id = 'B' THEN 1 ELSE 0 END ) AS B,
SUM( CASE WHEN t2.bucket_id = 'C' THEN 1 ELSE 0 END ) AS C,
SUM( CASE WHEN t2.bucket_id = 'D' THEN 1 ELSE 0 END ) AS D
FROM table1 t1
JOIN table1 t2 ON t1.item_id = t2.item_id
GROUP BY t1.bucket_id
ORDER BY 1
;
or you can use Oracle PIVOT clause (works on 11.2 and above):
SELECT * FROM (
SELECT t1.bucket_id AS Y_bid,
t2.bucket_id AS x_bid
FROM table1 t1
JOIN table1 t2 ON t1.item_id = t2.item_id
)
PIVOT (
count(*) FOR x_bid in ('A','B','C','D')
)
ORDER BY 1
;
Examples: http://sqlfiddle.com/#!4/39d30/7

Answer 2

I believe this should get you the data you need. Pivoting the table could then be done programmatically (or in Excel, etc.).
-- This gets the distinct pairs of buckets
select distinct
a.name,
b.name
from
bucket a
join bucket b
where
a.name < b.name
order by
a.name,
b.name
+ --------- + --------- +
| name | name |
+ --------- + --------- +
| A | B |
| A | C |
| A | D |
| B | C |
| B | D |
| C | D |
+ --------- + --------- +
6 rows
-- This gets the distinct pairs of buckets with the counts you are looking for
select distinct
a.name,
b.name,
count(distinct bi.item_id)
from
bucket a
join bucket b
left outer join bucket_item ai on ai.bucket_name = a.name
left outer join bucket_item bi on bi.bucket_name = b.name and ai.item_id = bi.item_id
where
a.name < b.name
group by
a.name,
b.name
order by
a.name,
b.name
+ --------- + --------- + ------------------------------- +
| name | name | count(distinct bi.item_id) |
+ --------- + --------- + ------------------------------- +
| A | B | 2 |
| A | C | 1 |
| A | D | 0 |
| B | C | 2 |
| B | D | 0 |
| C | D | 0 |
+ --------- + --------- + ------------------------------- +
6 rows
Here's the entire example with the DDL and inserts to set it up (this is in mysql but the same ideas apply elsewhere):
use example;
drop table if exists bucket;
drop table if exists item;
drop table bucket_item;
create table bucket (
name varchar(1)
);
create table item(
id int
);
create table bucket_item(
bucket_name varchar(1) references bucket(name),
item_id int references item(id)
);
insert into bucket values ('A');
insert into bucket values ('B');
insert into bucket values ('C');
insert into bucket values ('D');
insert into item values (111);
insert into item values (222);
insert into item values (333);
insert into item values (444);
insert into item values (555);
insert into bucket_item values ('A',111);
insert into bucket_item values ('A',222);
insert into bucket_item values ('A',333);
insert into bucket_item values ('B',222);
insert into bucket_item values ('B',333);
insert into bucket_item values ('B',444);
insert into bucket_item values ('C',333);
insert into bucket_item values ('C',444);
insert into bucket_item values ('D',555);
-- query to get distinct pairs of buckets
select distinct
a.name,
b.name
from
bucket a
join bucket b
where
a.name < b.name
order by
a.name,
b.name
;
select distinct
a.name,
b.name,
count(distinct bi.item_id)
from
bucket a
join bucket b
left outer join bucket_item ai on ai.bucket_name = a.name
left outer join bucket_item bi on bi.bucket_name = b.name and ai.item_id = bi.item_id
where
a.name < b.name
group by
a.name,
b.name
order by
a.name,
b.name
;