SQL like: How to calculate intersection and union of <item,user> data - sql

Question

Need help in SQL:
I have a data with the following columns:
ItemId
UserId
Each row indicates that some item was bought by some user.
Example:
ItemId UserId
200 user1
200 user3
200 user4
300 user5
300 user3
for each I would like to calculate the following output table:
users(i) : number of users bought i
users(j) : number of users bought j
users(i, j) : number of users bought both i and j
users(i, ~j) : number of users bought i but not j
users(~i, j) : number of users bought j but not i
Example of output (from the example above):
i_itemId j_itemId users(i) users(j) users(i,j) users(i,~j) users(~i, j)
200 200 3 3 3 0 0
200 300 3 2 1 2 1
300 300 2 2 2 0 0
300 200 3 2 1 1 2
Note:
The data table is huge (11 GB) located on the cloud. I have a framework of SQL to work with. So I cannot download the file and run python (for example)
So the solution has to be written in SQL in efficient way
The solution does not have to be one single SQL statement.
I am looking for an efficient solution
We can assume that is a key
If someone have a better alternative for the Question header here i will be glad to update it :)

Answer 1

I am not sure if there is an "easy" way to accomplish this. One method is rather brute force: use a cross join to generate all the rows. Then use subqueries for each of the individual counts:
select i1.itemid, i2.itemid, i1.num as cnt1, i2.num as cnt2,
(select count(*)
from t u1 join
t u2
on u1.userid = u2.userid
where u1.itemid = i1.itemid and u2.itemid = i2.itemid
) as cnt_1_2,
(select count(*)
from t u1 left join
t u2
on u1.userid = u2.userid and u2.itemid = i2.itemid
where u1.itemid = i1.itemid and u2.itemid is null
) as cnt_1_not2,
(select count(*)
from t u1 left join
t u2
on u1.userid = u2.userid and u1.itemid = i1.itemid
where u2.itemid = i2.itemid and u1.itemid is null
) as cnt_not1_2
from (select itemid, count(*) as num from t group by itemid) i1 cross join
(select itemid, count(*) as num from t group by itemid) i2;

Answer 2

Here's a recipe
1) Create a temporary table to gather the I and J totals.
Disclaimer :
This example uses a MS SQL server datatype: INT.
So change it to the numeric type that your RDBMS supports.
Btw, in MS SQL Server, temporary tables start with #
create table TempTotals (iItemId int, jItemId int, TotalUsers int);
2) Fill it up with totals
delete from TempTotals;
insert into TempTotals (iItemId, jItemId, TotalUsers)
select
t1.ItemId as iItemId,
t2.ItemId as jItemId,
count(distinct t1.UserId) as TotalUsers
from YourTable t1
full join YourTable t2 on (t1.UserId = t2.UserId)
group by t1.ItemId, t2.ItemId;
3) Self-join the temporary table to get all the totals
select
ij.iItemId,
ij.jItemId,
i.TotalUsers as Users_I,
j.TotalUsers as Users_J,
ij.TotalUsers as Users_I_and_J,
(i.TotalUsers - ij.TotalUsers) as Users_I_no_J,
(j.TotalUsers - ij.TotalUsers) as Users_J_no_I
from TempTotals ij
left join TempTotals i on (i.iItemId = ij.iItemId and i.iItemId = i.jItemId)
left join TempTotals j on (j.jItemId = ij.jItemId and j.iItemId = j.jItemId)

Answer 3

If you're using Oracle Database, you can compare nested tables (collections) with the multiset operators. And get the number of elements in a collection with cardinality.
So what you can do is:
Group by itemid, collecting all the users into a nested table
Cross join the output of this with itself
Use the multiset intersect/except operators to get number of elements in the sets as needed
Which looks a little like:
create table t (
ItemId int, UserId varchar2(10)
);
insert into t values ( 200 , 'user1');
insert into t values ( 200 , 'user3');
insert into t values ( 200 , 'user4');
insert into t values ( 300 , 'user5');
insert into t values ( 300 , 'user3');
commit;
create or replace type users_t as table of varchar2(10);
/
with grps as (
select itemid, cast ( collect ( userid ) as users_t ) users
from t
group by itemid
)
select g1.itemid i, g2.itemid j,
cardinality ( g1.users ) num_i,
cardinality ( g2.users ) num_j,
cardinality ( g1.users multiset intersect g2.users ) i_and_j,
cardinality ( g1.users multiset except g2.users ) i_not_j,
cardinality ( g2.users multiset except g1.users ) j_not_i
from grps g1
cross join grps g2;
I J NUM_I NUM_J I_AND_J I_NOT_J J_NOT_I
200 200 3 3 3 0 0
200 300 3 2 1 2 1
300 200 2 3 1 1 2
300 300 2 2 2 0 0
If necessary, you may be able to get more performance by skipping the except operators when i = j, e.g.:
case
when g1.itemid = g2.itemid then 0
else cardinality ( g1.users multiset intersect g2.users )
end