Athena: compare columns consisting array<strings> in two different tables - sql

Question

I have 2 external tables (parquet files in S3) in Athena, each of them has a column which is array of strings. One of the tables is a subset and I need to compare these array values with the other table having the superset array. I believe the problem would be clearer with the below illustration. Both tables do not have any duplicate records.
Table 1 (Sample Subset table)
+---+-----------+---------------------------+
|no | prod_name | article_list |
+---+-----------+---------------------------+
| 1 |sofa | ['ABC','PQR'] |
| 2 |cupboard | ['LMN','DEF','XYZ'] |
| 3 |table | ['DEF'] |
| 4 |chair | ['DEF','PQR','ABC'] |
| 5 |dresser | ['LMN','IJK','WXY','STU'] |
+---+--------------------+------------------+
Table 2 (Sample Superset table)
+---+---------+--------------+---------------------------------------------------+
|no | wh_code | restock_date | article_list |
+---+---------+--------------+---------------------------------------------------+
| 1 |WH0001 | 2020-01-12 | ['ABC','BCE','CDE','DEF','JKL','PQR','QRS','STU'] |
| 2 |WH0001 | 2020-04-15 | ['ABC','CDE','DEF','IJK','LMN','PQR','STU','XYZ'] |
| 3 |WH0002 | 2021-03-17 | ['BCE','DEF','IJK','LMN','PQR','RST','STU','WXY'] |
| 4 |WH0003 | 2021-08-20 | ['ABC','IJK','LMN','NOP','PQR','RST','STU','WXY'] |
| 5 |WH0003 | 2022-03-26 | ['DEF','IJK','LMN','NOP','PQR','RST','STU','XYZ'] |
+---+---------+--------------+---------------------------------------------------+
Required result
+------------------------+---------+-----------------+
|article_list (table 1) | wh_code | restock_date |
+---+--------------------+---------------------------+
| ['ABC','PQR'] | WH0001 | 2020-01-12 |
| ['ABC','PQR'] | WH0001 | 2020-04-15 |
| ['ABC','PQR'] | WH0003 | 2021-08-20 |
| ['LMN','DEF','XYZ'] | WH0001 | 2020-04-15 |
| ['LMN','DEF','XYZ'] | WH0003 | 2021-08-20 |
| ['DEF'] | WH0001 | 2020-01-12 |
| ['DEF'] | WH0001 | 2020-04-15 |
| ['DEF'] | WH0002 | 2021-03-17 |
| ['DEF'] | WH0003 | 2022-03-26 |
| . | . | . |
| . | . | . |
| . | . | . |
+------------------------+---------+-----------------+
The following query in Athena works to find a particular combination (['ABC', 'PQR']) in table 2 consisting of the superset array. It results in the first 3 rows of the required result.
SELECT ['ABC', 'PQR'] as article_list,
wh_code,
restock_date
FROM "table_2"
WHERE filter(ARRAY ['ABC', 'PQR'], x -> NOT CONTAINS(article_list, x)) = ARRAY[]
group by wh_code, restock_date
Request help to write a generic query (considering all the combinations from table 1) to get the desired result

Answer 1

Join the two table on the required condition. Also it seems that you should consider using array_except to simplify the query (also I use cardinality to count number of elements):
-- sample data
with table1(no, prod_name, article_list ) as (
values ( 1, 'sofa', array['ABC','PQR']),
( 2, 'cupboard', array['LMN','DEF','XYZ'] )
),
table2 (no, wh_code, restock_date, article_list) as (
values (1, 'WH0001', date '2020-01-12', array['ABC','BCE','CDE','DEF','JKL','PQR','QRS','STU']),
(2, 'WH0001', date '2020-04-15', array['ABC','CDE','DEF','IJK','LMN','PQR','STU','XYZ']),
(3, 'WH0002', date '2021-03-17', array['BCE','DEF','IJK','LMN','PQR','RST','STU','WXY']),
(4, 'WH0003', date '2021-08-20', array['ABC','IJK','LMN','NOP','PQR','RST','STU','WXY'])
)
-- query
select t1.article_list, t2.wh_code, t2.restock_date
from table1 t1
join table2 t2 on cardinality(array_except(t1.article_list, t2.article_list)) = 0;
Output:
article_list
wh_code
restock_date
[ABC, PQR]
WH0001
2020-01-12
[ABC, PQR]
WH0001
2020-04-15
[ABC, PQR]
WH0003
2021-08-20
[LMN, DEF, XYZ]
WH0001
2020-04-15
UPD
Try next one, but taking in account size of the data maybe you will need to partition the queries:
-- query
select arbitrary(article_list), wh_code, restock_date
from (select no, article_list, article
from table1, unnest (article_list) as t(article)) as t1
join (select no, wh_code, restock_date, article
from table2, unnest (article_list) as t(article)) as t2 on t1.article = t2.article
group by t1.no, wh_code, restock_date
having count(t1.article) = cardinality(arbitrary(article_list));