how to avoid duplicates in hive query - sql

Question

I have two tables:
table1
the_date | my_id |
02/03/2021,123
02/03/2021, 1234
02/03/2021, 12345
table2
the_date | my_id |seq | txt
02/03/2021, 1234, 1 , 'OK'
02/03/2021, 12345, 1, 'OK'
02/03/2021, 12345, 2, 'HELLO HI THERE'
02/03/2021, 123456, 1, 'Ok'
Here is my code:
WITH AB AS (
SELECT A1.my_id
FROM DB1.table1 A1 , DB1.MSG_REC A2 WHERE
A1.my_id=A2.my_id
),
BC AS (
SELECT AB.the_date
COUNT ( DISTINCT (CASE WHEN (TXT like '%OK%') THEN AB.my_id ELSE NULL END )) AS
CASE1 ,
COUNT ( DISTINCT (CASE WHEN (TXT like '%HELLO HI THERE%') THEN AB.my_id ELSE NULL END )) AS
CASE2
FROM AB left JOIN DB1.my_id BC ON AB.my_id =BC.my_id
The issue that stems from above is I am looping over the value '12345' twice because it satisfies both of the case statements.
That causes data duplicates when capturing metrics of the counts. Is there a way to execute the first case and then perform the second case but exclude looping any of the "my_id' records from the first case.
So for example, when it is time to run the above script and the first case executes, it will pick up the below records and the count would be 3
02/03/2021, 1234, 1 , 'OK'
02/03/2021, 12345, 1, 'OK'
02/03/2021, 123456, 1, 'Ok
The second case should only be looping through the below records and the count would be only 1
02/03/2021, 12345, 2, 'HELLO HI THERE'
CASE1 would be 4 and CASE2 would by 2 if I don't create a condition to circumvent this issue. Any tips or suggestions?

Answer 1

Assign case to each your ID before DISTINCT aggregation . After that do distinct aggregation, in such way you will eliminate same IDs counted in different cases. See comments in the code:
select --do final distinct aggregation
count(distinct (case when assigned_case='CASE1' then my_id else null end ) ) as CASE1,
count(distinct (case when assigned_case='CASE2' then my_id else null end ) ) as CASE2
from
(
select my_id
--assign single CASE to all rows with the same id based on some logic:
case when case1_flag = 1 then 'CASE1'
when case1_flag = 1 then 'CASE2'
else NULL
end as assigned_case
from
(--calculate all CASE flags for each ID
select AB.my_id,
max(CASE WHEN (TXT like '%OK%') THEN 1 ELSE NULL END) over (partition by AB.my_id) as case1_flag
max(CASE WHEN (TXT like '%HELLO HI THERE%') THEN 1 ELSE NULL END) over (partition by AB.my_id) as case2_flag
from ...
) s
) s