Remove duplicates after joining three different tables - sql

Question

I'm working now on SQL query for Polarion widget, where I want get items with unresolved comments.
Unfortunately, I have a problem with joining three tables where one has optional empty/null.
In Polarion is a few different tables to contain comments, most of them has a resolved status. I'm using WORKITEM, COMMENT, MODULECOMMENT tables, because sometimes workitem has comment here, sometimes in another table.
Difference is one: when workitem has "normal" comment, then there is always FK key in COMMENT table, but not each workitem has a FK MODULE key (table structure).
In default widget, I can use SQL query with a one big limit:
I can't replace whole select and from, because there is a static line: SELECT WORKITEM.C_URI and FROM WORKITEM, so I can only add something (I tried DISTINCT ON, but how can you see, I can't replace that line)
Polarion has another problem too, each document has headings, texts etc. and each of this element is separate workitem with the same FK_URI_MODULE (and documents use MODULECOMMENT table).
I want remove duplicates from MODULECOMMENT (ignore situation where workitem.FK_URI_MODULE is empty/null, because it's normal state).
The best what I created, It's here (in this situation I replace is null to is not null, because It's return less items):
select workitem.c_uri, workitem.FK_URI_MODULE
from workitem
left join COMMENT on COMMENT.FK_URI_WORKITEM = WORKITEM.C_URI
left join MODULECOMMENT on MODULECOMMENT.FK_URI_MODULE = WORKITEM.FK_URI_MODULE
where true
and workitem.fk_uri_project = 231
and (comment.c_resolved is not null
or modulecomment.c_resolved is not null);
and my results:
c_uri | fk_uri_module
--------+---------------
7952 | 7940
7949 | 7940
7953 | 7940
7964 | 7940
124141 | 124138
609 |
1347 |
609 |
but I want get something like that:
c_uri | fk_uri_module
--------+---------------
7952 | 7940
124141 | 124138
609 |
1347 |
609 |

Answer 1

Don't generate the duplicates to begin with by not joining, but using an EXISTS condition:
select workitem.c_uri, workitem.fk_uri_module
from workitem
where exists (select *
from comment
left join modulecomment on modulecomment.fk_uri_module = workitem.fk_uri_module
where comment.fk_uri_workitem = workitem.c_uri
and (comment.c_resolved is not null
or modulecomment.c_resolved is not null))
where workitem.fk_uri_project = 231

Answer 2

You can use DISTINCT ON for this, which returns only the first record of a group:
demos:db<>fiddle
select DISTINCT ON (workitem.FK_URI_MODULE)
workitem.c_uri, workitem.FK_URI_MODULE
from workitem
left join COMMENT on COMMENT.FK_URI_WORKITEM = WORKITEM.C_URI
left join MODULECOMMENT on MODULECOMMENT.FK_URI_MODULE = WORKITEM.FK_URI_MODULE
where true
and workitem.fk_uri_project = 231
and (comment.c_resolved is not null
or modulecomment.c_resolved is not null)
The interesting part is the NULL records. The query above returns only one of the NULL records, because DISTINCT ON also removes duplicates for NULL.
So, if you want to keep all NULL values, we need to treat the NULL records differently:
SELECT DISTINCT ON (fk_uri_module) -- 1
*
FROM <your_query>
WHERE fk_uri_module IS NOT NULL
UNION ALL
SELECT -- 2
*
FROM <your_query>
WHERE fk_uri_module IS NULL
Apply DISTINCT ON on all records with non-NULL values
UNION ALL records with NULL afterwards
To avoid executing your entire query twice, you can move them into a CTE (WITH clause) and reference it later:
WITH my_result AS (
-- <your_query>
)
SELECT DISTINCT ON (fk_uri_module)
*
FROM my_result
WHERE fk_uri_module IS NOT NULL
UNION ALL
SELECT
*
FROM my_result
WHERE fk_uri_module IS NULL
Edit:
demo:db<>fiddle
If you cannot use DISTINCT ON, a valid alternative for this is the row_number() window function:
SELECT
*
FROM (
select
workitem.c_uri, workitem.FK_URI_MODULE,
row_number() OVER (PARTITION BY workitem.FK_URI_MODULE)
from workitem
left join COMMENT on COMMENT.FK_URI_WORKITEM = WORKITEM.C_URI
left join MODULECOMMENT on MODULECOMMENT.FK_URI_MODULE = WORKITEM.FK_URI_MODULE
where true
and workitem.fk_uri_project = 231
and (comment.c_resolved is not null
or modulecomment.c_resolved is not null)
) s
WHERE row_number = 1