SQL Duplicate With Nulls - sql

Question

For somewhat insane business reasons related to getting rather messy data from our customer, I have the following problem;
1)I have a table with 6 semi-unique identifiers and one automatically incrementing unique ID. The table has more fields. But, those aren't important to this discussion. Nor is the exact type of data the fields hold.
2)I want to get a list of the unique IDs of all rows that participate in at least one duplicate relationship. (There's not any additional value in identifying all of the row pairs that indicate a duplication. But, if a solution provides that, it's fairly trivial to retrieve the set of duplicate rows. So, that would also be fine)
3)A duplicate is defined as;
3a)For each of these 6 fields, record A must either match record B or one of them must be null
3b)At least one field must match exactly (i.e. neither is null)
4)All of the potential duplicate fields of interest are strings and are not empty strings. Many rows have at least one of the fields of interest as null but (at least assuming our ingest logic is working) none of them can have more than 3 of these fields as null.
5)Exact string content matching is fine. We don't need any regex-based, case-insensitive... sort of matching.
6)Actual duplicates in the table are fairly rare.
7)We are running PostgreSQL 9. Using database-specific functionality is acceptable.
8)The table has 500,000 rows. So, the naive query I started out with, provided below, takes far too long to be viable. Presumably, it operates principally in exponential time. Ideally, the results should return in less than a minute, running on a midrange server.
SELECT a.id
FROM myTable a
JOIN myTable b ON a.id < b.id
AND (a.field1 = b.field1 OR a.field1 IS NULL OR b.field1 IS NULL )
AND (a.field2 = b.field2 OR a.field2 IS NULL OR b.field2 IS NULL)
....
WHERE
a.field1 = b.field1 OR a.field2 = b.field2 ...
9)I also looked into using "group by". But, "group by" does not consider two rows to be equal if a grouped column in one contains null and the other contains a value. Unless there is a way to achieve that behavior, group by won't work for my "both equal or at least one is null" logic.
10)The set of values that might be expected to appear in each row can be assumed non-overlapping with other columns. i.e., other than null, you will not expect a value from field 1 to appear in any rows for field 2.
Update: Sorry for the lack of information. I'll provide as close an approximation of the table schema as I can. Unfortunately, the project in question is in defense and even just the field names of the table could reveal information about operational security.
CREATE TABLE a (
id serial NOT NULL PRIMARY KEY,
f1 character varying,
f2 character varying,
f3 character varying,
f4 character varying,
f5 character varying,
f6 character varying,
...Other columns that aren't really relevant
)
CREATE INDEX f1_idx
ON public.a
USING btree
(f1 COLLATE pg_catalog."default");
...Same index for the other 5 fields.
For ease of reference, I'll copy Lorenze Albe's question and answer it here.
If you have the three rows
(1, 2, 3, 4, NULL, 6)
(1, 2, 3, NULL, 5, NULL)
(1, 2, 3, 4, 7, NULL)
which are duplicates?
(1, 2, 3, NULL, 5, NULL)
and
(1, 2, 3, 4, 7, NULL)
are not duplicates because field 5 is non-null in both and they are not equal. The other two are duplicates.
I'll give a few more examples of my own for clarity. (Just for completeness, I'll provide my row examples as strings. But, like I said, their string-iness isn't really important because we require exact string matches.
("1", "2", "3", "4", NULL, NULL)
AND
("1","2","3",NULL,"9",NULL)
are duplicates because columns 4, 5, and 6 are null in at least one and all other fields are equal.
("1", "2", "3", "4", NULL, "6")
AND
("1","2","3",NULL,"9","7")
are not duplicates because field 6 differs and neither is null
And two examples more typical of the actual data;
(NULL, NULL, "3", NULL, "5", "6")
and
("1", "2", NULL, "4", NULL, "6")
are duplicates because all fields wherein they differ, at least one side is null.
(NULL, NULL, "3", NULL, "5", "6")
and
("1", "2", NULL, "4", NULL, "6")
Yes, that does mean that
(NULL, NULL, NULL, "4", "5", "6")
and
("1", "2", "3", NULL, NULL, NULL)
would be duplicates if not for the requirement that at least one field match exactly. Which fields are null and which aren't is very nearly random. All that we require from our data provider is that at least 2 of the 6 fields must be provided.
Another Update: I've updated point 2 to reflect the fact that I want all rows that participate in at least one duplicate pair. So, for the three rows
(1, 2, 3, 4, NULL, 6)
(1, 2, 3, NULL, 5, NULL)
(1, 2, 3, 4, 7, NULL)
all three would be returned because even though rows 2 and 3 would not be considered duplicates of each other, row pairs 1,2 are duplicates and 1,3 are duplicates and therefore all three participate in a duplicate relationship and therefore would be returned.

Answer 1

Use count() over(partition by ...) then filter that result for any counts greater than 1:
CREATE TABLE mytable(
ID INTEGER NOT NULL PRIMARY KEY
,col1 VARCHAR(2) NOT NULL
,col2 VARCHAR(2)
,col3 VARCHAR(2) NOT NULL
,col4 VARCHAR(2)
,col5 VARCHAR(2)
,col6 VARCHAR(2)
);
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1001,'a1','b1','c1','d1','e1','f1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1002,'a1',NULL,'c1','d1','e1','f1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1003,'a1','b1','c1','d1','e1','f1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1004,'b1','c1','d1','e1','f1',NULL);
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1005,'a1','b1','c1',NULL,'e1','f1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1006,'b1','c1','d1','e1','f1',NULL);
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1007,'f1',NULL,'b1','c1','d1','e1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1008,'b1','c1','d1','e1','f1',NULL);
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1009,'c1','d1','e1','f1',NULL,NULL);
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1010,'c1','d1','e1','f1',NULL,'a1');
INSERT INTO mytable(ID,col1,col2,col3,col4,col5,col6) VALUES (1011,'a1','b1','c1','d1','e1','f1');
select
*
, count(*) over(partition by
coalesce(col1,'NULL')
, coalesce(col2,'NULL')
, coalesce(col3,'NULL')
, coalesce(col4,'NULL')
, coalesce(col5,'NULL')
, coalesce(col6,'NULL')
) cv
from mytable
id | col1 | col2 | col3 | col4 | col5 | col6 | cv
---: | :--- | :--- | :--- | :--- | :--- | :--- | -:
1001 | a1 | b1 | c1 | d1 | e1 | f1 | 3
1003 | a1 | b1 | c1 | d1 | e1 | f1 | 3
1011 | a1 | b1 | c1 | d1 | e1 | f1 | 3
1005 | a1 | b1 | c1 | null | e1 | f1 | 1
1002 | a1 | null | c1 | d1 | e1 | f1 | 1
1008 | b1 | c1 | d1 | e1 | f1 | null | 3
1004 | b1 | c1 | d1 | e1 | f1 | null | 3
1006 | b1 | c1 | d1 | e1 | f1 | null | 3
1010 | c1 | d1 | e1 | f1 | null | a1 | 1
1009 | c1 | d1 | e1 | f1 | null | null | 1
1007 | f1 | null | b1 | c1 | d1 | e1 | 1
Use the approach above as a subquery, and then use where cv > 1 to locate all rows that have "duplicates" in those 6 columns.
db<>fiddle here
Please note the power of having some sample data to work with. Really it is your responsibility to provide sample data with your question (as you already own that data anyway). Do NOT try to explain in words alone, use data to illustrate the "as is" and the "to be", you will find your questions easier to prepare and faster to answer. See Why should I provide a MCVE