Removing true duplicates from greenplum table - sql

Question

I am trying to remove true duplicates from a table. I have removed dupes multiple times in past but I'm not able to figure what's wrong with my syntax with this one.
My code -
DELETE
FROM my_table_name
WHERE (
column1, column2, column3, column4, column5, column6, column7, column8, column9) IN
(
SELECT Row_number() OVER( partition BY column1, column2,column3, column4,column5,column6,column7,column8 ORDER BY column2 DESC, column3 ASC ) AS row_num,
column1,
column2,
column3,
column4,
column5,
column6,
column7,
column8,
column9
FROM my_table_name
WHERE column1='some_value') a
WHERE row_num=2;
Error
********** Error **********
ERROR: syntax error at or near ""a""
SQL state: 42601
Character: 1607
I can see that the error is on creating the alias a subquery. But I'm not able to pin point what's wrong here.
Any help is appreciated
Edit 1 -
If I remove a, I get the below error
********** Error **********
ERROR: syntax error at or near "where"
SQL state: 42601
Character: 1608

Answer 1

If you have duplicate rows, you can't just delete all but one of the records in a single command. You have to delete all duplicates and then insert just one version for each duplicate row or build new table (preferred) without duplicates.
Let's start with the preferred method which is to create a new table without the duplicates. This solution utilizes disk space in the most efficient way possible rather than having a fragmented table.
Example:
create table foo
(id int, fname text)
with (appendonly=true)
distributed by (id);
Insert some data with duplicates:
insert into foo values (1, 'jon');
insert into foo values (1, 'jon');
insert into foo values (2, 'bill');
insert into foo values (2, 'bill');
insert into foo values (3, 'sue');
insert into foo values (4, 'ted');
insert into foo values (4, 'ted');
insert into foo values (4, 'ted');
insert into foo values (4, 'ted');
Create a new version of the table without the duplicates:
create table foo_new with (appendonly=true) as
select id, fname
from (
select row_number() over (partition by id) as row_num, id, fname
from foo
) as sub
where sub.row_num = 1
distributed by (id);
And now rename the tables:
alter table foo rename to foo_old;
alter table foo_new rename to foo;
The second method is to use DELETE but you'll see that it needs more steps to complete.
First, create a temp table with the IDs you want to delete. You typically don't have primary keys enforced in Greenplum but you still have a logical PK. Columns like customer_id, product_id, etc are all in your data. So, find the dups first based on the PK.
drop table if exists foo_pk_delete;
create temporary table foo_pk_delete with (appendonly=true) as
select id
from foo
group by id
having count(*) > 1
distributed by (id);
Next, get the entire row for each duplicate but only one version of it.
drop table if exists foo_dedup;
create temporary table foo_dedup with (appendonly=true) as
select id, fname
from (
select row_number() over (partition by f.id) as row_num, f.id, f.fname
from foo f
join foo_pk_delete fd on f.id = fd.id
) as sub
where sub.row_num = 1
distributed by (id);
Now you can delete the duplicates:
delete
from foo f
using foo_pk_delete fk
where f.id = fk.id;
And then you can insert the deduplicated data back into the table.
insert into foo (id, fname)
select id, fname from foo_dedup;
You'll want to vacuum your table after this data manipulation.
vacuum foo;