How to delete duplicates using row_number() without listing all the columns from the table. I have a hive table with 50+ columns. If i want to delete duplicates based on a 2 columns below are the steps i followed
Create temp table as Create temptable as select * from (select
*,row_number() over(col1,col2) as rn from maintable) where rn=1)
Insert overwrite table maintable select * from temptable
But here in insert it fails because the new column rn is present in temptable; To avoid this column i would have to list all the rest of the columns.
And there is no Drop column option in hive. There also you need to use REPLACE function which again needs listing all the rest of the columns.
So any better idea for deleting duplicates in Hive based on 2 columns?
Spell out all column names from the original table for insert overwrite as the query computes a new column. No temp table is needed for this.
Insert overwrite table maintable
select col1,col2,col3 ---...col50
from (select t.*
,row_number() over(order by col1,col2) as rn
from maintable
) t
where rn = 1
Related
I'm trying to deduplicate an ingestion-time partitioned table in BigQuery:
MERGE dataset.table_name targ
USING (
SELECT * EXCEPT(row_number)
FROM (
SELECT
*,
ROW_NUMBER() OVER (PARTITION BY uid ORDER BY _PARTITIONDATE DESC) row_number
FROM dataset.table_name
)
WHERE row_number = 1
) src
ON FALSE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED THEN INSERT ROW
Getting the following error:
Omitting INSERT target column list is unsupported for ingestion-time partitioned table dataset.table_name
CREATE OR REPLACE TABLE can't create partitioned tables.
Creating a partitioned table using DDL and then inserting requires defining schema inside the query, which I'm trying to avoid.
I'm looking for a simple universal query that I can apply to different tables with minimal adjustment. Like the one above.
You need to specify the columns in the INSERT statement. You can use the following query to get the column names:
MERGE dataset.table_name targ
USING (
SELECT * EXCEPT(row_number)
FROM (
SELECT
*,
_PARTITIONTIME,
ROW_NUMBER() OVER (PARTITION BY uid ORDER BY _PARTITIONTIME DESC) row_number
FROM dataset.table_name
)
WHERE row_number = 1
) src
ON FALSE
WHEN NOT MATCHED BY SOURCE THEN DELETE
WHEN NOT MATCHED THEN INSERT (uid, _PARTITIONTIME, column1, column2, column3)
VALUES (uid, _PARTITIONTIME, column1, column2, column3)
But it's not universal.
Is there a possible way to delete duplicate rows from a table like this without dropping the table and re-creating it with distinct rows?
DROP TABLE IF EXISTS #temp;
CREATE TABLE #temp (id INT);
INSERT INTO #temp
VALUES (1), (2), (2), (2), (3), (3);
Using a CTE and ROW_NUMBER grouped by the specified column you are looking at can cleanly remove duplicate values, as ROW_NUMBER counts incrementally with duplicate values of the PARTITION BY aggregation. When a new grouping is found when ordered, it resets the ROW_NUMBER to 1 starting at that next record of different grouped values.
WITH CTE AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY col1 ORDER BY col1) AS RN
FROM #temp
)
DELETE FROM CTE WHERE RN<>1
Here is your answer, First we have to find all recodes
DELETE t
FROM #temp t
WHERE (SELECT count(1)
FROM #temp t1
WHERE t.id =t1.id )
>1
Hope you will find the best solutions.
I have created a table tbl_Dist with Column names District and DistCode, there were many duplicate values in the District table so i have removed all the duplicates value using this statement:
select distinct District from tbl_Dist;
its done, but i am not getting how to update the results of the above executed query to the table tbl_Dist?
You can as the below:
-- Move temp table
SELECT DISTINCT District INTO TmpTable FROM tbl_Dist
-- Delete all data
DELETE FROM tbl_Dist
-- Insert data from temp table
INSERT INTO tbl_Dist
SELECT * FROM TmpTable
Updated
Firstly, run this query. You will have a temp table with distinct data of main table (tbl_Dist)
-- Move temp table
SELECT DISTINCT District INTO TmpTable FROM tbl_Dist
Then run the below query to delete all data
DELETE FROM tbl_Dist
Finally, run the below query to insert all distinct data to main table.
-- Insert data from temp table
INSERT INTO tbl_Dist
SELECT * FROM TmpTable
You need Delete not Update
;with cte as
(
Select row_number() over(partition by District order by (select null)) as rn,*
From yourtable
)
Delete from cte where Rn > 1
To check the records that will be deleted use this.
;with cte as
(
Select row_number() over(partition by District order by (select null)) as rn,*
From yourtable
)
Select * from cte where Rn > 1
If you want to keep this query you can keep it in a view the write an update query through that view.The table will be updated
try this script
DELETE FROM tbl_Dist
WHERE District = District
AND DistCode > DistCode
I am trying to delete some duplicate data in my redshift table.
Below is my query:-
With duplicates
As
(Select *, ROW_NUMBER() Over (PARTITION by record_indicator Order by record_indicator) as Duplicate From table_name)
delete from duplicates
Where Duplicate > 1 ;
This query is giving me an error.
Amazon Invalid operation: syntax error at or near "delete";
Not sure what the issue is as the syntax for with clause seems to be correct.
Has anybody faced this situation before?
Redshift being what it is (no enforced uniqueness for any column), Ziggy's 3rd option is probably best. Once we decide to go the temp table route it is more efficient to swap things out whole. Deletes and inserts are expensive in Redshift.
begin;
create table table_name_new as select distinct * from table_name;
alter table table_name rename to table_name_old;
alter table table_name_new rename to table_name;
drop table table_name_old;
commit;
If space isn't an issue you can keep the old table around for a while and use the other methods described here to validate that the row count in the original accounting for duplicates matches the row count in the new.
If you're doing constant loads to such a table you'll want to pause that process while this is going on.
If the number of duplicates is a small percentage of a large table, you might want to try copying distinct records of the duplicates to a temp table, then delete all records from the original that join with the temp. Then append the temp table back to the original. Make sure you vacuum the original table after (which you should be doing for large tables on a schedule anyway).
If you're dealing with a lot of data it's not always possible or smart to recreate the whole table. It may be easier to locate, delete those rows:
-- First identify all the rows that are duplicate
CREATE TEMP TABLE duplicate_saleids AS
SELECT saleid
FROM sales
WHERE saledateid BETWEEN 2224 AND 2231
GROUP BY saleid
HAVING COUNT(*) > 1;
-- Extract one copy of all the duplicate rows
CREATE TEMP TABLE new_sales(LIKE sales);
INSERT INTO new_sales
SELECT DISTINCT *
FROM sales
WHERE saledateid BETWEEN 2224 AND 2231
AND saleid IN(
SELECT saleid
FROM duplicate_saleids
);
-- Remove all rows that were duplicated (all copies).
DELETE FROM sales
WHERE saledateid BETWEEN 2224 AND 2231
AND saleid IN(
SELECT saleid
FROM duplicate_saleids
);
-- Insert back in the single copies
INSERT INTO sales
SELECT *
FROM new_sales;
-- Cleanup
DROP TABLE duplicate_saleids;
DROP TABLE new_sales;
COMMIT;
Full article: https://elliot.land/post/removing-duplicate-data-in-redshift
That should have worked. Alternative you can do:
With
duplicates As (
Select *, ROW_NUMBER() Over (PARTITION by record_indicator
Order by record_indicator) as Duplicate
From table_name)
delete from table_name
where id in (select id from duplicates Where Duplicate > 1);
or
delete from table_name
where id in (
select id
from (
Select id, ROW_NUMBER() Over (PARTITION by record_indicator
Order by record_indicator) as Duplicate
From table_name) x
Where Duplicate > 1);
If you have no primary key, you can do the following:
BEGIN;
CREATE TEMP TABLE mydups ON COMMIT DROP AS
SELECT DISTINCT ON (record_indicator) *
FROM table_name
ORDER BY record_indicator --, other_optional_priority_field DESC
;
DELETE FROM table_name
WHERE record_indicator IN (
SELECT record_indicator FROM mydups);
INSERT INTO table_name SELECT * FROM mydups;
COMMIT;
This method will preserve permissions and the table definition of the original_table.
The most upvoted answer does not preserve permissions on the table or the original definition of the table.
In real world production environment this method is how you should be doing as this is safest and easiest way to execute in production environment.
This will have a DOWN TIME in PROD.
Create Table with unique rows
CREATE TABLE unique_table as
(
SELECT DISTINCT * FROM original_table
)
;
Backup the original_table
CREATE TABLE backup_table as
(
SELECT * FROM original_table
)
;
Truncate the original_table
TRUNCATE original_table;
Insert records from unique_table into original_table
INSERT INTO original_table
(
SELECT * FROM unique_table
)
;
To avoid DOWN TIME run the below queries in a TRANSACTION and instead of TRUNCATE use DELETE
BEGIN transaction;
CREATE TABLE unique_table as
(
SELECT DISTINCT * FROM original_table
)
;
CREATE TABLE backup_table as
(
SELECT * FROM original_table
)
;
DELETE FROM original_table;
INSERT INTO original_table
(
SELECT * FROM unique_table
)
;
END transaction;
Simple answer to this question:
Firstly create a temporary table from the main table where value of row_number=1.
Secondly delete all the rows from the main table on which we had duplicates.
Then insert the values of temporary table into the main table.
Queries:
Temporary table
select id,date into #temp_a
from
(select *
from (select a.*,
row_number() over(partition by id order by etl_createdon desc) as rn
from table a
where a.id between 59 and 75 and a.date = '2018-05-24')
where rn =1)a
deleting all the rows from the main table.
delete from table a
where a.id between 59 and 75 and a.date = '2018-05-24'
inserting all values from temp table to main table
insert into table a select * from #temp_a.
The following deletes all records in 'tablename' that have a duplicate, it will not deduplicate the table:
DELETE FROM tablename
WHERE id IN (
SELECT id
FROM (
SELECT id,
ROW_NUMBER() OVER (partition BY column1, column2, column3 ORDER BY id) AS rnum
FROM tablename
) t
WHERE t.rnum > 1);
Postgres administrative snippets
Your query does not work because Redshift does not allow DELETE after the WITH clause. Only SELECT and UPDATE and a few others are allowed (see WITH clause)
Solution (in my situation):
I did have an id column on my table events that contained duplicate rows and uniquely identifies the record. This column id is the same as your record_indicator.
Unfortunately I was unable to create a temporary table because I ran into the following error using SELECT DISTINCT:
ERROR: Intermediate result row exceeds database block size
But this worked like a charm:
CREATE TABLE temp as (
SELECT *,ROW_NUMBER() OVER (PARTITION BY id ORDER BY id) AS rownumber
FROM events
);
resulting in the temp table:
id | rownumber | ...
----------------
1 | 1 | ...
1 | 2 | ...
2 | 1 | ...
2 | 2 | ...
Now the duplicates can be deleted by removing the rows having rownumber larger than 1:
DELETE FROM temp WHERE rownumber > 1
After that rename the tables and your done.
with duplicates as
(
select a.*, row_number (over (partition by first_name, last_name, email order by first_name, last_name, email) as rn from contacts a
)
delete from contacts
where contact_id in (
select contact_id from duplicates where rn >1
)
I've this script:
Create table #temp (id int,name varchar(10),city varchar(10),sal int)
Insert into #temp
Select 2,'kishor','hyd', 100
Union all
Select 3,'kumar','sec', 200
Union all
Select 4,'santosh','kp', 300
Union all
Select 1,'sudeep','myp', 300
now I want to generate row number as same as data inserted without using a create or insert or CTE or Update commands, using a single select statement.
So that even after sorting by any order the row number column should not change its values
You should add auto-increment field to your table to save information about inserting order.
And then use for example ROW_NUMBER() to get a rownumber:
select #temp.*,
ROW_NUMBER() over (order by <your autoincrement field here> ) as RowNumber
from #temp