I want to delete duplicates from a Redshift table that are true duplicates. Below is an example of two rows that are true duplicates.
Since it is Redshift, there are no primary keys to the table. Any help is appreciated.
id
Col 1
Col 2
1
Val 1
Val 2
1
Val 1
Val 2
I tried using window functions row_number(), rank(). Neither worked as when applying Delete command, SQL command cannot differentiate both rows.
Trial 1:
The below command deletes both rows
DELETE From test_table
where (id) IN
(
select \*,row_number() over(partition by id) as rownumber from test*table where row*number !=1
);
Trial 2:
The below command retains both rows.
DELETE From test_table
where (id) IN
(
select \*,rank() over(partition by id) as rownumber from test*table where row*number !=1
);
All row values are identical. Hence you unable to delete specific rows in that table.
In that I would recommend to create dummy table, and load unique records.
Steps to follow:
create table dummy as select * from main_table where 1=2
insert into dummy(col1,col2..coln) select distinct col1,col2..coln from main_table;
verify dummy table.
Alter table main_table rename to main_table_bk
alter table dummy rename to main.
after complete your testing and verification, drop main_table_bk
Hope it will help.
You cannot delete one without deleting the other as they are identical. The way to do this is to:
make a temp table with (one copy) of each duplicate row
(within a transaction) delete all rows from the source table that match rows in the temp table
Insert temp table rows into source table (commit)
I'm trying to create a temporary table in Hive as follows:
CREATE TEMPORARY TABLE mydb.tmp2
AS SELECT * FROM (VALUES (0, 'abc'))
AS T (id , mystr);
But that gives me the following error:
SemanticException [Error 10296]: Values clause with table constructor not yet supported
Is there another way to create a temporary table by explicitly and directly providing the values in the same command?
My ultimate goal is to run a MERGE command, and the temporary table would be inserted after the USING command. So something like this:
MERGE INTO mydb.mytbl
USING <temporary table>
...
Use subquery instead of temporary table:
MERGE INTO mydb.mytbl t
USING (SELECT 0 as id, 'abc' as mystr) tmp on tmp.id = t.id
Hive does not support values constructor yet. You can achieve this using below query:
CREATE TEMPORARY TABLE mydb.tmp2
AS SELECT 0 as id, 'abc' as mystr;
For merge, you can use temporary table as below:
merge into target_table
using ( select * from mydb.tmp2) temp
on temp.id = target_table.id
when matched then update set ...
when not matched then insert values (...);
I have created a backup for my country table.
create table country_bkp as select * from country;
What SQL should I use to restore the country table to it's original state?
I can do
insert into country select * from country_bkp;
but it will just have duplicate entries and probably fail as primary key would be same .
Is there an SQL command to merge data back?
Last alternative would be
DROP TABLE country;
create table country as select * from country_bkp;
but I want to avoid this as all the grants/permissions would get lost by this.
Other cleaner way would be
delete from country ;
insert into country select * from country_bkp;
But I am looking for more of a merge approach without having to clear data from original table.
Instead of dropping the table, which, as you noted, would lose all the permission defitions, you could truncate it to just remove all the data, and then insert-select the old data:
TRUNCATE TABLE country;
INSERT INTO country SELECT * FROM county_bkp;
In my case, INSERT INTO country SELECT * FROM county_bkp; didnt work because:
It wouldnt let me insert in Primary Key column due to
indentity_insert being off by default.
My table had TimeStamp columns.
In that case:
allow identity_insert in the OriginalTable
insert query in which you mention all the columns of OriginalTable (Excluding TimeStamp Columns) and in Values select all columns from BackupTable (Excluding TimeStamp Columns)
restrict identity_insert in the OriginalTable at the end.
EXAMPLE:
Set Identity_insert OriginalTable ON
insert into OriginalTable (a,b,c,d,e, ....) --[Exclude TimeStamp Columns here]
Select a,b,c,d,e, .... from BackupTable --[Exclude TimeStamp Columns here]
Set Identity_insert OriginalTable Off
Only One Solution to Recover Data from Backup table is Rename Original table with random name and than rename Backup table with Original Table name in case if Identity Insert is ON for Original Table.
for example
Original Table - Invoice
Back Up Table - Invoice_back
Now Rename these tables :
Original Table - Invoice_xxx
Back Up Table - Invoice
I am new to hive, and want to know if there is anyway to insert data into Hive table like we do in SQL. I want to insert my data into hive like
INSERT INTO tablename VALUES (value1,value2..)
I have read that you can load the data from a file to hive table or you can import data from one table to hive table but is there any way to append the data as in SQL?
Some of the answers here are out of date as of Hive 0.14
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingvaluesintotablesfromSQL
It is now possible to insert using syntax such as:
CREATE TABLE students (name VARCHAR(64), age INT, gpa DECIMAL(3, 2));
INSERT INTO TABLE students
VALUES ('fred flintstone', 35, 1.28), ('barney rubble', 32, 2.32);
You can use the table generating function stack to insert literal values into a table.
First you need a dummy table which contains only one line. You can generate it with the help of limit.
CREATE TABLE one AS
SELECT 1 AS one
FROM any_table_in_your_database
LIMIT 1;
Now you can create a new table with literal values like this:
CREATE TABLE my_table AS
SELECT stack(3
, "row1", 1
, "row2", 2
, "row3", 3
) AS (column1, column2)
FROM one
;
The first argument of stack is the number of rows you are generating.
You can also add values to an existing table:
INSERT INTO TABLE my_table
SELECT stack(2
, "row4", 1
, "row5", 2
) AS (column1, column2)
FROM one
;
Slightly better version of the unique2 suggestion is below:
insert overwrite table target_table
select * from
(
select stack(
3, # generating new table with 3 records
'John', 80, # record_1
'Bill', 61 # record_2
'Martha', 101 # record_3
)
) s;
Which does not require the hack with using an already exiting table.
You can use below approach. With this, You don't need to create temp table OR txt/csv file for further select and load respectively.
INSERT INTO TABLE tablename SELECT value1,value2 FROM tempTable_with_atleast_one_records LIMIT 1.
Where tempTable_with_atleast_one_records is any table with atleast one record.
But problem with this approach is that If you have INSERT statement which inserts multiple rows like below one.
INSERT INTO yourTable values (1 , 'value1') , (2 , 'value2') , (3 , 'value3') ;
Then, You need to have separate INSERT hive statement for each rows. See below.
INSERT INTO TABLE yourTable SELECT 1 , 'value1' FROM tempTable_with_atleast_one_records LIMIT 1;
INSERT INTO TABLE yourTable SELECT 2 , 'value2' FROM tempTable_with_atleast_one_records LIMIT 1;
INSERT INTO TABLE yourTable SELECT 3 , 'value3' FROM tempTable_with_atleast_one_records LIMIT 1;
No. This INSERT INTO tablename VALUES (x,y,z) syntax is currently not supported in Hive.
You could definitely append data into an existing table. (But it is actually not an append at the HDFS level). It's just that whenever you do a LOAD or INSERT operation on an existing Hive table without OVERWRITE clause the new data will be put without replacing the old data. A new file will be created for this newly inserted data inside the directory corresponding to that table. For example :
I have a file named demo.txt which has 2 lines :
ABC
XYZ
Create a table and load this file into it
hive> create table demo(foo string);
hive> load data inpath '/demo.txt' into table demo;
Now,if I do a SELECT on this table it'll give me :
hive> select * from demo;
OK
ABC
XYZ
Suppose, I have one more file named demo2.txt which has :
PQR
And I do a LOAD again on this table without using overwrite,
hive> load data inpath '/demo2.txt' into table demo;
Now, if I do a SELECT now, it'll give me,
hive> select * from demo;
OK
ABC
XYZ
PQR
HTH
Ways to insert data into Hive table:
for demonstration, I am using table name as table1 and table2
create table table2 as select * from table1 where 1=1;
or
create table table2 as select * from table1;
insert overwrite table table2 select * from table1;
--it will insert data from one to another. Note: It will refresh the target.
insert into table table2 select * from table1;
--it will insert data from one to another. Note: It will append into the target.
load data local inpath 'local_path' overwrite into table table1;
--it will load data from local into the target table and also refresh the target table.
load data inpath 'hdfs_path' overwrite into table table1;
--it will load data from hdfs location iand also refresh the target table.
or
create table table2(
col1 string,
col2 string,
col3 string)
row format delimited fields terminated by ','
location 'hdfs_location';
load data local inpath 'local_path' into table table1;
--it will load data from local and also append into the target table.
load data inpath 'hdfs_path' into table table1;
--it will load data from hdfs location and also append into the target table.
insert into table2 values('aa','bb','cc');
--Lets say table2 have 3 columns only.
Multiple insertion into hive table
Yes you can insert but not as similar to SQL.
In SQL we can insert the row level data, but here you can insert by fields (columns).
During this you have to make sure target table and the query should have same datatype and same number of columns.
eg:
CREATE TABLE test(stu_name STRING,stu_id INT,stu_marks INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
INSERT OVERWRITE TABLE test SELECT lang_name, lang_id, lang_legacy_id FROM export_table;
To insert entire data of table2 in table1. Below is a query:
INSERT INTO TABLE table1 SELECT * FROM table2;
You can't do insert into to insert single record. It's not supported by Hive. You may place all new records that you want to insert in a file and load that file into a temp table in Hive. Then using insert overwrite..select command insert those rows into a new partition of your main Hive table. The constraint here is your main table will have to be pre partitioned. If you don't use partition then your whole table will be replaced with these new records.
Enter the following command to insert data into the testlog table with some condition:
INSERT INTO TABLE testlog SELECT * FROM table1 WHERE some condition;
I think in such scenarios you should be using HBASE which facilitates such kind of insertion but it does not provide any SQL kind of query language. You need you use Java API of HBASE like the put method to do such kind of insertion. Moreover HBASE is column oriented no-sql database.
You still can insert into complex type in Hive - it works
(id is int, colleagues array)
insert into emp (id,colleagues) select 11, array('Alex','Jian') from (select '1')
you can add values to specific columns as well, just specify the column names in which you like to add corresponding values:
Insert into Table (Col1, Col2, Col4,col5,Col7) Values ('Va11','Va2','Val4','Val5','Val7');
Make sure the columns you skip dont have not null value type.
There are few properties to set to make a Hive table support ACID properties and to insert the values into tables as like in SQL .
Conditions to create a ACID table in Hive.
The table should be stored as ORC file. Only ORC format can support ACID prpoperties for now.
The table must be bucketed
Properties to set to create ACID table:
set hive.support.concurrency =true;
set hive.enforce.bucketing =true;
set hive.exec.dynamic.partition.mode =nonstrict
set hive.compactor.initiator.on = true;
set hive.compactor.worker.threads= 1;
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set the property hive.in.test to true in hive.site.xml
After setting all these properties , the table should be created with tblproperty 'transactional' ='true'. The table should be bucketed and saved as orc
CREATE TABLE table_name (col1 int,col2 string, col3 int) CLUSTERED BY col1 INTO 4
BUCKETS STORED AS orc tblproperties('transactional' ='true');
Now its possible to inserte values into the table like SQL query.
INSERT INTO TABLE table_name VALUES (1,'a',100),(2,'b',200),(3,'c',300);
Yes we can use Insert query in Hive.
hive> create table test (id int, name string);
INSERT: INSERT...VALUES is available starting in version 0.14.
hive> insert into table test values (1,'mytest');
This is going to work for insert. We have to use values keyword.
Note: User cannot insert data into a complex datatype column (array, map, struct, union) using the INSERT INTO...VALUES clause.
I have 2 sql tables that are exactly the same format, 1 being used as a temporary table and 1 being used as a static table.
Currently my code just wipes the static table and populates it each time with the new data from the temp table, but this isn't exactly what I need. I am trying to create some type of sql diff that compares the 2 tables, that will then delete the records that aren't in the temporary table but are in the static table, and it will add new records that are in the temp table but not the static table. So the static table is only being updated each run instead of being wiped and re-written.
So if my temp table has: ABC1, ABC2, ABC4
and my static table has: ABC1, ABC3, ABC4
My sql query would ideally return: ABC1, ABC2, ABC4
I have 2 queries that seem to select the values I want to remove and the ones I want to add, but I currently can't get the delete one to function properly so I am wondering if there is something I am missing from the query.
This query inserts the data that is in the temp table, but not the static table:
Insert into [static table]
SELECT *
FROM [temp table]
EXCEPT
SELECT *
FROM [static table]
This query should delete the data that's in static table but not the temp table:
delete from [static table]
where not exists
(SELECT *
FROM [static table]
EXCEPT
SELECT *
FROM [temp table] )
Can anyone suggest what the problem is with my query or if there's a better way to perform this task? Thanks
Take a look at MERGE which was introduced in SQL 2008.
e.g. untested, but something like...
MERGE StaticTable AS target
USING TempTable AS source ON target.ColumnA = source.ColumnA
-- value doesn't exist in the target table, so add it
WHEN NOT MATCHED BY TARGET THEN
INSERT (ColumnA) VALUES (source.ColumnA)
-- value doesn't exist in the source table, so delete from target
WHEN NOT MATCHED BY SOURCE THEN
DELETE
Edit: To work on multiple columns, something like:
MERGE StaticTable AS target
USING TempTable AS source ON target.ColumnA = source.ColumnA
AND target.ColumnB = source.ColumnB
-- value doesn't exist in the target table, so add it
WHEN NOT MATCHED BY TARGET THEN
INSERT (ColumnA, ColumnB) VALUES (source.ColumnA, source.ColumnB)
-- value doesn't exist in the source table, so delete from target
WHEN NOT MATCHED BY SOURCE THEN
DELETE
I guess that 'merge' should do what you want.
Here are the details:
http://msdn.microsoft.com/en-us/library/bb522522(v=sql.105).aspx
If your tables have any defined unique keys you can probably use IN syntax:
DELETE FROM static WHERE id NOT IN(SELECT id from temporary);
INSERT INTO static WHERE id IN(SELECT id from temporary) AND NOT id IN (SELECT id from static);