I have a main table (say tableA which has columns tab_a_id, field_code , field_id). There is another table, say tableB which has columns area_id , area_code. tab_a_id is a primary key of TableA. I want to update field_id of tableA based on field_code. field_code of tableA and area_code of tableB are matching but not identical, mean field_code has other values which are not matching with area_code column. I want to set field_id = area_id if field_code = area_code but, if not matched it should set to default value -1 which is 'unknown' field. I tried with subquery and bulk update (e.g Update tableA SET field_id = (SELECT area_code from tableB where area_code = field_code)). This worked for limited set of data. But I have 3 Million matching records , which means 3 million subqueries. Another problem is there are 7 million records, resulting 4 million unmatched records & useless subqueries.
Is there any optimal way to update such records with minimum time and better efficiency. I tried merge command but it has poor performance compare to forall loop query
Updating 3 out of 7 million rows seems to be the problem here.
I've created a test set in a database on a small machine, and the fasted way to get your results is to create a new table (CTAS) with the desired data and later swap names. I have not used the primary key column tab_a_id to simplify the answer.
CREATE TABLE a (field_id NUMBER, field_code VARCHAR2(30)) NOLOGGING;
CREATE TABLE b (area_id NUMBER, area_code VARCHAR2(30)) NOLOGGING;
Using MERGE and UPDATE is quite slow (15 minutes), presumably because of the amount of changes:
UPDATE a SET field_id=-1 WHERE field_code NOT IN (SELECT area_code FROM b);
5,599,989 rows updated. (560 seconds)
MERGE INTO a USING b ON (a.field_code=b.area_code)
WHEN MATCHED THEN UPDATE SET a.field_id = b.area_id;
2,400,011 rows merged. (232 seconds)
However, creating a new table with the changed data is 20 times faster and takes only 38 seconds:
CREATE TABLE x NOLOGGING AS
SELECT a.field_id, NVL(b.area_code, -1) AS field_code
FROM a JOIN b ON a.field_code=b.area_code;
Here is the test data generation:
INSERT /*+ APPEND */ INTO a (field_id, field_code) SELECT id, to_char(id) from (select level as id from dual connect by rownum <= 1000000); COMMIT;
INSERT /*+ APPEND */ INTO a (field_id, field_code) SELECT field_id+1000000, to_char(field_id+1000000) from a; COMMIT;
INSERT /*+ APPEND */ INTO a (field_id, field_code) SELECT field_id+2000000, to_char(field_id+2000000) from a; COMMIT;
INSERT /*+ APPEND */ INTO a (field_id, field_code) SELECT field_id+4000000, to_char(field_id+4000000) from a; COMMIT;
EXEC dbms_stats.gather_table_stats(null, 'a');
INSERT /*+ APPEND */ INTO b (area_id, area_code) SELECT -field_id, field_code FROM a SAMPLE (30);
exec dbms_stats.gather_table_stats(null, 'b');
Related
I have a table that looks like:
ID|CREATED |VALUE
1 |1649122158|200
1 |1649122158|200
1 |1649122158|200
That I'd like to look like:
ID|CREATED |VALUE
1 |1649122158|200
And I run the following query:
DELETE FROM MY_TABLE T USING (SELECT ID,CREATED,ROW_NUMBER() OVER (PARTITION BY ID ORDER BY CREATED DESC) AS RANK_IN_KEY FROM MY_TABLE T) X WHERE X.RANK_IN_KEY <> 1 AND T.ID = X.ID AND T.CREATED = X.CREATED
But it removes everything from MY_TABLE and not just other rows with the same value. This is more than just selecting distinct records, I'd like to enforce a unique constraint to get the latest value of ID and keep just one record for it, even if there were duplicates.
So
ID|CREATED |VALUE
1 |1649122158|200
1 |1649122159|300
2 |1649122158|200
2 |1649122158|200
3 |1649122170|500
3 |1649122160|200
Would become (using the same final unique constraint statement):
ID|CREATED |VALUE
1 |1649122159|300
2 |1649122158|200
3 |1649122170|500
How can I improve my logic to properly handle these unique constraint modifications?
Check out this post: https://community.snowflake.com/s/question/0D50Z00008EJgemSAD/how-to-delete-duplicate-records-
If all columns make up a unique records, the recommended solution is the insert all the records into a new table with SELECT DISTINCT * and do a swap. You could also do a INSERT OVERWRITE INTO the same table.
Something like INSERT OVERWRITE INTO tableA SELECT DISTINCT * FROM tableA;
The following setup should leave rows with id of 1 and 3. And not delete all rows as you say.
Schema
create table t (
id int,
created int ,
value int
);
insert into t values(1, 1649122158, 200);
insert into t values(1 ,1649122159, 300);
insert into t values(2 ,1649122158, 200);
insert into t values(2 ,1649122158, 200);
insert into t values(3 ,1649122170, 500);
insert into t values(3 ,1649122160, 200);
Delete statement
with x as (
SELECT
id, created,
row_number() over(partition by id) as r
FROM t
)
delete from t
using x
where x.id = t.id and x.r <> 1 and x.created = t.created
;
Output
select * from t;
1 1649122158 200
3 1649122170 500
The logic is such, that the table in the using clause is joined with the operated on table. Following the join logic, it just matches by some key. In your case, you have key as {id,created}. This key is duplicated for rows with id of 2. So the whole group is deleted.
I'm no savvy in database schemas. But as a thought, you may add a row with a rank to existing table. And after that you can proceed with deletion. This way you do not need to create other table and insert values to that. Be warned that data may become fragmented(physically, on disks). So you will need to run some kind of tune up later.
Update
You may find this almost one-liner interesting:
SO answer
I will duplicate code here, as it is so small and well written.
WITH
u AS (SELECT DISTINCT * FROM your_table),
x AS (DELETE FROM your_table)
INSERT INTO your_table SELECT * FROM u;
I am trying to get the row number of an inserted record so I can use it for a select statement. What I am trying to accomplish is insert a person into one table, get that row number and then select something from another table where the row numbers match. Here is what I got so far:
INSERT INTO TableA Values (‘Person’)
Select timeToken
From
(
Select
Row_Number() Over (Order By tokenOrder) As RowNum
, *
From TableB WHERE taken = false
) t2
Where RowNum = (Row Number of Inserted Item)
How do I get the row number of the inserted item, I want to compare ids as some records might have been deleted so they would not match.
TABLEA Data (primary key is id)
id name
3 John
12 Steve
TABLEB Data (primary key is id)
id timeToken tokenOrder taken
2 1:00am 1 false
3 2:00am 2 false
5 3:00am 3 true
6 4:00am 4 false
My expect result when I insert person, the select take would return 4:00am
I am doing this in a stored procedure.
It is an error to think that rows have numbers unless an ORDER BY clause is included.
The only way to find a row after you have inserted it is to search for it. Presumably your table has a primary key; use that to search for it.
Try This .It may help you out
Declare #TableA_PK BIGINT
INSERT INTO TableA Values ('Person')
SET #TableA_PK=SCOPE_IDENTITY()
Select timeToken
From
(
Select
Row_Number() Over (Order By tokenOrder) As RowNum
, *
From TableB WHERE taken = false
) t2
Where RowNum =#TableA_PK
SCOPE_IDENTITY(): Scope Identity will captures the last inserted record primary key value and which can be stored in a varaible and
and then it can be for further re-use
By the sounds of it you are trying to do something like what is listed on thhe following link LINK - SQL Server - Return value after INSERT
Basically :
INSERT INTO TableA (Person)
OUTPUT Inserted.ID
VALUES('bob');
Adding a foreign key constraint(referencing primary key in table A) in table b will be good since you won't be able to delete records from table A without deleting them from table B. It'll be helpful for comparing the records using ID.
Try this
declare #rowNum int;
INSERT INTO TableA Values ('Person')
SET #rowNum =SCOPE_IDENTITY()
select * from TableA where id = #rowNum
In an Oracle database (11gR2), I have a table my_table with columns (sequence, col1, col2, col3). I want to insert values into the table that are queried from other tables, i.e. insert into my_table select <query from other tables>. The problem is that the primary key is the four columns, hence I need to add a sequence starting from 0 up till the count of the rows to be inserted (order is not a problem).
I tried using a loop like this:
DECLARE
j NUMBER;
r_count number;
BEGIN
select count(1) into r_count from <my query to be inserted>;
FOR j IN 0 .. r_count
LOOP
INSERT INTO my_table
select <my query, incorporating r_count as sequence column> ;
END LOOP;
END;
But it didn't work, actually looped r_count times trying to insert the entire rows every time, as logically it shall do. How can I achieve the expected goal and insert rows with adding a sequence column?
Don't do this in a loop. Just use row_number():
INSERT INTO my_table(seq, . . .)
select row_number() over (order by NULL) - 1, . . .
from . . .;
Let's create table with sample data (to simulate your source of data)
-- This is your source query table (can be anything)
CREATE TABLE source_table
(
source_a VARCHAR(255),
source_b VARCHAR(255),
source_c VARCHAR(255)
);
insert into source_table (source_a, source_b, source_c) values ('A', 'B', 'C');
insert into source_table (source_a, source_b, source_c) values ('D', 'E', 'F');
insert into source_table (source_a, source_b, source_c) values ('G', 'H', 'I');
Then create target table, with id and 3 data columns.
-- This is your target_table
CREATE TABLE target_table
(
id NUMBER(9,0),
target_a VARCHAR2(255),
target_b VARCHAR2(255),
target_c VARCHAR2(255)
);
-- This is sequence used to ensure unique number in 1st column
CREATE sequence target_table_id_seq start with 0 minvalue 0 increment BY 1;
Finally, perform insert, loading id from sequence, rest of the data from source table.
INSERT INTO target_table
SELECT target_table_id_seq.nextval,
source_a,
source_b,
source_c
FROM source_table;
Results might look like
1 A B C
2 D E F
3 G H I
If you added some values later, they will continue with numbering 4,5,6 etc.. Or do you want to get order only inside the group ? Thus if you added 2 more rows JKL and MNO, target table would look like this
1 A B C
2 D E F
3 G H I
1 J K L
2 M N O
For that you need different solution (don't even need sequencer)
SELECT
RANK() OVER (ORDER BY source_a, source_b, source_c),
source_a,
source_b,
source_c
FROM source_table;
Technically you could use ROWNUM directly, BUT I opt for RANK() OVER analytical function due to consistent result. Please note, that this will breach your complex primary key if you try to insert the same rows twice (My first solution doesn't)
Clearly, you should use Oracle sequence.
First, create a sequence:
create sequence seq_my_table start with 0 minvalue 0 increment by 1;
Then use it:
INSERT INTO my_table (sequence, ...)
select seq_my_table.nextval, <the rest of my query>;
Sequence numbers will be inserted in succession.
So, you already have the table, it has the required number of rows, and now you want to add numbers from 0 to total number of rows minus one in the column named sequence? (perhaps not "sequence" but something less likely to clash with Oracle reserved words?)
Then this should work:
update my_table set seq = rownum - 1;
How do I update all columns at once without specifying the column name.
Say I have table A
ID NAME AGE
21 MATT 45
Table B
ID NAME AGE
21 SCOTT 24
What I expect
update table A
set A.columns=B.columns
Basically I'm trying to sync two tables.
As far as I know, there are no ways to update the table column values without specifying column name. Your purpose is to make the table A has the same values of table B, then you can delete all rows from the table A, and copy the data to table A.
delete from A;
insert into A select * from B;
commit;
If you have some reasons only to use update statement, and there are lots of columns, then you can generate the update statement by using dictionary.
select 'A.'||column_name||'=B.'||column_name||','
from all_tab_columns
where owner = 'your schema name'
and table_name = 'A';
You execute the query and copy the result and edit.
update A
set <paste the result of previous query here>
;
You can do it in PLSQL:
for src in ( select * from B ) loop
update A set ROW = src where A.id = src.id;
end loop;
or for insert
for src in ( select * from B ) loop
insert into A values src;
end loop;
I was wondering if it is possible to move all rows of data from one table to another, that match a certain query?
For example, I need to move all table rows from Table1 to Table2 where their username = 'X' and password = 'X', so that they will no longer appear in Table1.
I'm using SQL Server 2008 Management Studio.
Should be possible using two statements within one transaction, an insert and a delete:
BEGIN TRANSACTION;
INSERT INTO Table2 (<columns>)
SELECT <columns>
FROM Table1
WHERE <condition>;
DELETE FROM Table1
WHERE <condition>;
COMMIT;
This is the simplest form. If you have to worry about new matching records being inserted into table1 between the two statements, you can add an and exists <in table2>.
This is an ancient post, sorry, but I only came across it now and I wanted to give my solution to whoever might stumble upon this one day.
As some have mentioned, performing an INSERT and then a DELETE might lead to integrity issues, so perhaps a way to get around it, and to perform everything neatly in a single statement, is to take advantage of the [deleted] temporary table.
DELETE FROM [source]
OUTPUT [deleted].<column_list>
INTO [destination] (<column_list>)
All these answers run the same query for the INSERT and DELETE. As mentioned previously, this risks the DELETE picking up records inserted between statements and could be slow if the query is complex (although clever engines "should" make the second call fast).
The correct way (assuming the INSERT is into a fresh table) is to do the DELETE against table1 using the key field of table2.
The delete should be:
DELETE FROM tbl_OldTableName WHERE id in (SELECT id FROM tbl_NewTableName)
Excuse my syntax, I'm jumping between engines but you get the idea.
A cleaner representation of what some other answers have hinted at:
DELETE sourceTable
OUTPUT DELETED.*
INTO destTable (Comma, separated, list, of, columns)
WHERE <conditions (if any)>
Yes it is. First INSERT + SELECT and then DELETE orginals.
INSERT INTO Table2 (UserName,Password)
SELECT UserName,Password FROM Table1 WHERE UserName='X' AND Password='X'
then delete orginals
DELETE FROM Table1 WHERE UserName='X' AND Password='X'
you may want to preserve UserID or someother primary key, then you can use IDENTITY INSERT to preserve the key.
see more on SET IDENTITY_INSERT on MSDN
You should be able to with a subquery in the INSERT statement.
INSERT INTO table1(column1, column2) SELECT column1, column2 FROM table2 WHERE ...;
followed by deleting from table1.
Remember to run it as a single transaction so that if anything goes wrong you can roll the entire operation back.
Use this single sql statement which is safe no need of commit/rollback with multiple statements.
INSERT Table2 (
username,password
) SELECT username,password
FROM (
DELETE Table1
OUTPUT
DELETED.username,
DELETED.password
WHERE username = 'X' and password = 'X'
) AS RowsToMove ;
Works on SQL server make appropriate changes for MySql
Try this
INSERT INTO TABLE2 (Cols...) SELECT Cols... FROM TABLE1 WHERE Criteria
Then
DELETE FROM TABLE1 WHERE Criteria
You could try this:
SELECT * INTO tbl_NewTableName
FROM tbl_OldTableName
WHERE Condition1=#Condition1Value
Then run a simple delete:
DELETE FROM tbl_OldTableName
WHERE Condition1=#Condition1Value
You may use "Logical Partitioning" to switch data between tables:
By updating the Partition Column, data will be automatically moved to the other table:
here is the sample:
CREATE TABLE TBL_Part1
(id INT NOT NULL,
val VARCHAR(10) NULL,
PartitionColumn VARCHAR(10) CONSTRAINT CK_Part1 CHECK(PartitionColumn = 'TBL_Part1'),
CONSTRAINT TBL_Part1_PK PRIMARY KEY(PartitionColumn, id)
);
CREATE TABLE TBL_Part2
(id INT NOT NULL,
val VARCHAR(10) NULL,
PartitionColumn VARCHAR(10) CONSTRAINT CK_Part2 CHECK(PartitionColumn = 'TBL_Part2'),
CONSTRAINT TBL_Part2_PK PRIMARY KEY(PartitionColumn, id)
);
GO
CREATE VIEW TBL(id, val, PartitionColumn)
WITH SCHEMABINDING
AS
SELECT id, val, PartitionColumn FROM dbo.TBL_Part1
UNION ALL
SELECT id, val, PartitionColumn FROM dbo.TBL_Part2;
GO
--Insert sample to TBL ( will be inserted to Part1 )
INSERT INTO TBL
VALUES(1, 'rec1', 'TBL_Part1');
INSERT INTO TBL
VALUES(2, 'rec2', 'TBL_Part1');
GO
--Query sub table to verify
SELECT * FROM TBL_Part1
GO
--move the data to table TBL_Part2 by Logical Partition switching technique
UPDATE TBL
SET
PartitionColumn = 'TBL_Part2';
GO
--Query sub table to verify
SELECT * FROM TBL_Part2
Here is how do it with single statement
WITH deleted_rows AS (
DELETE FROM source_table WHERE id = 1
RETURNING *
)
INSERT INTO destination_table
SELECT * FROM deleted_rows;
EXAMPLE:
postgres=# select * from test1 ;
id | name
----+--------
1 | yogesh
2 | Raunak
3 | Varun
(3 rows)
postgres=# select * from test2;
id | name
----+------
(0 rows)
postgres=# WITH deleted_rows AS (
postgres(# DELETE FROM test1 WHERE id = 1
postgres(# RETURNING *
postgres(# )
postgres-# INSERT INTO test2
postgres-# SELECT * FROM deleted_rows;
INSERT 0 1
postgres=# select * from test2;
id | name
----+--------
1 | yogesh
(1 row)
postgres=# select * from test1;
id | name
----+--------
2 | Raunak
3 | Varun
If the two tables use the same ID or have a common UNIQUE key:
1) Insert the selected record in table 2
INSERT INTO table2 SELECT * FROM table1 WHERE (conditions)
2) delete the selected record from table1 if presents in table2
DELETE FROM table1 as A, table2 as B WHERE (A.conditions) AND (A.ID = B.ID)
It will create a table and copy all the data from old table to new table
SELECT * INTO event_log_temp FROM event_log
And you can clear the old table data.
DELETE FROM event_log
For some scenarios, it might be the easiest to script out Table1, rename the existing Table1 to Table2 and run the script to recreate Table1.