Suppose I have two tables source and destination . The source table changes over time, for example from one day to the other. To give an example I will create here two tables source_before and source_after but in reality this is the very same table where some DML statements happen over time.
create table source_before (
id int,
name varchar2(40),
creation_time date
);
insert all
into source_before values (1,'bola','01-Jan-20')
into source_before values (2,'gol','02-Jan-21')
into source_before values (3,'cav','02-Jan-23')
into source_before values (4,'bhf','02-Jan-25')
select * from dual;
select * from source_before;
1 bola 01-JAN-20
2 gol 02-JAN-21
3 cav 02-JAN-23
4 bhf 02-JAN-25
create table source_after (
id int,
name varchar2(40),
creation_time date
);
insert all
into source_after values (1,'bola','01-Jan-20')
into source_after values (2,'gol','02-Jan-21')
into source_after values (5,'zzz','02-Jan-28')
into source_after values (6,'sss','02-Jan-25')
select * from dual;
select * from source_after;
1 bola 01-JAN-20
2 gol 02-JAN-21
5 zzz 02-JAN-28
6 sss 02-JAN-25
Now let's assume there is a destination table that has been updated by the time source_before existed , and thus contain the same data as source_before.
create table destination (
id int,
name varchar2(40),
creation_time date
);
insert all
into destination values (1,'bola','01-Jan-20')
into destination values (2,'gol','02-Jan-21')
into destination values (3,'cav','02-Jan-23')
into destination values (4,'bhf','02-Jan-25')
select * from dual;
1 bola 01-JAN-20
2 gol 02-JAN-21
3 cav 02-JAN-23
4 bhf 02-JAN-25
Now if I want to update destination to the new changes that source_after created, which is the deletion of id 3 and 4 and the insertion of id 5 and 6. I can do this with this statement for the insertions, but not for the deletions.
merge into destination d
using (select * from source_after) sa on (d.id = sa.id)
when matched then update
set
d.name = sa.name,
d.creation_time = sa.creation_time
when not matched then
insert (
d.id,
d.name,
d.creation_time
)
values
(
sa.id,
sa.name,
sa.creation_time
);
select * from destination;
1 bola 01-JAN-20
2 gol 02-JAN-21
3 cav 02-JAN-23
4 bhf 02-JAN-25
6 sss 02-JAN-25
5 zzz 02-JAN-28
As we can see rows 5 and 6 have been inserted but the statement is unable to delete the the rows 3 and 4. So the goal is to reflect the changes and be able to capture both insertions and deletions on the source table. Therefore, the result of destination should be:
1 bola 01-JAN-20
2 gol 02-JAN-21
5 zzz 02-JAN-28
6 sss 02-JAN-25
You are going to need a database link of one kind or another, or you are going to need a middleware solution like NiFi or some such to act as an intermediary.
In either case with a DB link you need a merge to replicate the inserts and updates (such as the one in your example), and a separate delete command to handle the deletions. You can't do it all in a single command. If you can't modify the structure of the target database, then you'll have to create a link from the Oracle source database, and use something like this (no promises on performance):
merge into destination#dblink d
using (select * from source_after) sa on (d.id = sa.id)
when matched then update
set
d.name = sa.name,
d.creation_time = sa.creation_time
when not matched then
insert (
d.id,
d.name,
d.creation_time
)
values
(
sa.id,
sa.name,
sa.creation_time
);
delete from destination#dblink where id not in
(select id from source_after);
If you don't want a direct DB link (which isn't always possible or practical) then use a trigger in the source database to record the various transactions and their details like relevant column values in an audit table (Oracle doesn't have a convenient Change Data Capture feature in its most recent versions, so you have to make your own). Then use an external tool like NiFi to read that table, convert the audit records into individual DML commands that can be queued and applied to the target database. There are a number of different ways to do that with NiFi, and other data pipeline tools that could do the same thing as well, so I can't really get into specifics there, but hopefully you get the idea.
Also, if using an external tool like NiFi then anticipate the possibility that at some point the sync process will fail and you'll need to do a complete refresh on your target: plan a mechanism that can do a complete refresh (in fact do that first) as well as maintaining incremental updates over time.
Maybe you could consider defining your destination table with some additional columns.
With initial data in source_before as:
/* Initial data
source_before 0
----------------------
1 bola 01-JAN-20
2 gol 02-JAN-21
3 cav 02-JAN-23
4 bhf 02-JAN-25
*/
The same data are starting point for source_after so it looks the same, initialy.
You could have that initial data written in your destination table like below:
/*
ID NAME CREATION_TIME SNAP_ID_SEQUENCE SNAP_TIME STATUS
---- ---- ------------- ------------------ ------------------ -----------------
1 bola 01-Jan-20 0 01-Jan-23 19:00:00 INIT
2 gol 02-Jan-21 0 01-Jan-23 19:00:00 INIT
3 cav 02-Jan-23 0 01-Jan-23 19:00:00 INIT
4 bhf 02-Jan-25 0 01-Jan-23 19:00:00 INIT
*/
There are three columns added in destination table (you could do it differently) just as an idea of a potential solution.
SNAP_ID_SEQUENCE column's future values should be either sequence number or something similar as alternative so you could reference the right set of records to be compared with actual data in source_after table every time you take the snapshot.
Now, if your source_after table changes like in the question (rows 3 and 4 deleted and rows 5 and 6 added):
/*
after
----------------------
1 bola 01-JAN-20
2 gol 02-JAN-21
5 zzz 02-JAN-28
6 sss 02-JAN-25
*/
If you run the code below there will be 6 more rows in destination table:
INSERT INTO DESTINATION (ID, NAME, CREATION_TIME, SNAP_ID_SEQUENCE, SNAP_TIME, STATUS)
(
Select
COALESCE(a.ID, b.ID) "ID",
COALESCE(a.NAME, b.NAME) "NAME",
COALESCE(a.CREATION_TIME, b.CREATION_TIME) "CREATION_TIME",
1 "SNAP_ID_SEQUENCE", -- next_sequence_number = 1
SYSDATE "SNAP_TIME",
CASE WHEN Nvl(a.ID, -1) = b.ID THEN
CASE WHEN a.NAME != b.NAME THEN 'Name changed from ' || b.NAME || ' to ' || a.NAME END || '***' ||
CASE WHEN a.CREATION_TIME != b.CREATION_TIME THEN 'Creation changed from ' || To_Char(b.CREATION_TIME, 'dd-Mon-yy hh24:mi:ss') || ' to ' || To_Char(a.CREATION_TIME, 'dd-Mon-yy hh24:mi:ss') END
ELSE 'ID ' || b.ID || ' ' || 'IS DELETED'
END "STATUS"
From
destination b
Left Join
source_after a ON(a.ID = b.ID)
Where b.SNAP_ID_SEQUENCE = 0 -- next_sequence_number - 1 = 0
UNION ALL
Select
a.ID, a.NAME, a.CREATION_TIME,
1 "SNAP_ID_SEQUENCE", -- next_sequence_number = 1
SYSDATE "SNAP_TIME",
'NEW ROW * ID=' || a.ID "STATUS"
From
source_after a
Left Join
destination b ON(b.ID = a.ID And b.SNAP_ID_SEQUENCE = 0 ) -- next_sequence_number - 1 = 0
Where
b.ID Is Null
)
--
-- I n s e r t e d r o w s :
/* S n a p s h o t 1
ID NAME CREATION_TIME SNAP_ID_SEQUENCE SNAP_TIME STATUS
---------- -------------------------------- ------------- ---------------- --------- -------------------------
1 bola 01-JAN-20 1 04-JAN-23 ***
2 gol 02-JAN-21 1 04-JAN-23 ***
3 cav 02-JAN-23 1 04-JAN-23 ID 3 IS DELETED
4 bhf 02-JAN-25 1 04-JAN-23 ID 4 IS DELETED
5 zzz 02-JAN-28 1 04-JAN-23 NEW ROW * ID=5
6 sss 02-JAN-25 1 04-JAN-23 NEW ROW * ID=6
*/
Every time you do the next snapshot the value of SNAP_ID_SEQUENCE should be increased and passed to the code in the places marked in code. Note that in ON close of the second query and Where clause of the first query next_sequence number - 1 is used.
Second sample:
source_after changes: ID=6 is deleted, ID=7 is inserted and in ID=2 NAME has changed.
/*
after
----------------------
1 bola 01-JAN-20
2 gol_aaa 02-JAN-21
5 zzz 02-JAN-28
7 yyy 02-MAR-26
*/
If you run the above insert command but with sequence number 2 then the code and inserted rows would be like below:
INSERT INTO DESTINATION (ID, NAME, CREATION_TIME, SNAP_ID_SEQUENCE, SNAP_TIME, STATUS)
(
Select
COALESCE(a.ID, b.ID) "ID",
COALESCE(a.NAME, b.NAME) "NAME",
COALESCE(a.CREATION_TIME, b.CREATION_TIME) "CREATION_TIME",
2 "SNAP_ID_SEQUENCE", -- next_sequence_number = 2
SYSDATE "SNAP_TIME",
CASE WHEN Nvl(a.ID, -1) = b.ID THEN
CASE WHEN a.NAME != b.NAME THEN 'Name changed from ' || b.NAME || ' to ' || a.NAME END || '***' ||
CASE WHEN a.CREATION_TIME != b.CREATION_TIME THEN 'Creation changed from ' || To_Char(b.CREATION_TIME, 'dd-Mon-yy hh24:mi:ss') || ' to ' || To_Char(a.CREATION_TIME, 'dd-Mon-yy hh24:mi:ss') END
ELSE 'ID ' || b.ID || ' ' || 'IS DELETED'
END "STATUS"
From
destination b
Left Join
source_after a ON(a.ID = b.ID)
Where b.SNAP_ID_SEQUENCE = 1 -- next_sequence_number - 1 = 1
UNION ALL
Select
a.ID, a.NAME, a.CREATION_TIME,
2 "SNAP_ID_SEQUENCE", -- next_sequence_number = 2
SYSDATE "SNAP_TIME",
'NEW ROW * ID=' || a.ID "STATUS"
From
source_after a
Left Join
destination b ON(b.ID = a.ID And b.SNAP_ID_SEQUENCE = 1 ) -- next_sequence_number - 1 = 1
Where
b.ID Is Null
)
--
-- I n s e r t e d r o w s :
/* S n a p s h o t 2
ID NAME CREATION_TIME SNAP_ID_SEQUENCE SNAP_TIME STATUS
---------- -------------------------------- ------------- ---------------- --------- ---------------------------------------
1 bola 01-JAN-20 2 04-JAN-23 ***
2 gol_aaa 02-JAN-21 2 04-JAN-23 Name changed from gol to gol_aaa***
3 cav 02-JAN-23 2 04-JAN-23 ID 3 IS DELETED
4 bhf 02-JAN-25 2 04-JAN-23 ID 4 IS DELETED
5 zzz 02-JAN-28 2 04-JAN-23 ***
6 sss 02-JAN-25 2 04-JAN-23 ID 6 IS DELETED
7 yyy 02-MAR-26 2 04-JAN-23 NEW ROW * ID=7
*/
And if you want to know the data from snapshot 1 then:
Select ID, NAME, CREATION_TIME
From DESTINATION
Where SNAP_ID_SEQUENCE = 1 And
STATUS NOT LIKE '%IS DELETED'
Order By ID
/*
ID NAME CREATION_TIME
---------- -------------------------------- -------------
1 bola 01-JAN-20
2 gol 02-JAN-21
5 zzz 02-JAN-28
6 sss 02-JAN-25
*/
... for snapshot 2 :
/*
ID NAME CREATION_TIME
---------- -------------------------------- -------------
1 bola 01-JAN-20
2 gol_aaa 02-JAN-21
5 zzz 02-JAN-28
7 yyy 02-MAR-26
*/
A Trigger can solve this issue for you
For this, one option is, in the destination table add some field(s) to track date, etc, when it was deleted/updated. I even suggest that on the large scale, may be do insert of new version of the row for Update, and mark record deleted for delete. This will allow to keep the entire history.
CREATE OR REPLACE TRIGGER TakeChangesToDestination
AFTER UPDATE OR INSERT OR DELETE ON SourceTable
FOR EACH ROW
BEGIN
IF INSERTING then
-- insert into destination
ELSEIF UPDATING Then
-- update destination by ID OR Insert new row for tracking history
ELSEIF DELETING Then
-- update destination by ID - mark as DELETED
end if;
END;
Related
I have a scenario, where i have to mask the data with data within the table
let's say I have a table student_details(ID, CODE, NAME)
1 A XYZ
2 A 123
3 A QWERTY
I want the output as
1 A QWERTY
2 A XYZ
3 A 123
I want the name to be within the name list in that table
for same id I Want different name which is in the table.
select * from emp_details order by dbms_random.value;
is giving some random names which are not in list.
Can any one help me with this?
Here's one option: recalculate the ID value using ROW_NUMBER analytic function which orders rows by the hash value over concatenated name, code and id columns (that's just for example; you can pick something different).
SQL> with test (id, code, name) as
2 (select 1, 'A', 'XYZ' from dual union all
3 select 2, 'A', '123' from dual union all
4 select 3, 'A', 'QUERTY' from dual
5 ),
6 inter as
7 (select row_number() over (order by ora_hash(name || code || id)) id,
8 code, name
9 from test
10 )
11 select t.id, t.code, i.name
12 from test t join inter i on t.id = i.id;
ID C NAME
---------- - ------
1 A XYZ
2 A QUERTY
3 A 123
SQL>
If you intend to permutate selected columns in your table and leave the rest of the table unchanged, you may use a join with a key permutation table.
Assume your data as follows:
ID CODE NAME
---------- ---- ------
1001 A XYZ
1002 B 123
1004 C QUERTY
1005 A FOO
Note, that the PK is not continuous, wich is the generall case. If you have the PK a continuous sequence starting with 1, you may even simplify the solution (as proposed in other answer).
First lets define the permutation table assigning to each PK a new key in random order.
create table PERM as
with rn as (
select
id,
row_number() over (order by id) rn,
row_number() over (order by dbms_random.value) rn_new
from student)
select a.ID, b.ID ID_NEW
from rn a
join rn b
on a.RN = b.RN_NEW;
ID ID_NEW
---------- ----------
1001 1004
1002 1001
1004 1005
1005 1002
The query defines two row_number sequences, first in the order of the PK, second in random order. The final join gets the original and new (permutated) IDs.
Now to permute a selected colums is as easy as to join your table twice with the permutation table in between and choose preserved columns from the first table, the permuted columns from the second one.
select a.ID, a.code, b.name
from student a
join PERM p on a.id = p.id
join student b on p.id_new = b.id
order by a.id;
ID CODE NAME
---------- ---- ------
1001 A QUERTY
1002 B XYZ
1004 C FOO
1005 A 123
As far as you preserv the permutation table you can reconstruct the former state, if you drop it, there is no way to get the original data.
I have a Table that is joined from other tables. I want to update a null value in this Table by a specific number series. Below is the illustration:
The code for the Table which is called List_Codes
SELECT mlk.MLK_CODE
FROM zpt
LEFT OUTER JOIN mes ON mes.ZPT_ID = zpt.ZPT_ID
LEFT OUTER JOIN zmlk ON zpt.ZPT_ID = zmlk.ZPT_ID
LEFT OUTER JOIN mlk ON zmlk.MLK_ID = mlk.MLK_ID
WHERE zpt.zpt_id IS NOT NULL
and zpt.zpt_meteringcode = '123456'
ORDER BY mes.MES_STATUS DESC
Now I want to update this specific row's mlk.MLK_CODE from null to '789'. I have located this row based on the zpt.zpt_meteringcode. Any suggestions plz?
The tables look like this, and the List_Code Table is the result of the above code
Mlk Table
Mlk_id Mlk_code
1 123
2 456
Zpt Table
Zpt_id Zpt_meteringcode
10 123456
20 987654
30 654321
40 147852
Zmlk Table
Zpt_id Mlk_id
20 1
30 2
List_Code Table
Zpt_id Zpt_meteringcode Mlk_id Mlk_code
10 123456
20 987654 1 123
30 654321 2 456
40 147852
I think you need two inserts and one update statements like this.
INSERT INTO mlk (mlk_code)
VALUES ( '789' ); -- Primary key is generated
.....
INSERT INTO zmlk (zpt_id,
mlk_id)
SELECT (SELECT zpt_id
FROM zpt
WHERE zpt_meteringcode = '123456'),
(SELECT mlk_id
FROM mlk
WHERE mlk_code = '789')
FROM dual;
....
UPDATE list_code
SET ( mlk_id, mlk_code ) = (SELECT mlk_id,
mlk_code
FROM mlk
WHERE mlk_code = '789')
WHERE zpt_meteringcode = '123456';
First of all, I know that this is a case of a bad design, but I am just wondering how can I get single sql result from the jointure with the desired infos.
So it's a simple exemple, 2 tables connected with ids:
TableA TableB
id code id tableA_id (FK) start end
---- ------- ---- ---------------- ------- ------
1 codeA 1 2 NY null
2 codeB 2 2 null LA
So line 2 in TableA has 2 corresponding lines in TableB.
Desired result:
tableA_code tableB_start tableB_end
------------- -------------- ------------
codeB NY LA
So in the result in need to group the 2 lines from the table B into one line and eliminate the null values.
The query :
select A.code, B1.start, B2.end
from TableB B1, TableB B2, TableA A
where B1.tableA_id = B2.tableA_id
and B1.tableA_id = A.id
and B1.start is not null and B2.end is not null
Let me know if this works for you
SELECT A.code,
MAX(B.STRT) STRT,
MAX(B.END)
END
FROM
(SELECT 1 AS A_ID,'CODE A' AS CODE FROM DUAL
UNION ALL
SELECT 2 AS A_ID,'CODE B' AS CODE FROM DUAL
)A,
(SELECT 1 AS B_ID,2 AS A_ID,'NY' STRT, NULL AS END FROM DUAL
UNION ALL
SELECT 2 AS B_ID,2 AS A_ID, NULL STRT, 'LA' AS END FROM DUAL
)B
WHERE A.A_ID = B.A_ID
GROUP BY A.code;
I have two table table A and table B.
Table A Contain Id, Number, Time , Value1
Table B Contain Id, Data, Value2
Example of the Record on Table A:
Id Number Tried Value1
------- ---------- --------- ---------
1 123 23 5
2 124 23 6
3 1254 23 7
Example of the Record on Table B:
Id Data Value2
------ --------- -------
1 123,23 6
2 122,21 5
3 1254,23 7
My Purpose to Add Value 1 and Value 2 together by join condition of the table B Data with table A Number and Tried to match the record.
Example :
Id (Value1 + Value2)
------- -----------------
1 11
3 14
My Query:
select a.Id , a.Value1+ b.Value2
from a
join b on substring(b.Data,1,3) = a.Number and substring(b.Data,5,2) = a.Tried
I had tried substring but the value of Data Record Length is different compare on Id 1 and 3 and current of Query Result only show Id 1. Is there other way to join with 1 column field that split into 2 kind of value which take out ',' to join 2 field on table a?
Check if this query would help, in Oracle
select a.Id , (a.Value1+ b.Value2)
from a, b
where a.id = b.id and b.Data = (a.Number || ',' || a.Tried);
EDIT: Based on #Joachim Isaksson suggestion:
select a.Id ,b.Id (a.Value1+ b.Value2)
from a, b
where b.Data = (a.Number || ',' || a.Tried);
You can use the following query.
select A.Id, (A.Value1 + B.Value2) [Value1 + Value2] from tblA A
inner join tblB B on A.Number = SUBSTRING(B.Data, 1, charindex(',', B.Data, 1) - 1)
And A.Tried = SUBSTRING(B.Data, charindex(',', B.Data, 1) + 1, Len(A.Tried))
This I have tried in SQL Server.
Query of Nishanthi Grashia can also be converted in SQL Server. You just need to replace || with +
i am trying to use an insert into a column where the data for other columns already exists, but the data is not populating adjacent to the other columns, instead data is inserted after all the data in the table.
For example:
select * from tab1;
ID NAme Last_name
1 King
2 Queen
3 Rook
select * from tab2;
Id LastName_Name
1 Abc
2 def
3 xyz
SQL : Insert into tab1 (Last_name)
select tab2.LastName_Name from tab1,tab2, where tab1.Id=tab2.Id
Output:
Id Name Last_Name
1 King NULL
2 Queen NULL
3 Rook NULL
4 NULL Abc
5 NULL def
6 NULL xyz
But I want the data as below:
Id Name Last_Name
1 King Abc
2 Queen def
3 Rook xyz
Any work around for this? thanks in advance :)
Step2:
select * from tab1;
ID Name Id2
1 King NA
2 Queen NA
3 Rook NA
select * from tab2;
ID
1
2
3
4
5
6
I want the Output data as below:
The ID data in tab2 should populate in tab1 column (ID2) which are matching with TAB1.ID column values as below:
Id Name ID2
1 King 1
2 Queen 2
3 Rook 3
Can you please provide any query for this?
So you are wanting to UPDATE the rows in tab1 with the corresponding last names from tab2?
In which case, use an UPDATE statement instead of an INSERT:
UPDATE tab1
SET tab1.Last_name = tab2.LastName_Name
FROM tab1
JOIN tab2 ON tab1.Id = tab2.Id
You don't need an INSERT you need an UPDATE statement:
UPDATE tab1 SET tab1.Last_name = tab2.LastName_Name
FROM tab1 INNER JOIN tab2 ON tab1.Id = tab2.Id