I'm trying to merge a temporary table, populated with data from a c# application with a table in my SQL database.
Having read through some articles it seems that SQL Merge should be the most appropriate option and it works great when I'm updating or deleting groups of entries from the database.
My problem is that I'm unable to add a new row of data where the foreign keys don't match what is already in the database table, but instead it is just removing the rows in the database and replacing the new ones in the temporary table.
I have two foreign keys referenced in the table FirstDBTableID and SecondDBTableID.
This is my SQL so far:
ALTER PROCEDURE [dbo].[SP_UpdateTable]
#TempTable as TempTableType READONLY
AS
BEGIN
MERGE dbo.Table AS target
USING #TempTable AS source ON target.FirstDBTableID = source.FirstDBTableID
AND target.SecondDBTableID = source.SecondDBTableID
WHEN MATCHED THEN
UPDATE SET target.Provider = source.Provider, target.Value = source.Value, target.Quantity = source.Quantity
-- value doesn't exist in the target table, so add it
WHEN NOT MATCHED BY TARGET THEN
INSERT (FirstDBTableID, SecondDBTableID, Provider, Value, Quantity)
VALUES (source.FirstDBTableID, source.SecondDBTableID, source.Provider, source.Value, source.Quantity)
-- value doesn't exist in the source table, so delete from target
WHEN NOT MATCHED BY SOURCE THEN
DELETE;
END
Have a missed statement? Maybe SQL Merge isn't going to work here? If I haven't made it clear please ask me questions.
Thanks in advance.
Related
I am working on a task where my source is AWS RDS - SQL Server and my target is Azure SQL Server.
There's a table with 80M records in my source that needs to be merged with my target table.
This merging will happen every 15 mins and based on the business key, I need to -
Update the target table if the key is updated in the source table.
Insert a new key into the target table.
Mark isDeleted as true in the target if the key is no more present in the source.
IMP Note - The source row is hard-deleted and no history is maintained.
Since this merging happens every 15 mins and the source table is pretty big, I use lastUpdated column to select only limited records in the source query of my merge query.
With this, I am able to perfectly handle the "upsert" scenario, but on delete, it is deleting all the records from the target which is not desirable.
I have tried the below option -
Read the entire source table in a temp_table every 15 mins and then perform merge from temp_table to the target table. But this is very costly in terms of processing and time.
Is there any better way to handle this scenario? I am happy to share more information as needed.
I think you can solve the problem by adding new column called SyncStamp, the idea is, we update or insert the same value for SyncStamp, So the other rows that have not this value should be updated as IsDeleted.
I prefer to get the actual timestamp for SyncStamp but you can choose random numbers.
--get timestamp
Declare #SyncStamp bigint = (SELECT DATEDIFF_BIG(Second, '1970-01-01 00:00:00', GETUTCDATE()))
MERGE TargetTable AS Target
USING SourceTable AS Source
ON Source.BusinessKey = Target.BusinessKey
-- For Inserts
WHEN NOT MATCHED BY Target THEN
INSERT (ProductID,ProductName, SyncStamp)
VALUES (Source.ProductID,Source.ProductName, #SyncStamp)
-- For Updates
WHEN MATCHED THEN UPDATE SET
Target.ProductName = Source.ProductName,
Target.SyncStamp = #SyncStamp;
--Update isDeleted
UPDATE TargetTable
SET IsDeleted= 1
Where IsDeleted=0 and SyncStamp <> #SyncStamp
I am looking for the best solution, of a problem I've met.
I have several databases, which I align quite often to have the same data in it. In some referential tables there are Auto-Increment ID columns, and in other tables there are columns that match these ID values. Unfortunately, on the source DB IDs after changes might be like: (10,15,20), while, after inserting it into destination table it is (1,2,3). I hope my mock-ups clarify the situation:
Source DB RefTable:
ID_________Name
10_________a
15_________b
20_________c
Destination DB RefTable:
ID_________Name
1__________a
2__________b
3__________c
Source FactTable:
RefTableID___OtherColumns
10__________a
10__________b
15__________c
15__________d
The problem is that after migrating FactTable to the destination DB it does not match records from the RefTable. I had an idea to add another column that will be editable in contrast to ID column. The issue is that the end-user should not see it (it will not mean anything for him), so it should be also auto-increment. What should I use in this situation:
sequence?
-trigger?
-post add/save stored procedure?
Or maybe someone has better idea how to solve this problem?
In SQL Server, an identity columns can't be updated once the row was inserted into a table.
However, it's value can be inserted manually by using set identity_insert for the specified table.
In each session, you can have only a single table with identity_insert set to on at the same time, so always set it back to off once insert is complete.
The code might look something like this:
set identity_insert RefTable on;
insert into RefTable (id, name)
select id, name
from sourceDb.sourceSchema.RefTable s
where not exists (
select 1
from RefTable t
where t.Id = s.Id
);
set identity_insert RefTable off;
Currently I need to move three columns from table A to table B. And I am using the update join table script to copy the existing data to the new columns. Afterwards the old column at table A will be drop.
Alter table NewB add columnA integer
Alter table NewB add columnB integer
Update NewB
Set NewB.columnA = OldA.columnA, NewB.columnB = OldA.columnB
From NewB
Join OldA on NewB.ID = OldA.ID
Alter table OldA drop column columnA
Alter table OldA drop column columnB
These script will add new columns and update the existing data from the old table to the newly created columns. Then remove the old columns.
But due to system structure, I will required to run SQL Script for more than one times to makes sure the database is up to date.
Although I did If (Columns Exist) Begin (Alter Add, Update, Alter Drop) End to ensure the existence of columns required. But when the script runs at the next time, it will hit error that says the columns was not found from the old table in the "update" query. Because the columns were dropped when the script run at the first time.
Is there other ways to solve?
you will not be able to update using join, But you can do like this :
Update NewB set NewB.columnA = (select OldA.columnA from OldA where NewB.ID = OldA.ID);
Update NewB set NewB.columnB = (select OldA.columnB from OldA where NewB.ID = OldA.ID);
I don't know which database you are using, in database there are some system tables, from where you can get whether the column does exist in table or not, like in oracle, All_TAB_COLUMNS contains the information of all the columns of tables, so you can hit that table like below :
select 1 from ALL_TAB_COLUMNS where TABLE_NAME='OldA' and COLUMN_NAME in ('columnA','columnB');
if resulting records are empty that means specified columns are not present in the table so you can skip your queries.
There must be something wrong with your is column exists check. I have similar DDL and DML operations many times. As you did not show how you are checking column existence I am not able to tell you what's wrong.
Anyway, you are adding a new column to a table. We can check if such column exists, if not - run the script, if yes- skip the script. And here is the check:
IF EXISTS(SELECT 1 FROM [sys].[columns] WHERE OBJECT_ID('[dbo].[NewB]') = [object_id] AND [name] = 'columnA')
BEGIN
BEGIN TRANSACTION;
....
COMMIT TRANSACTION;
END;
I am using SQL Server 2012 and have two tables with identical structure. I want to insert new records from table 1 to table 2 if they don't already exist in table 2.
If they already exist, I want to update all of the existing records in table 2.
There are some 30 columns in my tables and I want to update all of them.
Can someone please help with this? I had a look at various links posted over internet, but quite don't understand how my statement should look like.
It's really not that hard....
You need:
a source table (or query) to provide data
a target table to merge it into
a condition on which those two tables are checked
a statement what to do if a match (on that condition) is found
a statement what to do if NO match (on that condition) is found
So basically, it's something like:
-- this is your TARGET table - this is where the data goes into
MERGE dbo.SomeTable AS target
-- this is your SOURCE table where the data comes from
USING dbo.AnotherTable AS source
-- this is the CONDITION they have to "meet" on
ON (target.SomeColumn = source.AnotherColumn)
-- if there's a match, so if that row already exists in the target table,
-- then just UPDATE whatever columns in the existing row you want to update
WHEN MATCHED THEN
UPDATE SET Name = source.Name,
OtherCol = source.SomeCol
-- if there's NO match, that is the row in the SOURCE does *NOT* exist in the TARGET yet,
-- then typically INSERT the new row with whichever columns you're interested in
WHEN NOT MATCHED THEN
INSERT (Col1, Col2, ...., ColN)
VALUES (source.Val1, source.Val2, ...., source.ValN);
I get this error:
Cannot create INSTEAD OF DELETE or INSTEAD OF UPDATE TRIGGER 'trig_Income_Updater' on table 'MYBUDGET.tbl_Income'. This is because the table has a FOREIGN KEY with cascading DELETE or UPDATE.
I can use 'FOR UPDATE'. but how to make it to ignore the original update ?
To ignore the origional update you will need to utilize a RAISERROR and then ROLLBACK.
Here is an example under the "Using a DML AFTER trigger to enforce a business rule between the PurchaseOrderHeader and Vendor tables" section
Just update the fields you want and then "undo" the original update using the "inserted" and "deleted" temporary tables that are provided to the trigger.
For example (untested):
--Do the stuff you want
UPDATE table SET fields = values WHERE some condition
--Undo the original update (minus anything you WANT changed above)
UPDATE table SET unchangingfield = deleted.unchangingfield WHERE ID = deleted.ID
The "inserted" table will contain the new values, and the "deleted" table contains the values that are being changed. You can join, query and otherwise treat them as though they were actual tables.