I have a table in SQL Server with many fields. Two of which are IDNumber and Resolved(Date) Resolved is Null in SQL Server. Then I have data with IDNumber and and Resolved(Date) that I want to import and update the table. I want it to find the matching IDNumber and put in the updated resolved date. How can I do this?
Thanks!
When doing updates SSIS will update a row at a time, which can be quite costly. I would advise to use an Execute SQL task to create a staging table:
IF (OBJECT_ID(N'dbo.tmpIDNumberStaging') IS NOT NULL
DROP TABLE dbo.tmpIDNumberStaging;
CREATE TABLE dbo.tmpIDNumberStaging
( IDNumber INT NOT NULL,
Resoved DATE NOT NULL
);
Then use a Data Flow task to import your data to this table (not sure what your source is, but your destination would be OLE DB Destination. You may need to set "Delay Validation" on the Data Flow task to false and/or "Validate External Meta Data" to false on the destination because the destination is created at run time).
Finally use this staging table to update your main table (and drop the staging table to clean up)
UPDATE YourMainTable
SET Resolved = st.Resolved
FROM YourMainTable t
INNER JOIN dbo.tmpIDNumberStaging st
ON st.IDNumber = t.IDNumber
WHERE t.Resolved IS NULL;
-- CLEAN UP AND DROP STAGING TABLE
IF (OBJECT_ID(N'dbo.tmpIDNumberStaging') IS NOT NULL
DROP TABLE dbo.tmpIDNumberStaging;
Related
What would be the best way to transfer certain number of records daily from source to destination and then remove from source?
DB - SQL server on cloud.
As the databases are in the same server, you can create a job that transfers the data do the other database.
Because the databases are in the same server you can easily access them, just by adding the database before the table in the query, look the test that i did:
CREATE DATABASE [_Source]
CREATE DATABASE [_Destination]
CREATE TABLE [_Source].dbo.FromTable
(
some_data varchar(10)
)
CREATE TABLE [_Destination].dbo.ToTable
(
some_data varchar(10)
)
INSERT INTO [_Source].dbo.FromTable VALUES ('PAULO')
--THE JOB WOULD BE SOMETHING LIKE THIS:
-- INSERT INTO DESTINATION GETTING THE DATA FROM THE SOURCE
INSERT INTO [_Destination].dbo.ToTable
SELECT some_data
FROM [_Source].dbo.FromTable
-- DELETE FROM SOURCE
DELETE [_Source].dbo.FromTable
I have a data factory pipeline that is taking data from SQL Server and copying it to table storage:
Azure Sql Server View --> Table Storage Cache
This step is doing a REPLACE of the specific row based on a guid from the SQL Server view.
If a record is deleted from the source view, how do we delete that same record from table storage?
During the Copy Active, we couldn't do operation to the table data. Data Factory does not support this operation.
I have created a data pipeline to copy data from my SQL Server to the table storage for the test. Before your Copy Active begin, you can delete or insert your record as you want. And you can preview the data in Data Factory. Once the Copy Active is published and finished, it means that your data have been copied to the table storage.
If your really need to delete the same record in Table Storage, one of the ways is that you can login you Azure portal and using the "Storage Explorer(preview)" to delete the record:
Hope this helps.
Here is a hint -- some code I have run in production for years... the idea is to have the source use a timestamp (or simpler flag) on deleted records and then to have your MERGE respect it.
CREATE PROCEDURE [DataFactory].[MergeCompany]
(
#Company [DataFactory].[CompanyType] READONLY
)
AS
BEGIN
MERGE INTO [Company] AS TGT
USING #Company As SRC
ON TGT.pk = SRC.pk
--
WHEN MATCHED AND SRC.aud_delete_date IS NOT NULL
THEN DELETE
--
WHEN MATCHED AND SRC.aud_delete_date IS NULL
THEN UPDATE SET
TGT.comp_number = SRC.comp_number
,TGT.comp_name = SRC.comp_name
,TGT.aud_update_date = SRC.aud_update_date
,TGT.aud_create_date = SRC.aud_create_date
--
WHEN NOT MATCHED BY TARGET
THEN INSERT(
pk
,comp_number
,comp_name
,aud_update_date
,aud_create_date
)
VALUES(
SRC.pk
,SRC.comp_number
,SRC.comp_name
,SRC.aud_update_date
,SRC.aud_create_date
)
; -- Required semicolon
END
GO
Currently I need to move three columns from table A to table B. And I am using the update join table script to copy the existing data to the new columns. Afterwards the old column at table A will be drop.
Alter table NewB add columnA integer
Alter table NewB add columnB integer
Update NewB
Set NewB.columnA = OldA.columnA, NewB.columnB = OldA.columnB
From NewB
Join OldA on NewB.ID = OldA.ID
Alter table OldA drop column columnA
Alter table OldA drop column columnB
These script will add new columns and update the existing data from the old table to the newly created columns. Then remove the old columns.
But due to system structure, I will required to run SQL Script for more than one times to makes sure the database is up to date.
Although I did If (Columns Exist) Begin (Alter Add, Update, Alter Drop) End to ensure the existence of columns required. But when the script runs at the next time, it will hit error that says the columns was not found from the old table in the "update" query. Because the columns were dropped when the script run at the first time.
Is there other ways to solve?
you will not be able to update using join, But you can do like this :
Update NewB set NewB.columnA = (select OldA.columnA from OldA where NewB.ID = OldA.ID);
Update NewB set NewB.columnB = (select OldA.columnB from OldA where NewB.ID = OldA.ID);
I don't know which database you are using, in database there are some system tables, from where you can get whether the column does exist in table or not, like in oracle, All_TAB_COLUMNS contains the information of all the columns of tables, so you can hit that table like below :
select 1 from ALL_TAB_COLUMNS where TABLE_NAME='OldA' and COLUMN_NAME in ('columnA','columnB');
if resulting records are empty that means specified columns are not present in the table so you can skip your queries.
There must be something wrong with your is column exists check. I have similar DDL and DML operations many times. As you did not show how you are checking column existence I am not able to tell you what's wrong.
Anyway, you are adding a new column to a table. We can check if such column exists, if not - run the script, if yes- skip the script. And here is the check:
IF EXISTS(SELECT 1 FROM [sys].[columns] WHERE OBJECT_ID('[dbo].[NewB]') = [object_id] AND [name] = 'columnA')
BEGIN
BEGIN TRANSACTION;
....
COMMIT TRANSACTION;
END;
I currently have a stored procedure that performs bulk insert into a table named "TomorrowPatients" from a .csv file. When performing the bulk insert, I need to determine if the record being added already exists within the table and if so DO NOT add the record. If the record does not exist then I need to APPEND it to the table. What is the most efficient way to go about this? Any help will be greatly appreciated.
EDIT: I have created a temp table called "TomorrowPatients_Temp". I am trying to use this table to determine which records to insert.
Insert your the whole data into a temporary table, say #TempData.Then use the following code :
INSERT INTO TommorowPatients
SELECT * FROM #TempTable TT
LEFT JOIN TommorowPatients TP ON TT.PatientId = TP.PatienId
AND TT.PatientName = TP.PatientName
AND TT.PatientSSN = TP.PatientSSN
WHERE TP.PatientId IS NULL
Where PatientId is you primary key for the TommorowPatients table.
DO NOT add the "RoomNumber" column with the LEFT JOIN like : TT.RoomNo = TP.RoomNo. This way even if the room number changes, new data won't be inserted as we have joined only based on patient specific data.
I want to create a DTS Package to pull data from an Oracle table into a SQL2K
table. How can I insert rows that are not already in the SQL2K table and
update rows that already exist in the SQL2K table?
I guess I could truncate and repopulate the entire table or create a
temporary table and then do updates/inserts from the temp table into the
destination table.
Is there any easier way using DTS?
Thanks,
Rokal
You can do that in a DTS package using two data driven query tasks: one for the inserts and one for the updates. The data driven query tasks are a bit of a pain to use, but they work. I've also done this (a "merge") in sql server 2000 with an AS/400 database using a dynamic t-sql. You'd write a t-sql script that outputs psql and runs it againt a linked server to the Oracle database.
UPDATE:
A DTS "data driven query task" will let you insert|update data from the sql server connection in DTS to an oracle server connection in DTS w/o a temp table or a linked server.
Update2; here's some more info on what I mean:
http://www.databasejournal.com/features/mssql/article.php/3315951
http://msdn.microsoft.com/en-us/library/aa933507(SQL.80).aspx
Are you keeping the same primary key values?
If you are you have a number of options, some versions of SQL support the MERGE statement which will update or insert just like you require.
Or you can write your own.
Something along the lines of loading all the rows into a staging table in your SQL database and row by row checking for the existence of your primary key in your main SQL table. If the key exists update the row and if not insert it.
Yes, the primary key values in the source and destination will match.
I was hoping to accomplish this task without the use of a temporary (staging) table.
Also, I am using sql server 2000 so the MERGE statement is not available.
Try:
DELETE FROM dbo.WhateverTable WHERE WhateverTableID IN (SELECT WhateverTableID FROM MySource)
It might be pretty slow, use join instead:
Delete a
from firstTable a join secondTable b on a.id = b.id
There's no way with TSQL to do a INSERT or UPDATE in the same statement, but you could very easily do it in two statements (as you mentioned above).
Statement 1:
DELETE FROM dbo.WhateverTable
WHERE WhateverTableID IN (SELECT WhateverTableID FROM MySource)
Statement 2:
INSERT INTO dbo.WhateverTable
SELECT * FROM MySource
Also, is there any reason why you don't want to use a temp table?