Insert Large data row by row in multiple reference tables - sql

Question

I went through a lot of posts on SO. However, they do not fit my situation.
We have a situation where we want to store a large dataset on sqlserver 2017 into multiple reference tables.
We have tried with cursor and it is working fine. However, we are concerned about the performance issue of loading large data(1+ million rows)
Example
T_Bulk is a input table, T_Bulk_Orignal is destination table and T_Bulk_reference is a reference table for t_Bulk_orignal
create table T_Bulk
(
Id uniqueidentifier,
ElementType nvarchar(max),
[Description] nvarchar(max)
)
create table T_Bulk_orignal
(
Id uniqueidentifier,
ElementType nvarchar(max),
[Description] nvarchar(max)
)
create table T_Bulk_reference
(
Id uniqueidentifier,
Description2 nvarchar(max)
)
create proc UseCursor
(
#udtT_Bulk as dbo.udt_T_Bulk READONLY
)
as
begin
DECLARE #Id uniqueidentifier, #ElementType varchar(500), #Description varchar(500),#Description2 varchar(500)
DECLARE MY_CURSOR CURSOR
LOCAL STATIC READ_ONLY FORWARD_ONLY
FOR
SELECT Id, ElementType, [Description]
FROM dbo.T_BULK
OPEN MY_CURSOR
FETCH NEXT FROM MY_CURSOR INTO #Id, #ElementType, #Description,#Description2
WHILE ##FETCH_STATUS = 0
BEGIN
BEGIN Transaction Trans1
BEgin TRy
IF EXISTS (select Id from T_Bulk_orignal where ElementType=#ElementType and Description=#Description)
select #Id = Id from T_Bulk_orignal where ElementType=#ElementType and Description=#Description
ELSE
BEGIN
insert T_Bulk_orignal(Id,ElementType,Description) values (#id, #ElementType,#Description)
END
INSERT T_Bulk_reference(Id,description2)
SELECT Id, Description2
FROM (select #Id as Id, #Description2 as Description2) F
WHERE NOT EXISTS (SELECT * FROM T_Bulk_reference C WHERE C.Id = F.Id and C.Description2 = F.Description2);
COMMIT TRANSACTION [DeleteTransaction]
FETCH NEXT FROM MY_CURSOR INTO #Id, #ElementType, #Description,#Description2
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION [Trans1]
SELECT ##Error
END CATCH
END
CLOSE MY_CURSOR
DEALLOCATE MY_CURSOR
end
We want this operation to execute in one go like bulk insertion however we also need to crosscheck any data discrepancy and if one row is not able to insert we need to rollback only that specific record
The only catch for bulk insertion is as there are reference table data present.
Please suggest best approach on this

Answer 1

This sounds like a job for SSIS (SQL Server Integration Services).
https://learn.microsoft.com/en-us/sql/integration-services/ssis-how-to-create-an-etl-package
In SSIS you can create a data migration job that can do reference checks. You can set it up to fail ,warn or ignore errors at each stage. To find resources on this google for ETL and SSIS.
I have done jobs like yours on 50+ million rows.
Sure it takes a while, and it rolls back everything (if set up like that) on an error, but it is the best tool for this kind of job.

Answer 2

I got a solution to upload large file with a go like bulk insert.
There is a Merge statement present in SQL.
The MERGE statement is used to make changes in one table based on
values matched from anther. It can be used to combine insert,
update, and delete operations into one statement
So we can pass the data using DataTable to StoredProcedure and then Source will be your UserDefinedDataTable and Target will be your actual SQL Table