SSIS delete duplicate data - sql

Question

I have a problem for making a SSIS package.Can anyone help?
Here is the case:
I have two tables: A & B, and the structure is the same. But they are stored on different servers.
I have already made an SSIS package to transfer data from A to B (about one million rows at a time, which takes between one and two minutes).
After that I want to delete table A's data, after having been transfered to B. The SSIS package I wrote would follow that. I use a merge join and Conditional Split command to select the same data.
After that I use the OLE DB Command to delete Table A's data (just use "Delete RE_FormTo Where ID=?" SQLCommand to delete). It can work, but it is too slow! It took about one hour to delete the duplicate data! Does anyone know of a more efficient way of doing this?
SSIS Package Link

Answer 1

The execution is bound to be slow because of the poor SSIS package design .
Kindly refer the document Best Practices of SSIS Design
Let me explain you the mistakes which are there in your package .
1.You are using a Blocking transformation (Sort Component) .These transformations doesn't reuse the input buffer but create a new buffer for output and mostly they are slower than Synchronous components such as Lookup ,Derived Column etc which try to re use the input buffer .
As per MSDN
Do not sort within Integration Services unless it is absolutely necessary. In
order to perform a sort, Integration Services allocates the memory space of the
entire data set that needs to be transformed. If possible, presort the data before
it goes into the pipeline. If you must sort data, try your best to sort only small
data sets in the pipeline. Instead of using Integration Services for sorting, use
an SQL statement with ORDER BY to sort large data sets in the database – mark
the output as sorted by changing the Integration Services pipeline metadata
on the data
source.
2.Merge Join is a semi-blocking transformation which does hamper the performance but much less than Blocking transformation
There are 2 ways in which you can solve the issue
Use Lookup
Use Execute SQL Task and write the Merge SQL
DECLARE #T TABLE(ID INT);
Merge #TableA as target
using #TableB as source
on target.ID=source.ID
when matched then
Delete OUTPUT source.ID INTO #T;
DELETE #TableA
WHERE ID in (SELECT ID
FROM #T);

Answer 2

After you Joining both tables just insert Sort element, he gonna remove duplicates....
http://sqlblog.com/blogs/jamie_thomson/archive/2009/11/12/sort-transform-arbitration-ssis.aspx