Need to load huge dataset (32 Million) into table using SSIS - sql

Question

I have a huge dataset to return in SQL Server( about 32 million rows) This is implemented in view and source code is as follows :
SELECT Idenitifier = ISNULL(mle.MIdeer, mle.Ider) + em.MemberId,
EffectiveDate = ISNULL(em.EffectiveDate,
(SELECT TOP 1 EffectiveDate
FROM c
WHERE SourceType = em.SourceType
AND GroupNumber = em.GroupNumber
AND ISNULL(GroupDivision, '') =
ISNULL(em.GroupDivision, '')))
FROM a em
JOIN b mle
ON mle.Identifier = em.GroupNumber + ISNULL('-' + em.GroupDivision, '')
-- Filter invalid legal entities
AND ISNULL(mle.Filter, 0) = 0
--- Gets a resultset of 531798 rows
CROSS JOIN -- this returns 63 rows , so
-- I am presuming 531798*63 rows here.
(SELECT *
FROM map
WHERE domaintype = 'MC')b;
I need to load this dataset using SSIS into a table. After 16 million rows, I am getting a system.out of memory exception in sql server when I am giving a select * from <<view>>. How to load this dataset in table using SSIS,avoiding this exception..
What other better methods to do this query efficiently as it takes more than 30 mins to run?

Answer 1

I'm still thinking through this, but you might need to separate the CROSS JOIN:
;WITH cte AS (SELECT ISNULL(mle.MIdeer, mle.Ider) + em.MemberId AS Idenitifier
, ISNULL(em.EffectiveDate,
( SELECT TOP 1
EffectiveDate
FROM c
WHERE SourceType = em.SourceType
AND GroupNumber = em.GroupNumber
AND ISNULL(GroupDivision, '') = ISNULL(em.GroupDivision,
'')
)) AS EffectiveDate
FROM a em
JOIN b mle ON mle.Identifier = em.GroupNumber + ISNULL('-'+ em.GroupDivision,'')
AND ISNULL(mle.Filter, 0) = 0)
SELECT *
FROM cte
CROSS JOIN ( SELECT *
FROM map
WHERE domaintype = 'MC') b;