SQL - Insert millions of records - sql

Question

I want to optimize the insertion of millions of records in SQL server.
while #i <= 2000000 begin
set #sql = 'select #max_c= isnull(max(c), 0) from nptb_data_v7'
Execute sp_executesql #sql, N'#max_c int output', #max_c output
set #max_c= #max_c+ 1
while #j <= 10 begin
set #sql_insert = #sql_insert + '(' + cast(#i as varchar) +',' + cast(#b as varchar) + ',' + cast(#max_c as varchar)+ '),'
if len(ltrim(rtrim(#sql_insert)))>=3800
begin
set #sql_insert = SUBSTRING(#sql_insert, 1 , len(ltrim(rtrim(#sql_insert)))-1)
Execute sp_executesql #sql_insert
set #sql_insert = 'insert into dbo.nptb_data_v7 values '
end
set #b = #b + 1
if #b > 100000
begin
set #b = 1
end
set #j = #j + 1
end
set #i = #i + 1
set #j=1
end
set #sql_insert = SUBSTRING(#sql_insert, 1 , len(ltrim(rtrim(#sql_insert)))-1)
Execute sp_executesql #sql_insert
end
I want to optimize the above code as it is taking hours to complete this.

Answer 1

There are quite a few critical things I want to hit on. First, iteratively doing just about anything (especially inserting millions of rows) in SQL is almost never the right way to go. So right off the bat, we need to look at throwing that model out the window.
Second, your approach appears to be to loop over a set of numbers millions of times and add a new set of parentheses to the VALUES clause in the insert, winding you up with a ridiciulously long string of brackets which just look like (1,1,2),(1,2,2)... etc. If you WERE going to do this iteratively, you'd want to make it so that every loop just did an insert rather than building an unwielding insert string.
Third, nothing here needs dynamic SQL; not the first assignment of #max_c, nor the insertion into nptb_data_v7. You could statically construct all of these statements without having to use dynamic sql and a) obfuscate your code and b) open yourself up to injection attack.
With those out of the way, now we can get down to brass tacks. All this appears to be doing is creating combinations of an auto incrementing number between 1 and 2 million, and based on some rules, a value for #i, #b and the current iteration.
The first thing you need here is a tally table (just a big table of integers). There are tons of ways to do this, but here's a succinct script which should get you started.
http://www.sqlservercentral.com/scripts/Advanced+SQL/62486/
Once you have this table made, your script becomes a matter of self joining your newly created numbers/tally table to itself so that the rules you define are satisfied for your insert. Since it's not 100% clear what you're code is trying to get at, nor can I run it from the information provided, this is where I have to leave it up to you. If you can provide a better summary of what your tables look like, your variable declarations and your objective, I may be able to help you write some code. But hopefully this should get you started.