5k+ SQL inserts takes a long time - how can I improve it? - sql

Question

Usually I only run this loop with less than 50 and it runs fine. Now I am trying to scale up and with 5k+ it takes serveral minuts.
Public Shared Sub Add(type As BufferType, objIDs As List(Of Integer), userToken As String)
Dim timeInUTCSeconds As Integer = Misc.UTCDateToSeconds(Date.Now)
For Each objID As Integer In objIDs
Dim insertStmt As String = "IF NOT EXISTS (SELECT ObjectID From " & TableName(type) & " where ObjectID = " & objID & " and UserToken = '" & userToken.ToString & "')" & _
" BEGIN INSERT INTO " & TableName(type) & "(ObjectID,UserToken,time) values(" & objID & ", '" & userToken.ToString & "', " & timeInUTCSeconds & ") END" & _
" ELSE BEGIN UPDATE " & TableName(type) & " set Time = " & timeInUTCSeconds & " where ObjectID = " & objID & " and UserToken = '" & userToken.ToString & "' END"
DAL.SQL.Insert(insertStmt)
Next
End Sub

Answer 1

You should always use SQLBULKCOPY for inserting a large amount of data. You can have a look at here which has been always considered as the best practice for inserting a large amount of the data into the table.
A demo code will make your fact clear, which has been taken from here
private static void PerformBulkCopy()
{
string connectionString =
#"Server=localhost;Database=Northwind;Trusted_Connection=true";
// get the source data
using (SqlConnection sourceConnection =
new SqlConnection(connectionString))
{
SqlCommand myCommand =
new SqlCommand("SELECT * FROM tablename", sourceConnection);
sourceConnection.Open();
SqlDataReader reader = myCommand.ExecuteReader();
// open the destination data
using (SqlConnection destinationConnection =
new SqlConnection(connectionString))
{
// open the connection
destinationConnection.Open();
using (SqlBulkCopy bulkCopy =
new SqlBulkCopy(destinationConnection.ConnectionString))
{
bulkCopy.BatchSize = 500;
bulkCopy.NotifyAfter = 1000;
bulkCopy.SqlRowsCopied +=
new SqlRowsCopiedEventHandler(bulkCopy_SqlRowsCopied);
bulkCopy.DestinationTableName = "Tablename";
bulkCopy.WriteToServer(reader);
}
}
reader.Close();
}
}
Also do not forget to read the documentation related to the Batchsize
Hope that helps.

Answer 2

Best option is to move whole work to SQL side. Pass your objIDs list as table parameter to stored procedure and utilize MERGE..UPDATE..INSERT statement.
If you require VB side code, there is number of optimization you can do.
SQL Parameters. Must do.
Check DAL.SQL class if it handles connection properly.
Construct single batch from commands and run it as whole batch.

Answer 3

(I'm assuming a reasonably recent version of SQL Server on probability given it's a .Net question.)
If at all possible, move this into a stored procedure that takes either a table-valued parameter or delimited string of IDs.
Parameterise your queries. When the DBMS is executing a string of code with no parameters, it can't work out which bits are constant and which are parameters - so it works out a new execution plan each time. With parameters for the data, it'd reuse execution plans and run faster.
With really big repetitive jobs, connection latency can become a surprisingly big issue. If you really have to execute something like this as ad-hoc SQL from the calling app (which I wouldn't for this, but it's your project), try batching the statements and executing them en masse rather than issuing 5000 individual connections. Should be significantly faster.