Bulk Updates Concept for SQL and VB.net winform - sql

Question

I would appreciate some ideas, discussion and feasibility of same, regarding bulk updates on a large table in SQL Server 2008+.
Currently I have a table with 10,000 rows and 160 columns. This table is updated very frequently with 1 to 100+ column changes per row depending on the process. Using the 'standard newbie' table update using a DataAdapter is very slow and unsuitable.
The quest is to find a faster way. I have tried fine-tuning the DataAdapter.Update with batch size, regardless the more heavy updates take 10-15 seconds. In the meanwhile SqlBulkCopy imports the whole table in (ball park) 1-3 seconds. When the update procedure takes place 30-50 times in a process the 10s-15s add up!
Being internet self thought, I have gaps in my experience, however there are 2 possibilities that I can think of that may be better at accomplishing the task of the update.
Dump the table content from the database and repopulate the table using SqlBulkcopy.
Using a stored procedure with a table passed to it with a merge SQL statement.
The main issue is data safety, although this is a local single user application there needs to be a way to handle errors roll back. From my understanding the dump and replace would be simpler, but perhaps more prone to data loss? The stored procedure would be far more extensive to set up as the update statement would have to have all the update columns typed individually and maintained for changes. Unless there is one 'Update *' statement :).
In trying to keep this short I want to keep this at a concept level only , but will appreciate any different ideas or links and advice.
EDIT further info:
The table has only one index, the ID column. Its a simple process of storing incoming (and changing) data to a simple datatable. and the update can be anywhere between 1 row to 1000 rows. The program stores the information to the database very often, and can be some or nearly all the columns. Building a stored procedure for each update would be impossible as I don't know which data will be updated, you can say that all of the columns will be updated (except the ID column and a few 'hard' data columns) it depends on what the update input is. So there is no fine tuning the update to specific columns unless I list nearly all of them each time. In which case one stored procedure would do it.
I think the issue is the number of 'calls' to the database are made using the current data adapter method.
EDIT:
3 WHat about a staging table where I bulk copy the data to and then have a store procedure do the update. Wouldn't that cut down the SQL trafic? I think that is the problem with the dataadapter update.
Edit: Posted an atempt of concept 1 in an answer to this thread.
Thank you

Answer 1

Dropping the table and reloading the entire thing with a bulk copy is not the correct way.
I suggest creating a stored procedure for each process that updates the table. The procedure should take as input only the columns that need to be updated for that specific process and then run a standard SQL update command to update those columns on the specified row. If possible, try to have indexes on the column(s) that you use to find the record(s) that need to be updated.
Alternately, depending on which version of the .Net framework you're using, you might try using Entity Framework if you don't want to maintain a whole list of stored procedures.

Answer 2

I have coded the following mockup to dump all rows from the table, bulkcopy the table in memory into an sql staging table and then move the data back into the original table. So by updating the data in that table.
Time taken 1.1 to 1.3 seconds
certainly a very attractive time compared to the 10-15s it takes to update. I have placed the truncete code for the staging table on top so that there is always one copy of the information in the database. Although the original table will not have the updated info untill the process completed.
What are the pitfals associate with this aproach ? What can I do about them? I must state that the table is unlikely to ever get beyond 10000 rows so the process will work.
Try
ESTP = "Start Bulk DBselection Update"
Dim oMainQueryT = "Truncate Table DBSelectionsSTAGE"
Using con As New SqlClient.SqlConnection(RacingConStr)
Using cmd As New SqlClient.SqlCommand(oMainQueryT, con)
con.Open()
cmd.ExecuteNonQuery()
con.Close()
End Using
End Using
ESTP = "Step 1 Bulk DBselection Update"
Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(RacingConStr)
bulkCopy.DestinationTableName = "DBSelectionsSTAGE"
bulkCopy.WriteToServer(DBSelectionsDS.Tables("DBSelectionsDetails"))
bulkCopy.Close()
End Using
ESTP = "Step 2 Bulk DBselection Update"
oMainQueryT = "Truncate Table DBSelections"
Using con As New SqlClient.SqlConnection(RacingConStr)
Using cmd As New SqlClient.SqlCommand(oMainQueryT, con)
con.Open()
cmd.ExecuteNonQuery()
con.Close()
End Using
End Using
ESTP = "Step 3 Bulk DBselection Update"
oMainQueryT = "Insert INTO DBSelections Select * FROM DBSelectionsSTAGE"
Using con As New SqlClient.SqlConnection(RacingConStr)
Using cmd As New SqlClient.SqlCommand(oMainQueryT, con)
con.Open()
cmd.ExecuteNonQuery()
con.Close()
End Using
End Using
Data_Base.TextBox25.Text = "Deleting data - DONE "
Data_Base.TextBox25.Refresh()
Catch ex As Exception
ErrMess = "ERROR - occured at " & ESTP & " " & ex.ToString
Call WriteError()
Call ViewError()
End Try