Send multiple commands to one SQLConnection - sql

Question

I am asking a question that is related to Execute multiple SQL commands in one round trip but not exactly the same because I am having this problem on a much bigger scale:
I have an input file with many different SQL commands (ExecuteNonQuery) that I have to process with a .net application.
Example:
INSERT INTO USERS (name, password) VALUES (#name, #pw); #name="abc"; #pw="def";
DELETE FROM USERS WHERE name=#name; #name="ghi";
INSERT INTO USERS (name, password) VALUES (#name, #pw); #name="mno"; #pw="pqr";
All of the commands have parameters so I would like the parameter mechanism that .net provides. But my application has to read these statements and execute them within an acceptable time span. There might be multiple thousand statements in one single file.
My first thought was to use SQLCommand with parameters since that would really be the way to do it properly (parameters are escaped by .net) but I can't afford to wait 50msec for each command to complete (network communication with DB server, ...). I need a way to chain the commands.
My second thought was to escape and insert the parameters myself so I could combine multiple commands in one SQLCommand:
INSERT INTO USERS (name, password) VALUES ('abc', 'def'); DELETE FROM USERS WHERE name=#name; #name='ghi'; INSERT INTO USERS (name, password) VALUES ('mno', 'pqr');
However I do feel uneasy with this solution because I don't like to escape the input myself if there are predefined functions to do it.
What would you do? Thanks for your answers, Chris

Answer 1

Assuming everything in the input is valid, what I would do is this:
Parse out the parameter names and values
Rewrite the parameter names so they are unique across all queries (i.e., so you would be able to execute two queries with a #name parameter in the same batch)
Group together a bunch of queries into a single batch and run the batches inside a transaction
The reason why you (likely) won't be able to run this all in a single batch is because there is a parameter limit of 2100 in a single batch (at least there was when I did this same thing with SQL Server); depending on the performance you get, you'll want to tweak the batch separation limit. 250-500 worked best for my workload; YMMV.
One thing I would not do is multi-thread this. If the input is arbitrary, the program has no idea if the order of the operations is important; therefore, you can't start splitting up the queries to run simultaneously.
Honestly, as long as you can get the queries to the server somehow, you're probably in good shape. With only "multiple thousands" of statements, the whole process won't take very long. (The application I wrote had to do this with several million statements.)

Answer 2

Interesting dilemma..
I would suggest any of these:
Have control over sql server? create a stored procedure that loads the file and do the work
Use sqlcommand, but then cache the parameters, and read only command type (delete, insert, etc) and the values from each line to execute. Parameter caching examples here, here, and here.
Use multiple threads.. A parent thread to read the lines and send them over to other threads: one to do the inserts, another to do the deletion, or as many as needed. Look at Tasks