I have a stored proc that I need to execute 100 times (one for each parameter). I was wondering if I could execute these all at the same time using batch or something similar so that it would speed up processing instead of executing one and then the next.
Thanks!
Can you rewrite your procedure to accept TABLE parameter, fill it with 100 values, and process table instead of 100 scalars?
To directly answer your question, you could open 100 separate connections and execute 100 separate queries concurrently.
But as has been mentioned, I don't think this is the solution for you.
As you have a table with the 100 values, it seems you have a few options...
Change the StoredProcedure to a View, and join on the view.
Change the StoredProcedure to a Table Valued Function, and use CROSS APPLY.
(Inline functions will perform a lot better than Multi-Statement functions.)
These two are limitted by the fact that neither a view nor a function can have any side-effects... No writing to tables, etc, etc.
If you can't refactor your code to use a view or function, you still need a stored procedure to encapsulate the code.
In that case, you could either:
- pass the table of values in as a Table Valued Parameter.
- or simply have the Stored Procedure read from the table directly.
Depending on your needs, you may even wish to create a table specifically for this SP to read from. This introduces a couple of extra issues though...
- Concurrency : How to keep my data separate from someone elses? Have a field to hold a unique identifier, such as the session's ##SPID.
- Clean Up : You don't want processes inserting data all day, but never deleting it.
The one thing I would strongly suggest you avoid is using loops/cursors. If you can find a set based approach, use it :)
EDIT
A comment you just left mentions that you have millions of records to process.
This makes using set based approaches much more preferable. You may, however, find that this creates extremely large transactions (if you're doing a lot of INSERTs, UPDATEs, etc). In which case, still find a set based approach, then find a way of doing this in smaller pieces (say split by day if the data is time related, or just 1000 records at a time, whatever fits.)
Related
Let me start by stating that when writing SELECT statements in a stored procedure or elsewhere in application code, I ALWAYS specify columns explicitly rather than using SELECT *.
That said, I have a case where I have many stored procedures that need exactly the same columns back because they are used to hydrate the same models on the client. In an effort to make the code less brittle and less prone forgetting to update a stored procedure with a new column, I am thinking of creating a view and selecting from that in my stored procedures using SELECT *.
To help clarify, here are examples of what the stored procedures might be named:
Entity_GetById
Entity_GetByX
Entity_GetForY
-- and on and on...
So in each of these stored procedures, I would have
SELECT *
FROM EntityView
WHERE...[criteria based on the stored procedure]
I'm wondering if there is any cost to doing this. I understand the pitfalls of SELECT * FROM Table but by selecting from a view that exists specifically to define the columns needed seems to mitigate this.
Are there any other reasons I wouldn't want to use this approach?
Personally, I don't see a problem with this approach.
However, there is a range of opinions on the use of select * in production code, which generally discourages it for the outermost query (of course, it is fine in subqueries).
You do want to make sure that the stored procedures are recompiled if there is any change to either the view or to the underlying tables supplying the view. You probably want to use schemabinding to ensure that (some) changes to the underlying tables do not inadvertently affect the view(s).
I don't know your system, but using a view would not affect performance?
SELECT * from the view makes sense, but does the view just selects specific columns from one table?
If not then look carefully into performance.
If I remember correctly in MS SQL stored procedure can return a recordset.
If I right you can try to wrap various queries into kind of sub queries stored procedure and have a one main which selects specific columns -- here complication should fail if you miss something in .
Even better would be having stored procedures which ask by various parameters and returns only primary keys (as record set or in temporary table) and one main which fetch all required columns based on returned primary keys.
I know many ways to avoid duplicating PHP code (in my case PHP). However, I am developing a rather big application that does some calculations on the database with the data it finds, and I have noticed the need to use the same code (parts of SQL) in other places.
I don't like the idea of copying and pasting the same thing over and over again. What is a good way to do this? Should I use stored procedures? I could almost calculate some of the stuff in PHP except that most of the times the queries are calculating values based on also data not returned by the query and it seems stupid to return extra data to PHP so that it could its calculations. Sometimes that may be okay, but now it does not feel so.
What should I do?
For example, all over in many SQL queries I am calculating similar to this:
...
(SELECT SUM(amount) FROM IT INNER JOIN Invoice I WHERE IT.invoiceId=I.id) AS total
...
FROM InvoiceTransaction IT
...
Note that I'm at home now so I'm writing this off the top of my head.
I think you have 2 solutions:
if the SQL returns a small amount of data, I would simply wrap the SQL invocation in a method call and call it (parameterising as necessary)
if the SQL handles a lot of data, I would keep that data in the database and use a stored procedure. You can then call that stored procedure without duplicating the code (but wrap the stored proc call in a function and call it - i.e. as in option 1)
I wouldn't necessarily shy away from stored procedures. But I would advise keeping business logic out of them (keep it in the application itself) and make sure you have sufficient unit testing around it.
I do not prefer store procedure, especially not for the sake of refactoring. You should consider writing a function that return the record you need, and put your SQL queries in that function so you can call it instead of putting your SQL everywhere.
I think we would need an example of a query. Stored procs might be a good option. Or an alternative might be to use views. One advantage in having your queries in views or stored procs is that you can often use the database to see where your tables are used. Disadvantage is that you are locking yourself into one database, however you are probably doing this anyway.
If I have a table with Field 1, Field 2, Field 3, Field 4 and for one instance need just Field 1 and Field 2, but another I need Field 3 and Field 4 and yet another need all of them...
Is it better to have a SP for each combination I need or one SP that always returns them all?
Very important question:
Writing many stored procs that run the same query will make you spend a lot of time documenting and apologising to future maintainers.
For every time anyone wants to introduce a change, they have to consider whether it should apply to all stored procs, or to some or to one only...
I would do only one stored proc.
I would just have one Stored Procedure as it will be easier to maintain.
Does it need to be a Stored Procedure? You could rewrite it as a View then simply select the columns that you need.
If network bandwidth and memory usage is more important than hours of work and project simplicity, then make a separate SP for each task. Otherwise there's no point. (the gains aren't that great, and are noticeable only when the rowset is extremely large, or there are a lot of simultaneous requests)
As a general rule it is good practice to select only the columns we need to serve a particular purpose. This is particularly true for tables which have:
lots of columns
LOB columns
sensitive or restricted data
However, if we have a complicated system with lots of tables it is obviously impractical to build a separate stored procedure for each distinct query. In fact it is probably undesirable to do so. The resultant API would be overwhelming to use and a lot of effort to maintain.
The solutions are several and various, and really depend on the nature of the applications. Views can help, although they share some of the same maintenance issues. Dynamic SQL is another approach. We can write complicated procedures which return many differnet result sets depending on the input parameters. Heck, sometimes we can even write SQL statements in the actual application.
Oh, and there is the simple procedure which basically wraps a SELECT * FROM some_table but that comes with its own suite of problems.
so I have written several stored procedures that act on individual rows of data by taking in an ID number. I would like to keep several stored procedures that can call this stored procedure at different levels of my database scheme. For instance, when a row is inserted I call this stored procedure. When something else is modified I would like to call this stored procedure for each line. This is so I can have one set of base code that can be called everywhere else but that acts on different amounts of data. I have been able to produce this result with Cursors, but I am told these are very inefficient. Is there any other way to produce this kind of functionality without sacrificing performance? Thanks.
Yes. Use standard joins to operate on sets rather than RBAR (Row By Agonising Row). i.e. Rather than call a function for each row, design a join that performs the required operation on every applicable row as a set operation.
I often see devs use the 'function operates on a each row', and although this seems to be the obvious way to encapsulate logic, it doesn't perform well on SQL Server or most DB engines.
In some circumstances, a table-valued function can be used effectively (MS SQL Server).
(BTW, you are correct in saying cursors are inefficient).
From the MSDN docs for create function:
User-defined functions cannot be used to perform actions that modify the database state.
My question is simply - why?
Yes, a UDF that modifies data may have potentially unwanted side-effects.
Yes, there is overhead involved if a UDF is called thousands of times.
But that is the whole point of design and testing - to ensure that such issues are ironed out before deployment. So why do DB vendors insist on imposing these artificial limitations on developers? What is the point of a language construct that can essentially only be used as a wrapper for select statements?
The reason for this question is as follows: I am writing a function to return a GUID for a certain unique integer ID. If a GUID is already allocated for that ID I simply return it; otherwise I want to generate a new GUID, store that into a table, and return the newly-generated GUID. (Yes, this sounds long-winded and possibly crazy, but when you're sending data to another dev company who believes their design was handed down by God and cannot be improved upon, it's easier just to smile and nod and do what they ask).
I know that I can use a stored procedure with an output parameter to achieve the same result, but then I have to declare a new variable just to hold the result of the sproc. Not only that, I then have to convert my simple select into a while loop that inserts into a temporary table, and call the sproc for every iteration of that loop.
It's usually best to think of the available tools as a spectrum, from Views, through UDFs, out to Stored Procedures. At the one end (Views) you have a lot of restrictions, but this means the optimizer can actually "see through" the code and make intelligent choices. At the other end (Stored Procedures), you've got lots of flexibility, but because you have such freedom, you lose some abilities (e.g. because you can return multiple result sets from a stored proc, you lose the ability to "compose" it as part of a larger query).
UDFs sit in a middle ground - you can do more than you can do in a view (multiple statements, for example), but you don't have as much flexibility as a stored proc. By giving up this freedom, it allows the outputs to be composed as part of a larger query. By not having side effects, you guarantee that, for example, it doesn't matter in which row order the UDF is applied in. If you could have side effects, the optimizer might have to give an ordering guarantee.
I understand your issue, I think, but taking this from your comment:
I want to do something like select my_udf(my_variable) from my_table, where my_udf either selects or creates the value it returns
So you want a select that (potentially) modifies data. Can you look at that sentence on its own and tell me that that reads perfectly OK? - I certainly can't.
Reading your description of what you actually need to do:
I am writing a function to return a
GUID for a certain unique integer ID.
If a GUID is already allocated for
that ID I simply return it; otherwise
I want to generate a new GUID, store
that into a table, and return the
newly-generated GUID.
I know that I can use a stored
procedure with an output parameter to
achieve the same result, but then I
have to declare a new variable just to
hold the result of the sproc. Not only
that, I then have to convert my simple
select into a while loop that inserts
into a temporary table, and call the
sproc for every iteration of that
loop.
from that last sentence it sounds like you have to process many rows at once, so how about a single INSERT that inserts the GUIDs for those IDs that don't already have them, followed by a single SELECT that returns all the GUIDs that (now) exist?
Sometimes if you cannot implement the solution you came up with, it may be an indication that your solution is not optimal.
Using a statement like this
INSERT INTO IntGuids(IntValue, GuidValue)
SELECT MyIntValues.IntValue, NEWID()
FROM MyIntValues
LEFT OUTER JOIN IntGuids ON MyIntValues.IntValue = IntGuids.IntValue
WHERE IntGuids.IntValue IS NULL
creates all the GUIDs you need to have in 1 statement. No need to SELECT+INSERT for every single value.