I've used dynamic SQL for many tasks and continuously run into the same problem: Printing values of variables used inside the Dynamic T-SQL statement.
EG:
Declare #SQL nvarchar(max), #Params nvarchar(max), #DebugMode bit, #Foobar int
select #DebugMode=1,#Foobar=364556423
set #SQL='Select #Foobar'
set #Params=N'#Foobar int'
if #DebugMode=1 print #SQL
exec sp_executeSQL #SQL,#Params
,#Foobar=#Foobar
The print results of the above code are simply "Select #Foobar". Is there any way to dynamically print the values & variable names of the sql being executed? Or when doing the print, replace parameters with their actual values so the SQL is re-runnable?
I have played with creating a function or two to accomplish something similar, but with data type conversions, pattern matching truncation issues, and non-dynamic solutions. I'm curious how other developers solve this issue without manually printing each and every variable manually.
I dont believe the evaluated statement is available, meaning your example query 'Select #FooBar' is never persisted anywhere as 'Select 364556243'
Even in a profiler trace you would see the statement hit the cache as '(#Foobar int)select #foobar'
This makes sense, since a big benefit of using sp_executesql is that it is able to cache the statement in a reliable form without variables evaluated, otherwise if it replaced the variables and executed that statement we would just see the execution plan bloat.
updated: Here's a step in right direction:
All of this could be cleaned up and wrapped in a nice function, with inputs (#Statement, #ParamDef, #ParamVal) and would return the "prepared" statement. I'll leave some of that as an exercise for you, but please post back when you improve it!
Uses split function from here link
set nocount on;
declare #Statement varchar(100), -- the raw sql statement
#ParamDef varchar(100), -- the raw param definition
#ParamVal xml -- the ParamName -to- ParamValue mapping as xml
-- the internal params:
declare #YakId int,
#Date datetime
select #YakId = 99,
#Date = getdate();
select #Statement = 'Select * from dbo.Yak where YakId = #YakId and CreatedOn > #Date;',
#ParamDef = '#YakId int, #Date datetime';
-- you need to construct this xml manually... maybe use a table var to clean this up
set #ParamVal = ( select *
from ( select '#YakId', cast(#YakId as varchar(max)) union all
select '#Date', cast(#Date as varchar(max))
) d (Name, Val)
for xml path('Parameter'), root('root')
)
-- do the work
declare #pStage table (pName varchar(100), pType varchar(25), pVal varchar(100));
;with
c_p (p)
as ( select replace(ltrim(rtrim(s)), ' ', '.')
from dbo.Split(',', #ParamDef)d
),
c_s (pName, pType)
as ( select parsename(p, 2), parsename(p, 1)
from c_p
),
c_v (pName, pVal)
as ( select p.n.value('Name[1]', 'varchar(100)'),
p.n.value('Val[1]', 'varchar(100)')
from #ParamVal.nodes('root/Parameter')p(n)
)
insert into #pStage
select s.pName, s.pType, case when s.pType = 'datetime' then quotename(v.pVal, '''') else v.pVal end -- expand this case to deal with other types
from c_s s
join c_v v on
s.pName = v.pName
-- replace pName with pValue in statement
select #Statement = replace(#Statement, pName, isnull(pVal, 'null'))
from #pStage
where charindex(pName, #Statement) > 0;
print #Statement;
On the topic of how most people do it, I will only speak to what I do:
Create a test script that will run the procedure using a wide range of valid and invalid input. If the parameter is an integer, I will send it '4' (instead of 4), but I'll only try 1 oddball string value like 'agd'.
Run the values against a data set of representative size and data value distribution for what I'm doing. Use your favorite data generation tool (there are several good ones on the market) to speed this up.
I'm generally debugging like this on a more ad hoc basis, so collecting the results from the SSMS results window is as far as I need to take it.
The best way I can think of is to capture the query as it comes across the wire using a SQL Trace. If you place something unique in your query string (as a comment), it is very easy to apply a filter for it in the trace so that you don't capture more than you need.
However, it isn't all peaches & cream.
This is only suitable for a Dev environment, maybe QA, depending on how rigid your shop is.
If the query takes a long time to run, you can mitigate that by adding "TOP 1", "WHERE 1=2", or a similar limiting clause to the query string if #DebugMode = 1. Otherwise, you could end up waiting a while for it to finish each time.
For long queries where you can't add something the query string only for debug mode, you could capture the command text in a StmtStarted event, then cancel the query as soon as you have the command.
If the query is an INSERT/UPDATE/DELETE, you will need to force a rollback if #DebugMode = 1 and you don't want the change to occur. In the event you're not currently using an explicit transaction, doing that would be extra overhead.
Should you go this route, there is some automation you can achieve to make life easier. You can create a template for the trace creation and start/stop actions. You can log the results to a file or table and process the command text from there programatically.
Related
I am struggling with this. I have looked at Table Level Variables but I am thinking this is way beyond my simple understanding at this stage of SQL.
The issue I have created is I have an array of ID values I am generating inside MS Access as a result of some other tasks in there. I am wanting to send these over to SQL Server to grab the jobs with the ID number that matches.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[get_Job]
#jobID VARCHAR,
#JobIDs id_List READONLY
AS
BEGIN
SELECT #JobID AS JobID;
SELECT *
FROM Job
END;
Is my current stored procedure, however whilst I have been able to get it to return the JobID variable any list I added generates an error. If I insert only 1 ID into JobIDs, this doesn't generate a result either.
As I said I think I am punching well above my weight and am getting a bit lost in all this. Perhaps I can be directed to a better training resource or a site that explains this in baby steps or a book I can purchase to help me understand this? I would appreciate help with fixing the errors above but a fish teaching is probably better.
Thanks in advance
The issue comes down to much is how long is the list of ID's you going to pass to t-sql is the issue?
You could take the passed list (assume it is a string), say like this from Access at a PT query
exec GetHotels '1,2,3,4,5,6,7,10,20,30'
So, the above is the PT query you can/could send to sql server from Access.
So, in above, we want to return records based on above?
The T-SQL would thus become:
CREATE PROCEDURE GetHotels
#IdList nvarchar(max)
AS
BEGIN
SET NOCOUNT ON;
DECLARE #MySQL nvarchar(max)
set #MySQL = 'select * from tblHotels where ID in (' + #IdList + ')'
EXECUTE sp_executesql #mysql
END
GO
Now, in Access, say you have that array of "ID" ? You code will look like this:
Sub MyListQuery(MyList() As String)
' above assumes a array of id
' take array - convert to a string list
Dim strMyList As String
strMyList = "'" & Join(MyList, ",") & "'"
Dim rst As DAO.Recordset
With CurrentDb.QueryDefs("qryPassR")
.SQL = "GetHotels " & strMyList
Set rst = .OpenRecordset
End With
rst.MoveLast
Debug.Print rst.RecordCount
End Sub
Unfortunately, creating t-sql on the fly is a "less" then ideal approach. In most cases, because the table is not known at runtime, you have to specific add EXEC permissions to the user.
eg:
GRANT EXECUTE ON dbo.GetHotels TO USERTEST3
You find that such users can execute + run + use "most" store procedures, but in this case, you have to add specific rights with above grant due to the "table" not being known or resolved until runtime.
So, the above is a way to send a "given" array that you have, but from a general permissions point of view, and that of creating t-sql on the fly - I can't recommend this approach unless you are stuck, and have no other choice.
Edit
Here is a solution that works the same as above, but we don't have to create a SQL statement as a string.
CREATE PROCEDURE [dbo].[GetHotels2]
#IdList nvarchar(max)
AS
BEGIN
SET NOCOUNT ON;
-- create a table from the passed list
declare #List table (ID int)
while charindex(',',#IdList) > 0
begin
insert into #List (ID) values(left(#IDList,charindex(',',#IdList)-1))
set #Idlist = right(#IdList,len(#IdList)-charindex(',',#IdList))
end
insert into #List (ID) values(#IdList)
select * from tblHotels where ID in (select ID from #list)
END
You didn't show us what that table-valued parameter looks like - but assuming id_List contains a column called Id, then you need to join this TVP to your base table something like this:
ALTER PROCEDURE [dbo].[get_Job]
#jobID VARCHAR,
#JobIDs id_List READONLY
AS
BEGIN
SELECT (list of columns)
FROM Job j
INNER JOIN id_List l ON j.JobId = l.Id;
END;
Seems pretty easy to me - and not really all that difficult to handle! Agree?
Also, check out Bad habits to kick : declaring VARCHAR without (length) - you should always provide a length for any varchar variables and parameters that you use. Otherwise, as in your case - that #jobID VARCHAR parameter will be exactly ONE character long - and this is typically not what you expect / want ....
Take the (simplified) stored procedure defined here:
create procedure get_some_stuffs
#max_records int = null
as
begin
set NOCOUNT on
select top (#max_records) *
from my_table
order by mothers_maiden_name
end
I want to restrict the number of records selected only if #max_records is provided.
Problems:
The real query is nasty and large; I want to avoid having it duplicated similar to this:
if(#max_records is null)
begin
select *
from {massive query}
end
else
begin
select top (#max_records)
from {massive query}
end
An arbitrary sentinel value doesn't feel right:
select top (ISNULL(#max_records, 2147483647)) *
from {massive query}
For example, if #max_records is null and {massive query} returns less than 2147483647 rows, would this be identical to:
select *
from {massive query}
or is there some kind of penalty for selecting top (2147483647) * from a table with only 50 rows?
Are there any other existing patterns that allow for an optionally count-restricted result set without duplicating queries or using sentinel values?
I was thinking about this, and although I like the explicitness of the IF statement in your Problem 1 statement, I understand the issue of duplication. As such, you could put the main query in a single CTE, and use some trickery to query from it (the bolded parts being the highlight of this solution):
CREATE PROC get_some_stuffs
(
#max_records int = NULL
)
AS
BEGIN
SET NOCOUNT ON;
WITH staged AS (
-- Only write the main query one time
SELECT * FROM {massive query}
)
-- This part below the main query never changes:
SELECT *
FROM (
-- A little switcheroo based on the value of #max_records
SELECT * FROM staged WHERE #max_records IS NULL
UNION ALL
SELECT TOP(ISNULL(#max_records, 0)) * FROM staged WHERE #max_records IS NOT NULL
) final
-- Can't use ORDER BY in combination with a UNION, so move it out here
ORDER BY mothers_maiden_name
END
I looked at the actual query plans for each and the optimizer is smart enough to completely avoid the part of the UNION ALL that doesn't need to run.
The ISNULL(#max_records, 0) is in there because TOP NULL isn't valid, and it will not compile.
You could use SET ROWCOUNT:
create procedure get_some_stuffs
#max_records int = null
as
begin
set NOCOUNT on
IF #max_records IS NOT NULL
BEGIN
SET ROWCOUNT #max_records
END
select top (#max_records) *
from my_table
order by mothers_maiden_name
SET ROWCOUNT 0
end
There are a few methods, but as you probably notice these all look ugly or are unnecessarily complicated. Furthermore, do you really need that ORDER BY?
You could use TOP (100) PERCENT and a View, but the PERCENT only works if you do not really need that expensive ORDER BY, since SQL Server will ignore your ORDER BY if you try it.
I suggest taking advantage of stored procedures, but first lets explain the difference in the type of procs:
Hard Coded Parameter Sniffing
--Note the lack of a real parametrized column. See notes below.
IF OBJECT_ID('[dbo].[USP_TopQuery]', 'U') IS NULL
EXECUTE('CREATE PROC dbo.USP_TopQuery AS ')
GO
ALTER PROC [dbo].[USP_TopQuery] #MaxRows NVARCHAR(50)
AS
BEGIN
DECLARE #SQL NVARCHAR(4000) = N'SELECT * FROM dbo.ThisFile'
, #Option NVARCHAR(50) = 'TOP (' + #MaxRows + ') *'
IF ISNUMERIC(#MaxRows) = 0
EXEC sp_executesql #SQL
ELSE
BEGIN
SET #SQL = REPLACE(#SQL, '*', #Option)
EXEC sp_executesql #SQL
END
END
Local Variable Parameter Sniffing
IF OBJECT_ID('[dbo].[USP_TopQuery2]', 'U') IS NULL
EXECUTE('CREATE PROC dbo.USP_TopQuery2 AS ')
GO
ALTER PROC [dbo].[USP_TopQuery2] #MaxRows INT NULL
AS
BEGIN
DECLARE #Rows INT;
SET #Rows = #MaxRows;
IF #MaxRows IS NULL
SELECT *
FROM dbo.THisFile
ELSE
SELECT TOP (#Rows) *
FROM dbo.THisFile
END
No Parameter Sniffing, old method
IF OBJECT_ID('[dbo].[USP_TopQuery3]', 'U') IS NULL
EXECUTE('CREATE PROC dbo.USP_TopQuery3 AS ')
GO
ALTER PROC [dbo].[USP_TopQuery3] #MaxRows INT NULL
AS
BEGIN
IF #MaxRows IS NULL
SELECT *
FROM dbo.THisFile
ELSE
SELECT TOP (#MaxRows) *
FROM dbo.THisFile
END
PLEASE NOTE ABOUT PARAMETER SNIFFING:
SQL Server initializes variables in Stored Procs at the time of compile, not when it parses.
This means that SQL Server will be unable to guess the query and will
choose the last valid execution plan for the query, regardless of
whether it is even good.
There are two methods, hard coding an local variables that allow the Optimizer to guess.
Hard Coding for Parameter Sniffing
Use sp_executesql to not only reuse the query, but prevent SQL Injection.
However, in this type of query, will not always perform substantially better since a TOP Operator is not a column or table (so the statement effectively has no variables in this version I used)
Statistics at the time of the creation of your compiled plan will dictate how affective the method is if you are not using a variable on a predicate (ON, WHERE, HAVING)
Can use options or hint to RECOMPILE to overcome this issue.
Variable Parameter Sniffing
Variable Paramter sniffing, on the other hand, is flexible enough to work witht the statistics here, and in my own testing it seemed the variable parameter had the advantage of the query using statistics (particularly after I updated the statistics).
Ultimately, the issue of performance is about which method will use the least amount of steps to traverse through the leaflets. Statistics, the rows in your table, and the rules for when SQL Server will decide to use a Scan vs Seek impact the performance.
Running different values will show performances change significantly, though typically better than USP_TopQuery3. So DO NOT ASSUME one method is necessarily better than the other.
Also note you can use a table-valued function to do the same, but as Dave Pinal would say:
If you are going to answer that ‘To avoid repeating code, you use
Function’ ‑ please think harder! Stored procedure can do the same...
if you are going to answer
with ‘Function can be used in SELECT, whereas Stored Procedure cannot
be used’ ‑ again think harder!
SQL SERVER – Question to You – When to use Function and When to use Stored Procedure
You could do it like this (using your example):
create procedure get_some_stuffs
#max_records int = null
as
begin
set NOCOUNT on
select top (ISNULL(#max_records,1000)) *
from my_table
order by mothers_maiden_name
end
I know you don't like this (according to your point 2), but that's pretty much how it's done (in my experience).
How about something like this (you're have to really look at execution plans and I didn't have time to set anything up)?
create procedure get_some_stuffs
#max_records int = null
as
begin
set NOCOUNT on
select *, ROW_NUMBER(OVER order by mothers_maiden_name) AS row_num
from {massive query}
WHERE #max_records IS NULL OR row_num < #max_records
end
Another thing you can do with {massive query} is make a view or inline table-valued function (it it's parametrized), which is generally a pretty good practice for anything big and repetitively used.
I have a stored procedure that uses several parameters to build a dynamic query, which I execute. The query works fine, however, this procedure will be the data source for a Crystal Report which needs a "static" SELECT with field names it can reference. The Crystal Report is called from a Visual Basic application, and gets it's parameters passed to it from the application. It, in turn, passes the parameters to the SQL Server stored procedure.
Somehow I need to
SELECT fieldname1, fieldname2
FROM Exec(#MydynamcSQL)
after I build #MydynamcSQL. It is a complicated application accessing specific tables based on year, and specific databases based on the user. I am pretty new to SQL, so maybe there are other methods I could use that I am unaware of?
Try creating a temporary table to insert the data temporarily, then select from that table:
DECLARE #MydynamcSQL varchar(1000);
SET #MydynamcSQL = 'select fieldname1, fieldname1 from table1';
CREATE TABLE #Result
(
fieldname1 varchar(1000),
fieldname2 varchar(1000)
)
INSERT #Result Exec(#MydynamcSQL)
SELECT fieldname1, fieldname1 -- here you have "static SELECT with field names"
FROM #Result
DROP TABLE #Result
Did you try making the who thing dynamic, such as:
Exec( 'SELECT fieldname1, fieldname2 FROM ' + #MydynamcSQL)
It's worth noting although out of scope, ensure you are not vulnarable to sql injection attacks. A parameterized dynamic query potentially leaves you exposed.
Summary
Is there an efficient way to run large numbers of Dynamic SQL (on SQL Server 2005)?
Details
Our system allows users to create "email alert" subscriptions - where new matches on the system are emailed to them on a daily basis.
The subscription allows for multiple options, including the use of search keywords. A parser written by myself outputs the appropriate SQL code, taking into account and, or and brackets (). The parser will not allow anything through that could be used for SQL Injection.
For example, the keywords might be entered by the user as this (that or other) and the resultant query would end up roughly as...
SELECT *
FROM [VW_EMAIL_ALERT]
WHERE ([SEARCH] LIKE '%this%' AND ([SEARCH] LIKE '%that%' OR [SEARCH] LIKE '%other%'))
Each night, all those subscriptions are processed individually, because each one is potentially unique. The result is that the batch processing has to run a cursor over every subscription and run the SQL through sp_executesql.
Obviously this is highly inefficient, and can cause serious overloading - leading in some cases to timeouts. The stored-procedure that runs this processing is coded to split the subscriptions into blocks, so they're not all being called at once.
Is there a better/more efficient way to do this?
Note: Unfortunately we are currently stuck supporting a minimum of SQL Server 2005, as some of our clients still use that technology
If you are looking for keywords that is the least efficient way you could do it
A like '%anything does not use an index
Use a FullText search to index the words
Or write you own parser to index the unique words
You would build up a keywords table
And index the keyword
This is a very efficient query
select id
from keywords
where keyword = 'this'
intersect
select id
from keywords
where keyword in ( 'that','other')
Even with wildcards in the keywords it is still much more efficient than searching the entire text
I hope this will help. At my work, we replaced cursor with this kind of implementation.
DECLARE
#strSQL NVARCHAR(MAX) = ''
CREATE TABLE #tmp
(
Result_Query VARCHAR(MAX)
)
INSERT INTO
#tmp
SELECT
'SELECT * FROM [VW_EMAIL_ALERT] WHERE ([SEARCH] = ''%this%'' AND ([SEARCH] = ''%that%'' OR [SEARCH] = ''%other%''))'
UNION
SELECT
'SELECT * FROM [VW_EMAIL_ALERT] WHERE ([SEARCH] = ''%this1%'' AND ([SEARCH] = ''%that1%'' OR [SEARCH] = ''%other1%''))'
SELECT
#strSQL = #strSQL + Result_Query + ';'
FROM
#tmp
SET
#strSQL = LEFT(#strSQL, LEN(#strSQL) - 1)
PRINT #strSQL
EXEC(#strSQL)
We all know that prepared statements are one of the best way of fending of SQL injection attacks. What is the best way of creating a prepared statement with an "IN" clause. Is there an easy way to do this with an unspecified number of values? Take the following query for example.
SELECT ID,Column1,Column2 FROM MyTable WHERE ID IN (1,2,3)
Currently I'm using a loop over my possible values to build up a string such as.
SELECT ID,Column1,Column2 FROM MyTable WHERE ID IN (#IDVAL_1,#IDVAL_2,#IDVAL_3)
Is it possible to use just pass an array as the value of the query paramter and use a query as follows?
SELECT ID,Column1,Column2 FROM MyTable WHERE ID IN (#IDArray)
In case it's important I'm working with SQL Server 2000, in VB.Net
Here you go - first create the following function...
Create Function [dbo].[SeparateValues]
(
#data VARCHAR(MAX),
#delimiter VARCHAR(10)
)
RETURNS #tbldata TABLE(col VARCHAR(10))
As
Begin
DECLARE #pos INT
DECLARE #prevpos INT
SET #pos = 1
SET #prevpos = 0
WHILE #pos > 0
BEGIN
SET #pos = CHARINDEX(#delimiter, #data, #prevpos+1)
if #pos > 0
INSERT INTO #tbldata(col) VALUES(LTRIM(RTRIM(SUBSTRING(#data, #prevpos+1, #pos-#prevpos-1))))
else
INSERT INTO #tbldata(col) VALUES(LTRIM(RTRIM(SUBSTRING(#data, #prevpos+1, len(#data)-#prevpos))))
SET #prevpos = #pos
End
RETURN
END
then use the following...
Declare #CommaSeparated varchar(50)
Set #CommaSeparated = '112,112,122'
SELECT ID,Column1,Column2 FROM MyTable WHERE ID IN (select col FROM [SeparateValues](#CommaSeparated, ','))
I think sql server 2008 will allow table functions.
UPDATE
You'll squeeze some extra performance using the following syntax...
SELECT ID,Column1,Column2 FROM MyTable
Cross Apply [SeparateValues](#CommaSeparated, ',') s
Where MyTable.id = s.col
Because the previous syntax causes SQL Server to run an extra "Sort" command using the "IN" clause. Plus - in my opinion it looks nicer :D!
If you would like to pass an array, you will need a function in sql that can turn that array into a sub-select.
These functions are very common, and most home grown systems take advantage of them.
Most commercial, or rather professional ORM's do ins by doing a bunch of variables, so if you have that working, I think that is the standard method.
You could create a temporary table TempTable with a single column VALUE and insert all IDs. Then you could do it with a subselect:
SELECT ID,Column1,Column2 FROM MyTable WHERE ID IN (SELECT VALUE FROM TempTable)
Go with the solution posted by digiguru. It's a great reusable solution and we use the same technique as well. New team members love it, as it saves time and keeps our stored procedures consistent. The solution also works well with SQL Reports, as the parameters passed to stored procedures to create the recordsets pass in varchar(8000). You just hook it up and go.
In SQL Server 2008, they finally got around to addressing this classic problem by adding a new "table" datatype. Apparently, that lets you pass in an array of values, which can be used in a sub-select to accomplish the same as an IN statement.
If you're using SQL Server 2008, then you might look into that.
Here's one technique I use
ALTER Procedure GetProductsBySearchString
#SearchString varchar(1000),
as
set nocount on
declare #sqlstring varchar(6000)
select #sqlstring = 'set nocount on
select a.productid, count(a.productid) as SumOf, sum(a.relevence) as CountOf
from productkeywords a
where rtrim(ltrim(a.term)) in (''' + Replace(#SearchString,' ', ''',''') + ''')
group by a.productid order by SumOf desc, CountOf desc'
exec(#sqlstring)