Is it better to use Custom TABLE TYPE as parameter instead of SQL "IN" clause when passing a large comma separated value - sql

I have a stored procedure it takes comma separated string as input. Which might be too large some times approximately more than 8 thousand characters or more. In that situation, query performance goes down sometimes. And I think there is a limitation for the character length inside the IN clause. For that, sometimes I get errors. Now, I need to know is it better to use a Custom TABLE TYPE as parameter and use Inner JOIN to find the result. If it is then why is it. Here are my 2 stored procedures (minimal code):
CREATE TYPE [dbo].[INTList] AS TABLE(
[ID] [int] NULL
)
Procedure 1
CREATE PROCEDURE [report].[GetSKU]
#list [INTList] READONLY,
AS
Select sk.SKUID,sk.Code SCode,sk.SName
FROM SKUs sk
INNER JOIN #list sst ON sst.ID=sk.SKUID
Procedure 2
CREATE PROCEDURE [report].[GetSKU]
#params varchar(max),
AS
Select sk.SKUID,sk.Code SCode,sk.SName
FROM SKUs sk
WHere CHARINDEX(','+cast( sk.SKUID as varchar(MAX))+',', #params) > 0
Now, which procedures is better to use.
Note: Original Stored Procedures does have few more Joins.

As this question did raise quite some discussion in comments but did not get any viable answer, I'd like to add the major points in order to help future research.
This question is about: How do I pass a (large) list of values into a query?
In most cases, people need this either in a WHERE SomeColumn IN(SomeValueList)-filter or to JOIN against this with something like FROM MyTable INNER JOIN SomeValueList ON....
Very important is the SQL-Server's version, as with v2016 we got two great tools: native STRING_SPLIT() (not position-safe!) and JSON support.
Furthermore, and rather obvious, we have to think about the scales and values.
Do we pass in a simple list of some IDs or a huge list with thousands of values?
Do we talk about simple integers or GUIDs?
And what's about text values, where we have to think about dangerous characters (like [ { " in JSON or < & in XML - there are many more...)?
What about CSV-lists, where the separating character might appear within the content (quoting / escaping)?
In some cases we might even want to pass several columns at once...
There are several options:
Table valued parameter (TVP, CREATE TYPE ...),
CSV together with string splitting functions (native since v2016, various home brewed, CLR...),
and text-based containers: XML or JSON (since v2016)
Table valued paramter (TVP - the best choice)
A table valued parameter (TVP) must be created in advance (this might be a draw back) but will behave as any other table once created. You can add indexes, you can use it in various use cases and you do not have to bother about anything under the hood.
Sometimes we cannot use this due to missing rights to use CREATE TYPE...
Character separated values (CSV)
With CSV we see three approaches
Dynamic Sql: Create a statement, where the CSV list is simply stuffed into the IN() and execute this dynamically. This can be a very efficient approach, but will be open to various obstacles (no ad-hoc-usage, injection threat, breaking on bad values...)
String splitting functions: There are tons of examples around... All of them have in common that the separated string will be returned as a list of items. Common issues here: performance, missing ordinal position, limits for the separator, handling of duplicate or empty values, handling of quoted or escaped values, handling of separators within the content. Aaron Bertrand did some great research about the various approaches of string splitting. Similar to TVPs one draw back might be, that this function must exist in the database in advance or that we need to be allowed to execute CREATE FUNCTION if not.
ad-hoc-splitters: Before v2016 the most used approach was XML based, since then we have moved to JSON based splitters. Both use some string methods to transform the CSV string to 1) separated elements (XML) or 2) into a JSON-array. The result is queried by 1) XQuery (.value() and .nodes()) or 2) JSON's OPENJSON() or JSON_VALUE().
Text based containers
We can pass the list as string, but within a defined format:
Using ["a","b","c"] instead of a,b,c allows for immediate usage of OPENJSON().
Using <x>a</x><x>b</x><x>c</x> instead allows for XML queries.
The biggest advantage here: Any programming language provides support for these formats.
Common obstacles like date and number formatting is solved implicitly. Passing JSON or XML is - in most cases - just some few lines of code.
Both approaches allow for type- and position-safe queries.
We can solve our needs without the need to rely on anything existing in advance.

For the very best performance you can use this function:
CREATE FUNCTION [dbo].StringSplit
(
#String VARCHAR(MAX), #Separator CHAR(1)
)
RETURNS #RESULT TABLE(Value VARCHAR(MAX))
AS
BEGIN
DECLARE #SeparatorPosition INT = CHARINDEX(#Separator, #String ),
#Value VARCHAR(MAX), #StartPosition INT = 1
IF #SeparatorPosition = 0
BEGIN
INSERT INTO #RESULT VALUES(#String)
RETURN
END
SET #String = #String + #Separator
WHILE #SeparatorPosition > 0
BEGIN
SET #Value = SUBSTRING(#String , #StartPosition, #SeparatorPosition- #StartPosition)
IF( #Value <> '' )
INSERT INTO #RESULT VALUES(#Value)
SET #StartPosition = #SeparatorPosition + 1
SET #SeparatorPosition = CHARINDEX(#Separator, #String , #StartPosition)
END
RETURN
END
This function return table - select * from StringSplit('12,13,14,15,16', ',') so you can join this function to your table or can use IN on the where clause.

Related

FnSplit not working for SQL stored procedure to take multiple parameters

I have a stored procedure that currently takes in one value(ChainId) for a parameter. I am trying to allow the user to select multiple values of(ChainId). My where statement is below. Could someone help point me in a better direction than I am going now. Currently the query will run and return no data if I select multiple values for the parameter.
WHERE EndAuth is null AND CL.CHIND in(
SELECT [Value] FROM dbo.FnSplit(#ChainId, ','))
ORDER BY CL.CHIND
This is a popular function in SQL Server, so I'll assume you're working with that. Make sure your parameter is of type Varchar(MAX). #ChainId is passed as your string (ideally for SSRS) and ',' is passed as your delimiter. In SSRS, if you have a text box for your users to manually enter multiple values, they will enter something like 'value1, value2, value3'.
Test this out:
Declare #Yes_No Varchar(Max)
Set #Yes_No = 'y,n'
Select #yes_no
Select * from SplitString('y,n',',')
Select * from SplitString(#Yes_No,',')
Your results will be
y,n
----
y
n
----
y
n
Why I say to use Varchar(Max) and not int, or Varchar(10) for example, is because that would stop the function from reading all the values prematurely.
Try this:
Declare #Yes_No Varchar(1)
Set #Yes_No = 'y,n'
Select * from SplitString(#Yes_No,',')
The result will be:
y
The reason is because the function only accepts a value of 1 character in length, and splits that. As you can see, there isn't much to split.
This is just the way SSRS accepts parameters. FN_Split isn't necessarily a built-in function, but a widely popular one designed to allow you to pass multiple values to a string, with a pre-specified delimiter. So make sure you also go to your parameter in the report and specify that it will allow for multiple values. You will also want to supply a list of potential values for your users to select from. You'll either do this by manually populating a small list or providing another data source in the form of a stored procedure or table.

How do you pass values for a parameter by position when you need to check multiple values?

I created a stored procedure (spBalanceRange) with 2 optional parameters. They've been set to a default value and the sp works fine when I pass only 1 value per parameter by position. However, I have a situation where I'm trying to pass, by position, two strings immediately followed by a wildcard. I want the user to be able to search for Vendor names that start with either 'C%' or 'F%'. Here's the gist of the CREATE PROC statement:
CREATE PROC spBalanceRange
#VendorVar varchar(40) = '%',
#BalanceMin money = 1.0
...
Here's what I've tried so far, but doesn't work:
EXEC spBalanceRange '(C%|F%)', 200.00;
EXEC spBalanceRange 'C%|F%', 200.00;
Is there a way to check for 2 or more string values with a wildcard when passed by position? Thanks.
EDIT: According to your comments you are looking for the first letter of a vendor's name only.
In this special case I could suggest an easy, not well performing but really simple approach. CHARINDEX returns a number greater than zero, if a character appears within a string. So you just have to pass in all your lookup-first-characters as a simple "chain":
DECLARE #DummyVendors TABLE(VendorName VARCHAR(100));
INSERT INTO #DummyVendors VALUES
('Camel Industries')
,('Fritz and Fox')
,('some other');
DECLARE #ListOfFirstLetters VARCHAR(100)='CF';
SELECT VendorName
FROM #DummyVendors AS dv
WHERE CHARINDEX(LEFT(dv.VendorName,1),#ListOfFirstLetters)>0
This was the former answer
Checking against more than one value needs either a dedicated list of compares
WHERE val=#prm1 OR val=#prm2 OR ... (you know the count before)
...or you use the IN-clause
WHERE LEFT(VenoderName,1) IN ('C','F', ...)
...but you cannot pass the IN-list with a parameter like ... IN(#allValues)
You might think about a created TYPE to pass in all your values like a table and use an INNER JOIN as filter: https://stackoverflow.com/a/337864/5089204 (and a lot of other examples there...)
Or you might think of dynamic SQL: https://stackoverflow.com/a/5192765/5089204
And last but not least you might think of one of the many split string approaches. This is one of my own answers, section "dynamic IN-statement": https://stackoverflow.com/a/33658220/5089204
I'm answering my own question, and maybe other solutions exist but here is what had to happen with my stored procedure in order to pass variables by position:
CREATE PROC spBalanceRange
#VendorVar varchar(40) = '%',
#BalanceMin money = 1.0
AS
IF (#VendorVar = '%' AND #BalanceMin IS NULL OR #BalanceMin = '')
BEGIN
PRINT 'BalanceMin cannot be null.';
END
IF (#VendorVar = % AND #BalanceMin IS NOT NULL)
BEGIN
(sql statement using parameters)
END
EXEC spBalanceRange '[C,F]%', 200.00;
That's what I know.

SQL: SELECT number text base on a number

Background: I have an SQL database that contain a column (foo) of a text type and not integer. In the column I store integer in a text form.
Question: Is it possible to SELECT the row that contains (in foo column) number greater/lesser than n?
PS: I have a very good reason to store them as text form. Please refrain from commenting on that.
Update: (Forgot to mention) I am storing it in SQLite3.
SELECT foo
FROM Table
WHERE CAST(foo as int)>#n
select *
from tableName
where cast(textColumn as int) > 5
A simple CAST in the WHERE clause will work as long as you are sure that the data in the foo column is going to properly convert to an integer. If not, your SELECT statement will throw an error. I would suggest you add an extra step here and take out the non-numeric characters before casting the field to an int. Here is a link on how to do something similar:
http://blog.sqlauthority.com/2007/05/13/sql-server-udf-function-to-parse-alphanumeric-characters-from-string/
The only real modification you would need to do on this function would be to change the following lines:
PATINDEX('%[^0-9A-Za-z]%', #string)
to
PATINDEX('%[^0-9]%', #string)
The results from that UDF should then be castable to an int without it throwing an error. It will further slow down your query, but it will be safer. You could even put your CAST inside the UDF and make it one call. The final UDF would look like this:
CREATE FUNCTION dbo.UDF_ParseAlphaChars
(
#string VARCHAR(8000)
)
RETURNS int
AS
BEGIN
DECLARE #IncorrectCharLoc SMALLINT
SET #IncorrectCharLoc = PATINDEX('%[^0-9]%', #string)
WHILE #IncorrectCharLoc > 0
BEGIN
SET #string = STUFF(#string, #IncorrectCharLoc, 1, '')
SET #IncorrectCharLoc = PATINDEX('%[^0-9]%', #string)
END
SET #string = #string
RETURN CAST(#string as int)
END
GO
Your final SELECT statement would look something like this:
SELECT *
FROM Table
WHERE UDF_ParseAlphaChars(Foo) > 5
EDIT
Based upon the new information that the database is SQLite, the above probably won't work directly. I don't believe SQLite has native support for UDFs. You might be able to create a type of UDF using your programming language of choice (like this: http://www.christian-etter.de/?p=439)
The other option I see to safely get all of your data (an IsNumeric would exclude certain rows from your results, which might not be what you want) would probably be to create an extra column that has the int representation of the string. It is a little more dangerous in that you need to keep two fields in sync, but it will allow you to quickly sort and filter the table data.
SELECT *
FROM Table
WHERE CAST(foo as int) > 2000

how to write the store procedure for searching (CSV)?

how can i write the store procedure for searching particular string in a column of table, for given set of strings (CSV string).
like : select * from xxx where tags like ('oscar','rahman','slumdog')
how can i write the procedure for that combination of tags.
To create a comma seperated string...
You could then apply this list to Oded example to create the LIKE parts of the WHERE cluase on the fly.
DECLARE #pos int, #curruntLocation char(20), #input varchar(2048)
SELECT #pos=0
SELECT #input = 'oscar,rahman,slumdog'
SELECT #input = #input + ','
CREATE TABLE #tempTable (temp varchar(100) )
WHILE CHARINDEX(',',#input) > 0
BEGIN
SELECT #pos=CHARINDEX(',',#input)
SELECT #curruntLocation = RTRIM(LTRIM(SUBSTRING(#input,1,#pos-1)))
INSERT INTO #tempTable (temp) VALUES (#curruntLocation)
SELECT #input=SUBSTRING(#input,#pos+1,2048)
END
SELECT * FROM #tempTable
DR0P TABLE #tempTable
First off, the use of like for exact matches is sub-optimal. Might as well use =, and if doing so, you can use the IN syntax:
select * from xxx
where tags IN ('oscar', 'rahman', 'slumdog')
I am guessing you are not looking for an exact match, but for any record where the tags field contains all of the tags.
This would be something like this:
select * from xxx
where tags like '%oscar%'
and tags like '%rahman%'
and tags like '%slumdog%'
This would be not be very fast or performant though.
Think about moving this kind of logic into your application, where it is faster and easier to do.
Edit:
Following the comments - there are lots of examples on how to parse delimited strings out there. You can put these in a table and use dynamic sql to generate your query.
But, this will have bad performance and SQL Server will not be able to cache query plans for this kind of thing. As I said above - think about moving this kind of logic to application level.

Filtering With Multi-Select Boxes With SQL Server

I need to filter result sets from sql server based on selections from a multi-select list box. I've been through the idea of doing an instring to determine if the row value exists in the selected filter values, but that's prone to partial matches (e.g. Car matches Carpet).
I also went through splitting the string into a table and joining/matching based on that, but I have reservations about how that is going to perform.
Seeing as this is a seemingly common task, I'm looking to the Stack Overflow community for some feedback and maybe a couple suggestions on the most commonly utilized approach to solving this problem.
I solved this one by writing a table-valued function (we're using 2005) which takes a delimited string and returns a table. You can then join to that or use WHERE EXISTS or WHERE x IN. We haven't done full stress testing yet, but with limited use and reasonably small sets of items I think that performance should be ok.
Below is one of the functions as a starting point for you. I also have one written to specifically accept a delimited list of INTs for ID values in lookup tables, etc.
Another possibility is to use LIKE with the delimiters to make sure that partial matches are ignore, but you can't use indexes with that, so performance will be poor for any large table. For example:
SELECT
my_column
FROM
My_Table
WHERE
#my_string LIKE '%|' + my_column + '|%'
.
/*
Name: GetTableFromStringList
Description: Returns a table of values extracted from a delimited list
Parameters:
#StringList - A delimited list of strings
#Delimiter - The delimiter used in the delimited list
History:
Date Name Comments
---------- ------------- ----------------------------------------------------
2008-12-03 T. Hummel Initial Creation
*/
CREATE FUNCTION dbo.GetTableFromStringList
(
#StringList VARCHAR(1000),
#Delimiter CHAR(1) = ','
)
RETURNS #Results TABLE
(
String VARCHAR(1000) NOT NULL
)
AS
BEGIN
DECLARE
#string VARCHAR(1000),
#position SMALLINT
SET #StringList = LTRIM(RTRIM(#StringList)) + #Delimiter
SET #position = CHARINDEX(#Delimiter, #StringList)
WHILE (#position > 0)
BEGIN
SET #string = LTRIM(RTRIM(LEFT(#StringList, #position - 1)))
IF (#string <> '')
BEGIN
INSERT INTO #Results (String) VALUES (#string)
END
SET #StringList = RIGHT(#StringList, LEN(#StringList) - #position)
SET #position = CHARINDEX(#Delimiter, #StringList, 1)
END
RETURN
END
I've been through the idea of doing an
instring to determine if the row value
exists in the selected filter values,
but that's prone to partial matches
(e.g. Car matches Carpet)
It sounds to me like you aren't including a unique ID, or possibly the primary key as part of values in your list box. Ideally each option will have a unique identifier that matches a column in the table you are searching on. If your listbox was like below then you would be able to filter for specifically for cars because you would get the unique value 3.
<option value="3">Car</option>
<option value="4">Carpret</option>
Then you just build a where clause that will allow you to find the values you need.
Updated, to answer comment.
How would I do the related join
considering that the user can select
and arbitrary number of options from
the list box? SELECT * FROM tblTable
JOIN tblOptions ON tblTable.FK = ? The
problem here is that I need to join on
multiple values.
I answered a similar question here.
One method would be to build a temporary table and add each selected option as a row to the temporary table. Then you would simply do a join to your temporary table.
If you want to simply create your sql dynamically you can do something like this.
SELECT * FROM tblTable WHERE option IN (selected_option_1, selected_option_2, selected_option_n)
I've found that a CLR table-valued function which takes your delimited string and calls Split on the string (returning the array as the IEnumerable) is more performant than anything written in T-SQL (it starts to break down when you have around one million items in the delimited list, but that's much further out than the T-SQL solution).
And then, you could join on the table or check with EXISTS.