I would really like some advice here, to give some background info I am working with inserting Message Tracking logs from Exchange 2007 into SQL. As we have millions upon millions of rows per day I am using a Bulk Insert statement to insert the data into a SQL table.
In fact I actually Bulk Insert into a temp table and then from there I MERGE the data into the live table, this is for test parsing issues as certain fields otherwise have quotes and such around the values.
This works well, with the exception of the fact that the recipient-address column is a delimited field seperated by a ; character, and it can be incredibly long sometimes as there can be many email recipients.
I would like to take this column, and split the values into multiple rows which would then be inserted into another table. Problem is anything I am trying is either taking too long or not working the way I want.
Take this example data:
message-id recipient-address
2D5E558D4B5A3D4F962DA5051EE364BE06CF37A3A5#Server.com user1#domain1.com
E52F650C53A275488552FFD49F98E9A6BEA1262E#Server.com user2#domain2.com
4fd70c47.4d600e0a.0a7b.ffff87e1#Server.com user3#domain3.com;user4#domain4.com;user5#domain5.com
I would like this to be formatted as followed in my Recipients table:
message-id recipient-address
2D5E558D4B5A3D4F962DA5051EE364BE06CF37A3A5#Server.com user1#domain1.com
E52F650C53A275488552FFD49F98E9A6BEA1262E#Server.com user2#domain2.com
4fd70c47.4d600e0a.0a7b.ffff87e1#Server.com user3#domain3.com
4fd70c47.4d600e0a.0a7b.ffff87e1#Server.com user4#domain4.com
4fd70c47.4d600e0a.0a7b.ffff87e1#Server.com user5#domain5.com
Does anyone have any ideas about how I can go about doing this?
I know PowerShell pretty well, so I tried in that, but a foreach loop even on 28K records took forever to process, I need something that will run as quickly/efficiently as possible.
Thanks!
If you are on SQL Server 2016+
You can use the new STRING_SPLIT function, which I've blogged about here, and Brent Ozar has blogged about here.
SELECT s.[message-id], f.value
FROM dbo.SourceData AS s
CROSS APPLY STRING_SPLIT(s.[recipient-address], ';') as f;
If you are still on a version prior to SQL Server 2016
Create a split function. This is just one of many examples out there:
CREATE FUNCTION dbo.SplitStrings
(
#List NVARCHAR(MAX),
#Delimiter NVARCHAR(255)
)
RETURNS TABLE
AS
RETURN (SELECT Number = ROW_NUMBER() OVER (ORDER BY Number),
Item FROM (SELECT Number, Item = LTRIM(RTRIM(SUBSTRING(#List, Number,
CHARINDEX(#Delimiter, #List + #Delimiter, Number) - Number)))
FROM (SELECT ROW_NUMBER() OVER (ORDER BY s1.[object_id])
FROM sys.all_objects AS s1 CROSS APPLY sys.all_objects) AS n(Number)
WHERE Number <= CONVERT(INT, LEN(#List))
AND SUBSTRING(#Delimiter + #List, Number, 1) = #Delimiter
) AS y);
GO
I've discussed a few others here, here, and a better approach than splitting in the first place here.
Now you can extrapolate simply by:
SELECT s.[message-id], f.Item
FROM dbo.SourceData AS s
CROSS APPLY dbo.SplitStrings(s.[recipient-address], ';') as f;
Also I suggest not putting dashes in column names. It means you always have to put them in [square brackets].
SQL Server 2016 include a new table function string_split(), similar to the previous solution.
The only requirement is Set compatibility level to 130 (SQL Server 2016)
You may use CROSS APPLY (available in SQL Server 2005 and above) and STRING_SPLIT function (available in SQL Server 2016 and above):
DECLARE #delimiter nvarchar(255) = ';';
-- create tables
CREATE TABLE MessageRecipients (MessageId int, Recipients nvarchar(max));
CREATE TABLE MessageRecipient (MessageId int, Recipient nvarchar(max));
-- insert data
INSERT INTO MessageRecipients VALUES (1, 'user1#domain.com; user2#domain.com; user3#domain.com');
INSERT INTO MessageRecipients VALUES (2, 'user#domain1.com; user#domain2.com');
-- insert into MessageRecipient
INSERT INTO MessageRecipient
SELECT MessageId, ltrim(rtrim(value))
FROM MessageRecipients
CROSS APPLY STRING_SPLIT(Recipients, #delimiter)
-- output results
SELECT * FROM MessageRecipients;
SELECT * FROM MessageRecipient;
-- delete tables
DROP TABLE MessageRecipients;
DROP TABLE MessageRecipient;
Results:
MessageId Recipients
----------- ----------------------------------------------------
1 user1#domain.com; user2#domain.com; user3#domain.com
2 user#domain1.com; user#domain2.com
and
MessageId Recipient
----------- ----------------
1 user1#domain.com
1 user2#domain.com
1 user3#domain.com
2 user#domain1.com
2 user#domain2.com
for table = "yelp_business", split the column categories values separated by ; into rows and display as category column.
SELECT unnest(string_to_array(categories, ';')) AS category
FROM yelp_business;
Related
I have created a temp table, the idea being that I want to loop through it, match all records with the same email address and then populate a string, which will go in to an email, then drop the table. This will be run as a stored procedure.
I've used a cursor that first grabbed all the unique email addresses and then coalesce the records but with potentially 100k-500k records performance won't be acceptable, and I know there must be a far more efficient way of doing it.
Example data (apologies, don't know how to format it properly)
#temptable
temp_email, temp_string
test#test.com string1
test#test.com string2
test2#test.com string3
test2#test.com string4
test3#test.com string5
I then want to populate another table with this data
emailto... emailbody
test#test.com 'string1<br / > string2'
test2#test.com 'string3<br / > string4'
test3#test.com 'string5'
Thank you.
The STUFF and FOR XML PATH method achieves this nicely in SQl Server 2014 and prior. Because you have the characters < and > however, they need to be "un-escaped" afterwards:
WITH VTE AS(
SELECT *
FROM (VALUES('test#test.com','string1'),
('test#test.com','string2'),
('test2#test.com','string3'),
('test2#test.com','string4'),
('test3#test.com','string5')) V(Email, String))
SELECT Email,
STUFF(REPLACE(REPLACE((SELECT '<br/>' + sq.String
FROM VTE sq
WHERE sq.Email = V.Email
FOR XML PATH('')),'<','<'),'>','>'),1,5,'')
FROM VTE V
GROUP BY Email;
you dont need to use cursor, please use string_agg function.
Create table #temptable
(temp_email varchar(50), temp_string varchar(50))
INSERT INTO #temptable
VALUES ('test#test.com', 'string1'),
('test#test.com', 'string2'),
('test2#test.com', 'string3'),
('test2#test.com', 'string4'),
('test3#test.com', 'string5')
Select temp_email, STRING_AGG(temp_string,' <br/>')
from #temptable
Group by temp_email
In SQL Server 2014 there are ways of doing this without a cursor but they are basically quite convoluted hacks and lead to pretty unreadable SQL in my opinion. See here for details:
How to concatenate text from multiple rows into a single text string in SQL server?
A cursor is arguably the best way in SQL 2014 because at least its readable.
In Sql Server 2017 there is an official aggregation function for this:
String_Agg
... but thats no use to you at the mo. Sorry.
You can do something like this :
-- Create temp table
CREATE TABLE #temptable (temp_email varchar(50), temp_string varchar(50))
-- populate table with data
INSERT INTO #temptable
VALUES ('test#test.com', 'string1'),
('test#test.com', 'string2'),
('test2#test.com', 'string3'),
('test2#test.com', 'string4'),
('test3#test.com', 'string5')
-- actual query
;WITH CTE_table AS(
SELECT C.temp_email,
REPLACE(REPLACE(STUFF(
(SELECT CAST(temp_string AS VARCHAR(20))+'<br/>' AS [text()]
FROM #temptable AS O
WHERE C.temp_email= o.temp_email
FOR XML PATH('')), 1, 0, NULL)
,'<','<') -- replace this < with html code <
,'>','>') -- replace this > with html code >
AS temp_string
,ROW_NUMBER() OVER (partition by temp_email order by temp_email) rownumber
FROM #temptable AS C
)
-- Get only unique records
SELECT temp_email,temp_string FROM CTE_table
Where rownumber=1
I am trying to create a piece of code in sql server 2008 that will grab specific values from each distinct string within my dbo table. The ultimate goal is to make a drop down box within Visual Studio so that one can choose all lines from the database that contain a specific product code (see definition of product code below). Example strings:
in_0314_95pf_500_w_0315
in_0314_500_95pf_0315_w
The part of these strings I am wishing to identify is the 3 digit numeric code (in this case let us call it product code) that appears once within each string. There are roughly 300 different product codes.
The problem is that these product code values do not appear in the same position within each unique string. Hence, I am having a hard time determining the product code because I can't use substring, charindex, like, etc.
Any ideas? Any help is MUCH appreciated.
This can be done with PATINDEX:
DECLARE #s NVARCHAR(100) = 'in_0314_95pf_500_w_0315'
SELECT SUBSTRING(#s, PATINDEX('%[_][0-9][0-9][0-9][_]%', #s) + 1, 3)
Output:
500
If there are no underscores then:
SELECT SUBSTRING(#s, PATINDEX('%[^0-9][0-9][0-9][0-9][^0-9]%', #s) + 1, 3)
This means 3 digits between any symbols that are not digits.
EDIT:
Apply to table like:
SELECT SUBSTRING(ColumnName, PATINDEX('%[^0-9][0-9][0-9][0-9][^0-9]%', ColumnName) + 1, 3)
FROM TableName
One approach is to use a String splitting table function like this one which breaks the string up into its components. You can then filter the components based on your criteria:
SELECT Name
FROM dbo.splitstring('in_0314_95pf_500_w_0315', '_')
WHERE ISNUMERIC(Name) = 1 AND LEN(Name) = 3;
I've amended the function slightly to accept the delimiter as a parameter.
CREATE FUNCTION dbo.splitstring ( #stringToSplit VARCHAR(MAX), #delimiter VARCHAR(50))
RETURNS
#returnList TABLE ([Name] [nvarchar] (500))
AS
BEGIN
DECLARE #name NVARCHAR(255)
DECLARE #pos INT
WHILE CHARINDEX(#delimiter, #stringToSplit) > 0
BEGIN
SELECT #pos = CHARINDEX(#delimiter, #stringToSplit)
SELECT #name = SUBSTRING(#stringToSplit, len(#delimiter), #pos-len(#delimiter))
INSERT INTO #returnList
SELECT #name
SELECT #stringToSplit = SUBSTRING(#stringToSplit, #pos+LEN(#delimiter),
LEN(#stringToSplit)-#pos)
END
INSERT INTO #returnList
SELECT #stringToSplit
RETURN
END
To apply this to your table, use CROSS APPLY (Single Delimiter):
SELECT mt.Name, x.Name AS ProductCode
FROM MyTable mt
CROSS APPLY dbo.splitstring(mt.Name, '_') x
WHERE ISNUMERIC(x.Name) = 1 AND LEN(x.Name) = 3
Update, Multiple Delimiters
I guess the real underlying problem is that ultimately the product codes need to be normalized out of the composite key (e.g. add a distinct ProductId or ProductCode column to the same table), derived using a query like this, and then stored back in the table via an update. Reverse engineering the product codes out of the string appears to be a trial and error process.
Nonetheless, you can continue to keep passing the split strings through further splitting functions (one per each type of delimiter), before applying your final discriminating filter:
SELECT *
FROM MyTable mt
CROSS APPLY dbo.splitstring(mt.Name, 'test') y -- First alias
CROSS APPLY dbo.splitstring(y.Name, '_') x -- Reference the preceding alias
WHERE ISNUMERIC(x.Name) = 1 AND LEN(x.Name) = 3; -- Must reference the last alias (x)
Note that the stringsplit function has again been changed to accommodate multicharacter delimiters.
If you have a table (or can generate in inline view) of the product codes, you can join the list of long strings to the product codes with a like clause.
Create Table longcodes (
longcode varchar(20)
)
Create Table products (
prodCode char(3)
)
insert products values('100')
insert products values('111')
insert products values('123')
insert longcodes values ('abc_a_100_test')
insert longcodes values ('asdf_111_bob')
insert longcodes values ('in_0314_123_95pf')
insert longcodes values ('f_100_u')
insert longcodes values ('hihi_111_bye')
insert longcodes values ('in_123_0314_95pf')
insert longcodes values ('a_b__c_d_100_efg')
select *
from products p
join longcodes l on l.longcode like '%_' + p.prodCode + '_%'
And they get aligned with the product codes like this:
prodCode longcode
100 abc_a_100_test
100 f_100_u
100 a_b__c_d_100_efg
111 asdf_111_bob
111 hihi_111_bye
123 in_0314_123_95pf
123 in_123_0314_95pf
EDIT: Seeing the developments in the other answer, you can simplify the like clause to
like p.prodCode
and just deal with the fact that you have a much greater chance of a single composite string producing multiple matches.
I have a product table with a tag column, each product has multiple tags stored in this format: "|technology|mobile|acer|laptop|" ...second product's tags could look like this "|computer|laptop|toshiba|"
I am using MS SQL Server 2008 and stored procedure, I would like to know how I could pass a string like "|computer|laptop|" and get both records returned as they both have the tag laptop in them and if I passed "|computer|" only the second record would return as it is the only one comtainning that tag.
What is the best way of doing this without performance penalties using stored procedure?
I have so far had no luck with different codes i have found on the internet, I really hope you guys can maybe help me with this, thank you.
I agree with the other posters that storing data in a column like that is going to cause headaches. You really want to store those tags in a child table so you can easily and efficiently join them. If it's an inherited system or something you can't refactor right away you can write a split function.
The typical sql split implementation uses a while loop and a table variable in a multi-statement TVF. Every iteration incurs more I/O and CPU overhead. Performance testing on SQL 2005 SP1 showed that this overhead is hidden from the I/O Stats and query plan. Profiling the code will reveal the true cost.
Rewriting that function into a inline TVF is much more efficient. The primary difference between an inline and multi-statement TVF is the Query Optimizer will merge the inline function into the query before processing; this eliminates the overhead from the function call. Also, since there is no table variable required, the additional I/O cost is eliminated. Finally, you avoid the costly iterative processing.
Here is the fastest, most scalable split function I could come up with including unit tests and summary.
This function requires a numbers table:
CREATE TABLE dbo.Numbers
(
NUM INT PRIMARY KEY CLUSTERED
)
;WITH Nbrs ( n ) AS
(
SELECT 1 UNION ALL
SELECT 1 + n FROM Nbrs WHERE n < 10000
)
INSERT INTO dbo.Numbers
SELECT n FROM Nbrs
OPTION ( MAXRECURSION 10000 )
The source of the function is here:
IF EXISTS (
SELECT 1
FROM dbo.sysobjects
WHERE id = object_id(N'[dbo].[ParseString]')
AND xtype in (N'FN', N'IF', N'TF'))
BEGIN
DROP FUNCTION [dbo].[ParseString]
END
GO
CREATE FUNCTION dbo.ParseString (#String VARCHAR(8000), #Delimiter VARCHAR(10))
RETURNS TABLE
AS
/*******************************************************************************************************
* dbo.ParseString
*
* Creator: MagicMike
* Date: 9/12/2006
*
*
* Outline: A set-based string tokenizer
* Takes a string that is delimited by another string (of one or more characters),
* parses it out into tokens and returns the tokens in table format. Leading
* and trailing spaces in each token are removed, and empty tokens are thrown
* away.
*
*
* Usage examples/test cases:
Single-byte delimiter:
select * from dbo.ParseString2('|HDI|TR|YUM|||', '|')
select * from dbo.ParseString('HDI| || TR |YUM', '|')
select * from dbo.ParseString(' HDI| || S P A C E S |YUM | ', '|')
select * from dbo.ParseString2('HDI|||TR|YUM', '|')
select * from dbo.ParseString('', '|')
select * from dbo.ParseString('YUM', '|')
select * from dbo.ParseString('||||', '|')
select * from dbo.ParseString('HDI TR YUM', ' ')
select * from dbo.ParseString(' HDI| || S P A C E S |YUM | ', ' ') order by Ident
select * from dbo.ParseString(' HDI| || S P A C E S |YUM | ', ' ') order by StringValue
Multi-byte delimiter:
select * from dbo.ParseString('HDI and TR', 'and')
select * from dbo.ParseString('Pebbles and Bamm Bamm', 'and')
select * from dbo.ParseString('Pebbles and sandbars', 'and')
select * from dbo.ParseString('Pebbles and sandbars', ' and ')
select * from dbo.ParseString('Pebbles and sand', 'and')
select * from dbo.ParseString('Pebbles and sand', ' and ')
*
*
* Notes:
1. A delimiter is optional. If a blank delimiter is given, each byte is returned in it's own row (including spaces).
select * from dbo.ParseString3('|HDI|TR|YUM|||', '')
2. In order to maintain compatibility with SQL 2000, ident is not sequential but can still be used in an order clause
If you are running on SQL2005 or later
SELECT Ident, StringValue FROM
with
SELECT Ident = ROW_NUMBER() OVER (ORDER BY ident), StringValue FROM
*
*
* Modifications
*
*
********************************************************************************************************/
RETURN (
SELECT Ident, StringValue FROM
(
SELECT Num as Ident,
CASE
WHEN DATALENGTH(#delimiter) = 0 or #delimiter IS NULL
THEN LTRIM(SUBSTRING(#string, num, 1)) --replace this line with '' if you prefer it to return nothing when no delimiter is supplied. Remove LTRIM if you want to return spaces when no delimiter is supplied
ELSE
LTRIM(RTRIM(SUBSTRING(#String,
CASE
WHEN (Num = 1 AND SUBSTRING(#String,num ,DATALENGTH(#delimiter)) <> #delimiter) THEN 1
ELSE Num + DATALENGTH(#delimiter)
END,
CASE CHARINDEX(#Delimiter, #String, Num + DATALENGTH(#delimiter))
WHEN 0 THEN LEN(#String) - Num + DATALENGTH(#delimiter)
ELSE CHARINDEX(#Delimiter, #String, Num + DATALENGTH(#delimiter)) - Num -
CASE
WHEN Num > 1 OR (Num = 1 AND SUBSTRING(#String,num ,DATALENGTH(#delimiter)) = #delimiter)
THEN DATALENGTH(#delimiter)
ELSE 0
END
END
)))
End AS StringValue
FROM dbo.Numbers
WHERE Num <= LEN(#String)
AND (
SUBSTRING(#String, Num, DATALENGTH(ISNULL(#delimiter,''))) = #Delimiter
OR Num = 1
OR DATALENGTH(ISNULL(#delimiter,'')) = 0
)
) R WHERE StringValue <> ''
)
For your case, you could use it like this:
--SAMPLE DATA
CREATE TABLE #products
(
productid INT IDENTITY PRIMARY KEY CLUSTERED ,
prodname VARCHAR(200),
tags VARCHAR(200)
)
INSERT INTO #products (prodname, tags)
SELECT 'toshiba laptop', '|laptop|toshiba|notebook|'
UNION ALL
SELECT 'toshiba netbook', '|netbook|toshiba|'
UNION ALL
SELECT 'Apple macbook', '|laptop|apple|notebook|'
UNION ALL
SELECT 'Apple mouse', '|apple|mouse'
--Actual solution
DECLARE #searchTags VARCHAR(200)
SET #searchTags = '|apple|laptop|' --This would the string that would get passed in if it were a stored procedure
--First we convert the supplied tags into a table for use later
--My (2005) dev box raised a severe error attempting to do the search in 1 step
--hence the temp table
CREATE TABLE #tags
(
tag VARCHAR(200) PRIMARY KEY CLUSTERED
)
INSERT INTO #tags --The function splits the string up into one record for each value
SELECT stringValue
FROM dbo.parsestring(#searchTags,'|') --SQL 2005 has a real problem joining to a TVF twice, apparently
SELECT DISTINCT p.*
FROM #products P --we join the products table with the function to get a row for each tag so we can compare with the temp table
CROSS APPLY (SELECT stringValue FROM dbo.parsestring(P.tags,'|')) T
WHERE EXISTS(SELECT * FROM #tags WHERE tag = T.stringValue) --we compare the rows with our temp table and if we get matches, the products are returned
/*This will return the Apple Macbook and the Toshiba Laptop because they both contain
the 'laptop' tag and the Apple mouse because it contains the 'apple' tag. The
toshiba netbook contains neither tag so it won't be returned.*/
But, with your tags in a separate table as suggested (1-many for a simplified example) It would look like this:
SELECT * FROM Products P
WHERE EXISTS (SELECT *
FROM tags T
INNER JOIN dbo.parsestring(#tags,'|') Q
ON T.tag = Q.StringValue
WHERE T.productid = P.productiId )
You have a many-to-many relationship between products and tags. The best way of doing this is to redesign your database. Create a table of tags and a junction table that links products with tags.
That's not a very good design. Combining like terms into one field and separating them with a delimiter such as a vertical bar does not scale well and it is very limiting.
I recommend you read up on how to design databases. The best book I ever bought regarding database design was Database Design for Mere Mortals by Michael Hernandez ISBN: 0-201-69471-9. Amazon Listing I noticed he has a second edition.
He walks you through the entire process of (from start to finish) of designing a database. I recommend you start with this book.
You have to learn to look at things in groups or chunks. Database design has simple building blocks just like programming does. If you gain a thorough understanding of these simple building blocks you can tackle any database design.
In programming you have:
If Constructs
If Else Constructs
Do While Loops
Do Until Loops
Case Constructs
With databases you have:
Data Tables
Lookup Tables
One to One relationships
One to Many Relationships
Many to Many relationships
Primary keys
Foreign keys
The simpler you make things the better. A database is nothing more than a place where you put data into cubbie holes. Start by identifying what these cubbie holes are and what kind of stuff you want in them.
You are never going to create the perfect database design the first time you try. This is a fact. Your design will go through several refinements during the process. Sometimes things won't seem apparent until you start entering data, and then you have an ahh ha moment.
The web brings it's own sets of challenges. Bandwith issues. Statelessness. Erroneous data from processes that start but never get finished.
make a split with CLR function return a table with the value or pass as xml and load it into a table varible an make a join
create procedure search
(
#data xml
)
AS
BEGIN
--declare #data xml
declare #LoadData table
(
dataToFind varchar(max)
)
--set #data= cast(
--'<data>
-- <item>computer</item>
-- <item>television</item>
--</data>' as xml)
insert into #LoadData
SELECT T2.Loc.value('.','varchar(max)')
FROM (select #data as data )T
CROSS APPLY data.nodes('/data/item') as T2(Loc)
select * from #LoadData--use for join
END
I would suggest you write an extra couplle of tables that with "proper design,
Populate those tables from the existing not well designed bit - this way y our search will work properly buy others using the old | pipe approach won't notice till you have time to refactor
I have a row of strings that are in the following format:
'Order was assigned to lastname,firsname'
I need to cut this string down into just the last and first name but it is always a different name for each record.
The 'Order was assigned to' part is always the same.......
Thanks
I am using SQL Server. It is multiple records with different names in each record.
In your specific case you can use something like:
SELECT SUBSTRING(str, 23) FROM table
However, this is not very scalable, should the format of your strings ever change.
If you are using an Oracle database, you would want to use SUBSTR instead.
Edit:
For databases where the third parameter is not optional, you could use SUBSTRING(str, 23, LEN(str))
Somebody would have to test to see if this is better or worse than subtraction, as in Martin Smith's solution but gives you the same result in the end.
In addition to the SUBSTRING methods, you could also use a REPLACE function. I don't know which would have better performance over millions of rows, although I suspect that it would be the SUBSTRING - especially if you were working with CHAR instead of VARCHAR.
SELECT REPLACE(my_column, 'Order was assigned to ', '')
For SQL Server
WITH testData AS
(
SELECT 'Order was assigned to lastname,firsname' as Col1 UNION ALL
SELECT 'Order was assigned to Bloggs, Jo' as Col1
)
SELECT SUBSTRING(Col1,23,LEN(Col1)-22) AS Name
from testData
Returns
Name
---------------------------------------
lastname,firsname
Bloggs, Jo
on MS SQL Server:
declare #str varchar(100) = 'Order was assigned to lastname,firsname'
declare #strLen1 int = DATALENGTH('Order was assigned to ')
declare #strLen2 int = len(#str)
select #strlen1, #strLen2, substring(#str,#strLen1,#strLen2),
RIGHT(#str, #strlen2-#strlen1)
I would require that a colon or some other delimiter be between the message and the name.
Then you could just search for the index of that character and know that anything after it was the data you need...
Example with format changing over time:
CREATE TABLE #Temp (OrderInfo NVARCHAR(MAX))
INSERT INTO #Temp VALUES ('Order was assigned to :Smith,Mary')
INSERT INTO #Temp VALUES ('Order was assigned to :Holmes,Larry')
INSERT INTO #Temp VALUES ('New Format over time :LootAt,Me')
SELECT SUBSTRING(OrderInfo, CHARINDEX(':',OrderInfo)+1, LEN(OrderInfo))
FROM #Temp
DROP TABLE #Temp
I am in a situation where I am given a comma-separated VarChar as input to a stored procedure. I want to do something like this:
SELECT * FROM tblMyTable
INNER JOIN /*Bunch of inner joins here*/
WHERE ItemID IN ($MyList);
However, you can't use a VarChar with the IN statement. There are two ways to get around this problem:
(The Wrong Way) Create the SQL query in a String, like so:
SET $SQL = '
SELECT * FROM tblMyTable
INNER JOIN /*Bunch of inner joins here*/
WHERE ItemID IN (' + $MyList + ');
EXEC($SQL);
(The Right Way) Create a temporary table that contains the values of $MyList, then join that table in the initial query.
My question is:
Option 2 has a relatively large performance hit with creating a temporary table, which is less than ideal.
While Option 1 is open to an SQL injection attack, since my SPROC is being called from an authenticated source, does it really matter? Only trusted sources will execute this SPROC, so if they choose to bugger up the database, that is their prerogative.
So, how far would you go to make your code secure?
What database are you using? in SQL Server you can create a split function that can split a long string and return a table sub-second. you use the table function call like a regular table in a query (no temp table necessary)
You need to create a split function, or if you have one just use it. This is how a split function can be used:
SELECT
*
FROM YourTable y
INNER JOIN dbo.yourSplitFunction(#Parameter) s ON y.ID=s.Value
I prefer the number table approach to split a string in TSQL but there are numerous ways to split strings in SQL Server, see the previous link, which explains the PROs and CONs of each.
For the Numbers Table method to work, you need to do this one time table setup, which will create a table Numbers that contains rows from 1 to 10,000:
SELECT TOP 10000 IDENTITY(int,1,1) AS Number
INTO Numbers
FROM sys.objects s1
CROSS JOIN sys.objects s2
ALTER TABLE Numbers ADD CONSTRAINT PK_Numbers PRIMARY KEY CLUSTERED (Number)
Once the Numbers table is set up, create this split function:
CREATE FUNCTION [dbo].[FN_ListToTable]
(
#SplitOn char(1) --REQUIRED, the character to split the #List string on
,#List varchar(8000)--REQUIRED, the list to split apart
)
RETURNS TABLE
AS
RETURN
(
----------------
--SINGLE QUERY-- --this will not return empty rows
----------------
SELECT
ListValue
FROM (SELECT
LTRIM(RTRIM(SUBSTRING(List2, number+1, CHARINDEX(#SplitOn, List2, number+1)-number - 1))) AS ListValue
FROM (
SELECT #SplitOn + #List + #SplitOn AS List2
) AS dt
INNER JOIN Numbers n ON n.Number < LEN(dt.List2)
WHERE SUBSTRING(List2, number, 1) = #SplitOn
) dt2
WHERE ListValue IS NOT NULL AND ListValue!=''
);
GO
You can now easily split a CSV string into a table and join on it:
select * from dbo.FN_ListToTable(',','1,2,3,,,4,5,6777,,,')
OUTPUT:
ListValue
-----------------------
1
2
3
4
5
6777
(6 row(s) affected)
Your can use the CSV string like this, not temp table necessary:
SELECT * FROM tblMyTable
INNER JOIN /*Bunch of inner joins here*/
WHERE ItemID IN (select ListValue from dbo.FN_ListToTable(',',$MyList));
I would personally prefer option 2 in that just because a source is authenticated, does not mean you should be letting your guard down. You would leave yourself open to potential rights escalations where an authenticated low lvl user, is able to still execute commands against the database you had not intended.
The phrase you use of 'trusted sources' - it might be better if you assume an X-Files aproach and to trust no-one.
If someone buggers up the database you might still be getting a call.
A good option that is similar to option two is to use a function to create a table in memory from the CSV list. It is reasonably fast and offers the protections of option two. Then that table can be joined to the Inner Join, e.g.
CREATE FUNCTION [dbo].[simple_strlist_to_tbl] (#list nvarchar(MAX))
RETURNS #tbl TABLE (str varchar(4000) NOT NULL) AS
BEGIN
DECLARE #pos int,
#nextpos int,
#valuelen int
SELECT #pos = 0, #nextpos = 1
WHILE #nextpos > 0
BEGIN
SELECT #nextpos = charindex(',', #list, #pos + 1)
SELECT #valuelen = CASE WHEN #nextpos > 0
THEN #nextpos
ELSE len(#list) + 1
END - #pos - 1
INSERT #tbl (str)
VALUES (substring(#list, #pos + 1, #valuelen))
SELECT #pos = #nextpos
END
RETURN
END
Then in the join:
tblMyTable INNER JOIN
simple_strlist_to_tbl(#MyList) list ON tblMyTable.itemId = list.str
Option 3 is to confirm each item in the list is in fact an integer before concatenating the string to your SQL statement.
Do this by parsing the input string (e.g., split into an array), loop through and convert each value to an int, and then recreate the list yourself before concatenating back to the SQL statement. This will give you reasonable assurance that SQL injection cannot occur.
It is safer to concatenate strings that have been created by your application, because you can do things like check for int, but it also means your code is written in a way that a subsequent developer may modify slightly, thus opening back up the risk of SQL injection, because they do not realize that is what your code is protecting against. Make sure you comment well what you are doing if you go this route.
A third option: pass the values to the stored procedure in an array. Then you can either assemble the comma-separated string in your code and use the dynamic SQL option, or (if your flavour of RDBMS permits it) use the array directly in the SELECT statement.
Why don't You write an CLR split function, that will do all the job nice and easy? You can write user Defined table functions which will return a table doing string splitting with .Net infructure. Hell in SQL 2008 you can even give them hints if they return the strings sorted in any way... like ascending or something which can help the optimizer?
Or maybe You can't do CLR integration then You have to stick to the tsql but I personally would go for the CLR soluton