I have in my application a user defined function which takes a comma separated list as an argument. It splits the items and plugs them in to a table variable and returns the result.
This function works well, except that when the items in the comma separated list exceed 1000, it ignores the remainder. That is to say, if I plug in 1239, the first 1000 rows will be returned and the remaining 239 are entirely ignored. There are no errors when this occurs.
I can't help but feel that this is due to some sort of limitation that I should know about, but I can't seem to find any information about it. Is it a limitation on the amount of rows that can be stored in a table variable? Or am I missing something in the actual code itself? Can anyone assist? Going squirrely-eyed over here.
ALTER FUNCTION [dbo].[ufnConvertArrayToIntTable] (#IntArray VARCHAR(8000))
RETURNS #retIntTable TABLE
(
ID int
)
AS
BEGIN
DECLARE #Delimiter char(1)
SET #Delimiter = ','
DECLARE #Item varchar(8)
IF CHARINDEX(#Delimiter,#IntArray,0) <> 0
BEGIN
WHILE CHARINDEX(#Delimiter,#IntArray,0) <> 0
BEGIN
SELECT
#Item = RTRIM(LTRIM(SUBSTRING(#IntArray,1,CHARINDEX(#Delimiter,#IntArray,0)-1))),
#IntArray = RTRIM(LTRIM(SUBSTRING(#IntArray,CHARINDEX(#Delimiter,#IntArray,0)+1,LEN(#IntArray))))
IF LEN(#Item) > 0
INSERT INTO #retIntTable SELECT #Item
END
IF LEN(#IntArray) > 0
INSERT INTO #retIntTable SELECT #IntArray
END
ELSE
BEGIN
IF LEN(#IntArray) > 0
INSERT INTO #retIntTable SELECT #IntArray
END
RETURN
END;
You define your input variable as varchar(8000) and your #Item variable is varchar(8). Are your items typically 8 characters each? Is the string you send in w/ over 1000 items more than 8000 characters? Try changing your input to varchar(max) instead.
Are all of your comma seperated values 8 chars long? If so, then the input parameter will only be able to hold 888 (8000 / 9(including the comma) of them..
It's because your input parameter is limited to 8000 characters.
You might try calling the function using substring... Maybe:
WHERE
[myField] IN(Select ID from [dbo].[ufnConvertArrayToIntTable](substring(#inputarray, 1, 4000))
OR
[myField] IN(Select ID from [dbo].[ufnConvertArrayToIntTable](substring(#inputarray, 4001, 8000))
...
Related
I have a table with the following columns:
dbo.SomeInfo
- Id
- Name
- InfoCode
Now I need to update the above table's InfoCode as
Update dbo.SomeInfo
Set InfoCode= REPLACE(Replace(RTRIM(LOWER(Name)),' ','-'),':','')
This replaces all spaces with - & lowercase the name
When I do check the InfoCode, I see there are Names with some special characters like
Cathe Friedrich''s Low Impact
coffeyfit-cardio-box-&-burn
Jillian Michaels: Cardio
Then I am manually writing the update sql against this as
Update dbo.SomeInfo
SET InfoCode= 'cathe-friedrichs-low-impact'
where Name ='Cathe Friedrich''s Low Impact '
Now, this solution is not realistic for me. I checked the following links related to Regex & others around it.
UPDATE and REPLACE part of a string
https://www.codeproject.com/Questions/456246/replace-special-characters-in-sql
But none of them is hitting the requirement.
What I need is if there is any character other [a-z0-9] replace it - & also there should not be continuous -- in InfoCode
The above Update sql has set some values of InfoCode as the-dancer's-workout®----starter-package
Some Names have value as
Sleek Technique™
The Dancer's-workout®
How can I write Update sql that could handle all such special characters?
Using NGrams8K you could split the string into characters and then rather than replacing every non-acceptable character, retain only certain ones:
SELECT (SELECT '' + CASE WHEN N.token COLLATE Latin1_General_BIN LIKE '[A-z0-9]'THEN token ELSE '-' END
FROM dbo.NGrams8k(V.S,1) N
ORDER BY position
FOR XML PATH(''))
FROM (VALUES('Sleek Technique™'),('The Dancer''s-workout®'))V(S);
I use COLLATE here as on my default collation in my instance the '™' is ignored, therefore I use a binary collation. You may want to use COLLATE to switch the string back to its original collation outside of the subquery.
This approach is fully inlinable:
First we need a mock-up table with some test data:
DECLARe #SomeInfo TABLE (Id INT IDENTITY, InfoCode VARCHAR(100));
INSERT INTO #SomeInfo (InfoCode) VALUES
('Cathe Friedrich''s Low Impact')
,('coffeyfit-cardio-box-&-burn')
,('Jillian Michaels: Cardio')
,('Sleek Technique™')
,('The Dancer''s-workout®');
--This is the query
WITH cte AS
(
SELECT 1 AS position
,si.Id
,LOWER(si.InfoCode) AS SourceText
,SUBSTRING(LOWER(si.InfoCode),1,1) AS OneChar
FROM #SomeInfo si
UNION ALL
SELECT cte.position +1
,cte.Id
,cte.SourceText
,SUBSTRING(LOWER(cte.SourceText),cte.position+1,1) AS OneChar
FROM cte
WHERE position < DATALENGTH(SourceText)
)
,Cleaned AS
(
SELECT cte.Id
,(
SELECT CASE WHEN ASCII(cte2.OneChar) BETWEEN 65 AND 90 --A-Z
OR ASCII(cte2.OneChar) BETWEEN 97 AND 122--a-z
OR ASCII(cte2.OneChar) BETWEEN 48 AND 57 --0-9
--You can easily add more ranges
THEN cte2.OneChar ELSE '-'
--You can easily nest another CASE to deal with special characters like the single quote in your examples...
END
FROM cte AS cte2
WHERE cte2.Id=cte.Id
ORDER BY cte2.position
FOR XML PATH('')
) AS normalised
FROM cte
GROUP BY cte.Id
)
,NoDoubleHyphens AS
(
SELECT REPLACE(REPLACE(REPLACE(normalised,'-','<>'),'><',''),'<>','-') AS normalised2
FROM Cleaned
)
SELECT CASE WHEN RIGHT(normalised2,1)='-' THEN SUBSTRING(normalised2,1,LEN(normalised2)-1) ELSE normalised2 END AS FinalResult
FROM NoDoubleHyphens;
The first CTE will recursively (well, rather iteratively) travers down the string, character by character and a return a very slim set with one row per character.
The second CTE will then GROUP the Ids. This allows for a correlated sub-query, where the actual check is performed using ASCII-ranges. FOR XML PATH('') is used to re-concatenate the string. With SQL-Server 2017+ I'd suggest to use STRING_AGG() instead.
The third CTE will use a well known trick to get rid of multiple occurances of a character. Take any two characters which will never occur in your string, I use < and >. A string like a--b---c will come back as a<><>b<><><>c. After replacing >< with nothing we get a<>b<>c. Well, that's it...
The final SELECT will cut away a trailing hyphen. If needed you can add similar logic to get rid of a leading hyphen. With v2017+ There was TRIM('-') to make this easier...
The result
cathe-friedrich-s-low-impact
coffeyfit-cardio-box-burn
jillian-michaels-cardio
sleek-technique
the-dancer-s-workout
You can create a User-Defined-Function for something like that.
Then use the UDF in the update.
CREATE FUNCTION [dbo].LowerDashString (#str varchar(255))
RETURNS varchar(255)
AS
BEGIN
DECLARE #result varchar(255);
DECLARE #chr varchar(1);
DECLARE #pos int;
SET #result = '';
SET #pos = 1;
-- lowercase the input and remove the single-quotes
SET #str = REPLACE(LOWER(#str),'''','');
-- loop through the characters
-- while replacing anything that's not a letter to a dash
WHILE #pos <= LEN(#str)
BEGIN
SET #chr = SUBSTRING(#str, #pos, 1)
IF #chr LIKE '[a-z]' SET #result += #chr;
ELSE SET #result += '-';
SET #pos += 1;
END;
-- SET #result = TRIM('-' FROM #result); -- SqlServer 2017 and beyond
-- multiple dashes to one dash
WHILE #result LIKE '%--%' SET #result = REPLACE(#result,'--','-');
RETURN #result;
END;
GO
Example snippet using the function:
-- using a table variable for demonstration purposes
declare #SomeInfo table (Id int primary key identity(1,1) not null, InfoCode varchar(100) not null);
-- sample data
insert into #SomeInfo (InfoCode) values
('Cathe Friedrich''s Low Impact'),
('coffeyfit-cardio-box-&-burn'),
('Jillian Michaels: Cardio'),
('Sleek Technique™'),
('The Dancer''s-workout®');
update #SomeInfo
set InfoCode = dbo.LowerDashString(InfoCode)
where (InfoCode LIKE '%[^A-Z-]%' OR InfoCode != LOWER(InfoCode));
select *
from #SomeInfo;
Result:
Id InfoCode
-- -----------------------------
1 cathe-friedrichs-low-impact
2 coffeyfit-cardio-box-burn
3 jillian-michaels-cardio
4 sleek-technique-
5 the-dancers-workout-
I'm running an update script to obfuscate data and am occasionally experiencing the arithmetic overflow error message, as in the title. The table being updated has 260k records and yet the update script will need to be run several times to produce the error. Although it's so rare I can't rely on the code until it's fixed as it's a pain to debug.
Looking at other similar questions, this is often resolved by changing the data type e.g from INT to BIGINT either in the table or in a calculation. However, I can't see where this could be required. I've reduced the script to the below as I've managed to pin point it to the update of one column.
A function is being called by the update and I've included this below. I suspect that, due to the randomness of the error, the use of the NEW_ID function could be causing it but I haven't been able to re-create the error when just running this part of the function multiple times. The NEW_ID function can't be used in functions so it's being called from a view, also included below.
Update script:
UPDATE dbo.Addresses
SET HouseNumber = CASE WHEN LEN(HouseNumber) > 0
THEN dbo.fn_GenerateRandomString (LEN(HouseNumber), 1, 1, 1)
ELSE HouseNumber
END
NEW_ID view and random string function
CREATE VIEW dbo.vw_GetNewID
AS
SELECT NEWID() AS New_ID
CREATE FUNCTION dbo.fn_GenerateRandomString (
#stringLength int,
#upperCaseBit bit,
#lowerCaseBit bit,
#numberBit bit
)
RETURNS nvarchar(100)
AS
BEGIN
-- Sanitise string length values.
IF ISNULL(#stringLength, -1) < 0
SET #stringLength = 0
-- Generate a random string from the specified character sets.
DECLARE #string nvarchar(100) = ''
SELECT
#string += c2
FROM
(
SELECT TOP (#stringLength) c2 FROM (
SELECT c1 FROM
(
VALUES ('A'),('B'),('C')
) AS T1(c1)
WHERE #upperCaseBit = 1
UNION ALL
SELECT c1 FROM
(
VALUES ('a'),('b'),('c')
) AS T1(c1)
WHERE #lowerCaseBit = 1
SELECT c1 FROM
(
VALUES ('0'),('1'),('2'),('3'),('4'),('5'),('6'),('7'),('8'),('9')
) AS T1(c1)
WHERE #numberBit = 1
)
AS T2(c2)
ORDER BY (SELECT ABS(CHECKSUM(New_ID)) from vw_GetNewID)
) AS T2
RETURN #string
END
Addresses table (for testing):
CREATE TABLE dbo.Addresses(HouseNumber nchar(32) NULL)
INSERT Addresses(HouseNumber)
VALUES ('DSjkmf jkghjsh35hjk h2jkhj3h jhf'),
('SDjfksj3548 ksjk'),
(NULL),
(''),
('2a'),
('1234567890'),
('An2b')
Note: only 7k of the rows in the addresses table have a value entered i.e. LEN(HouseNumber) > 0.
An arithmetic overflow in what is otherwise string-based code is confounding. But there is one thing that could be causing the arithmetic overflow. That is your ORDER BY clause:
ORDER BY (SELECT ABS(CHECKSUM(New_ID)) from vw_GetNewID)
CHECKSUM() returns an integer, whose range is -2,147,483,648 to 2,147,483,647. Note the absolute value of the smallest number is 2,147,483,648, and that is just outside the range. You can verify that SELECT ABS(CAST('-2147483648' as int)) generates the arithmetic overflow error.
You don't need the checksum(). Alas, you do need the view because this logic is in a function and NEWID() is side-effecting. But, you can use:
ORDER BY (SELECT New_ID from vw_GetNewID)
I suspect that the reason you are seeing this every million or so rows rather than every 4 billion rows or so is because the ORDER BY value is being evaluated multiple times for each row as part of the sorting process. Eventually, it is going to hit the lower limit.
EDIT:
If you care about efficiency, it is probably faster to do this using string operations rather than tables. I might suggest this version of the function:
CREATE VIEW vw_rand AS SELECT rand() as rand;
GO
CREATE FUNCTION dbo.fn_GenerateRandomString (
#stringLength int,
#upperCaseBit bit,
#lowerCaseBit bit,
#numberBit bit
)
RETURNS nvarchar(100)
AS
BEGIN
DECLARE #string NVARCHAR(255) = '';
-- Sanitise string length values.
IF ISNULL(#stringLength, -1) < 0
SET #stringLength = 0;
DECLARE #lets VARCHAR(255) = '';
IF (#upperCaseBit = 1) SET #lets = #lets + 'ABC';
IF (#lowerCaseBit = 1) SET #lets = #lets + 'abc';
IF (#numberBit = 1) SET #lets = #lets + '0123456789';
DECLARE #len int = len(#lets);
WHILE #stringLength > 0 BEGIN
SELECT #string += SUBSTRING(#lets, 1 + CAST(rand * #len as INT), 1)
FROM vw_rand;
SET #stringLength = #stringLength - 1;
END;
RETURN #string
END;
As a note: rand() is documented as being exclusive of the end of its range, so you don't have to worry about it returning exactly 1.
Also, this version is subtly different from your version because it can pull the same letter more than once (and as a consequence can also handle longer strings). I think this is actually a benefit.
A simple table contains the one column with integer vales.The Figure is given below.
I am using COALESCE to construct the 'numbers' by comma.
So, now there is a problem when i check a above constructed value in IF Condition like below. It shows an error for cannot convert the varchar datatype to integer.
Now how to check the constructed values in IF condition without changing a logic? I am new for T-SQL.Thank you
When you concatenate all the numbers you are converting it a string so it no longer acts like an integer. If you want to check for a value in a list, do it directly like. SQL was made to have different values in different rows and work with them that way. Try this out:
DECLARE #failIds INT = 23;
IF #failIds IN (SELECT numbers FROM model)
BEGIN
PRINT 'YES'
END
ELSE
BEGIN
PRINT 'NO'
END
Instead of the comma separated string, you should use a table variable, temp. table or just the original table. For example, something like this:
declare #Ids table (id int)
insert into #Ids
select numbers
from model
where numbers in (23,234)
declare #failIds int = 23
if (exists (select 1 from #Ids where id = #failIds)) begin
print 'Yes'
end else begin
print 'No'
end
But you could of course do this too:
if (exists (select 1 from model where numbers = #failIds)) begin
print 'Yes'
end else begin
print 'No'
end
I am very lazy to think about it, Finally i get answer like give below. Its very simple.
I am trying to use wildcard expression to fetch data related to a sequence of numbers. Can I know how to use a series of numbers inside wildcard expression LIKE [0-10].
here is my query:
select grade from table where grade LIKE [1-12]?
output: is 1 and 2
I referred to t-SQL book and they talk about LIKE N[1-12]. What's the difference between LIKE [1-12] and N[1-12]?
I can use between 1 and 12 to fetch my data. But I am just curious how to use a wildcard for series of numbers with LIKE operator?
In SQL Server, like has three wildcards. Underscore '_' represents any single character. % represents zero or more characters. And square brackets.
The expression between the square brackets represents one single character. So,
x like '[abc]'
matches "a", "b", or "c" -- and nothing else. The following matches any digit:
x like '[0123456789]'
This, however, starts to get cumbersome to type out. So, SQL Server offers the shorthand:
x like '[0-9]'
This just means any character from the range starting with 0 and ending at 9.
You could match any hex character with:
x like '[0-9ABCDEF]'
So, additional characters are allowed in the range.
When you write
x like '[1-12]'
You are saying x like the range of characters from 1 to 1, plus the character 2. This is more easily written as:
x like '[12]'
In any case, you shouldn't store numeric values as strings, and you shouldn't use like on numbers. It is much better to write:
grade between 1 and 12
Or something like that.
But if you already have a column with a sequence of numbers and don't know the size, what I've done was this function:
CREATE FUNCTION Keep_Only_Int (#X VARCHAR(MAX)) RETURNS BIGINT AS BEGIN
IF #X IS NULL RETURN NULL
DECLARE #T AS INT = LEN(#X), #I AS INT = 0, #J AS CHAR(1), #RET AS VARCHAR(50) = ''
WHILE #I < #T BEGIN
SET #I += 1
SET #J = SUBSTRING(#X, #I, 1)
IF ASCII(#J) BETWEEN 48 AND 57 --Numbers, is needed because ¹, ² and ³ are going to return true in the link
SET #RET += #J
END
IF LEN(#RET) > 19 RETURN NULL --Bigger then bigint
RETURN NULLIF(#RET, '')
END
An example of usage:
create table #a (content varchar(100))
insert #a values ('My number is 123, whatever')
insert #a values ('My number is 1234, whatever')
insert #a values ('My number is ¹²³4, whatever') --> Special numbers
insert #a values ('My number is one, whatever') --> No number
insert #a values ('My number is 1234567890123456789, whatever')
insert #a values ('My number is 12345678901234567890, whatever')--> This is too big!
select *
, dbo.Keep_Only_Int(content)
from #a
The function already convert the field to BIGINT, so you can use an between statement
select *
from #a
where dbo.Keep_Only_Int(content) between 1 and 2000
It is not focused on a great performance, if you are using a table too big I'd recomend creating a specific code for that
I need to replace a list of characters in a string with some mapped characters.
I have a table 'dbo.CharacterMappings' with 2 columns: 'CharacterToFilter' and 'ReplacementCharacter'.
Say that there are 3 records in this table:
Filter Replacement
$ s
# a
0 o
How would I replace all of the filter characters in a string based on these mappings?
i.e. 'Hell0 c#t$' needs to become 'Hello cats'.
I cant really think of any way of doing this without resorting to a table variable and then looping through it. I.e. have a table variable with a 'count' column then use a loop to select 1 row at a time based on this column. Then I can use the REPLACE function to update the characters one at a time.
Edit: I should note that I always want to strip out these characters (I don't need to worry about $5 -> s5 for example).
declare #s varchar(50)= 'Hell0 c#t$'
select #s = REPLACE(#s, CharacterToFilter, ReplacementCharacter)
from CharacterMappings
select #s
You could create a function:
CREATE FUNCTION [dbo].[ReplaceAll]
(
#text varchar(8000)
)
RETURNS VARCHAR(8000)
AS
BEGIN
SELECT #text =
REPLACE(#text,cm.Filter, cm.Replacement)
FROM CharacterMappings cm;
RETURN #text
END
Then this
select dbo.[ReplaceAll]('Hell0 c#t$');
returns Hello cats