I am self taught in T-SQL, so I am sure that I can gain efficiency in my code writing, so any pointers are welcomed, even if unrelated to this specific problem.
I am having a problem during a nightly routine I wrote. The database program that is creating the initial data is out of my control and is loosely written, so I have bad data that can blow up my script from time to time. I am looking for assistance in adding error checking into my script so I lose one record instead of the whole thing blowing up.
The code looks like this:
SELECT convert(bigint,(SUBSTRING(pin, 1, 2)+ SUBSTRING(pin, 3, 4)+ SUBSTRING(pin, 7, 5) + SUBSTRING(pin, 13, 3))) AS PARCEL, taxyear, subdivisn, township, propclass, paddress1, paddress2, pcity
INTO [ASSESS].[dbo].[vpams_temp]
FROM [ASSESS].[dbo].[Property]
WHERE parcelstat='F'
GO
The problem is in the first part of this where the concatenation occurs. I am attempting to convert this string (11-1111-11111.000) into this number (11111111111000). If they put their data in correctly, there is punctuation in exactly the correct spots and numbers in the right spots. If they make a mistake, then I end up with punctuation in the wrong spots and it creates a string that cannot be converted into a number.
How about simply replacing "-" and "." with "" before CONVERT to BIGINT?
To do that you would simply replace part of your code with
SELECT CONVERT(BIGINT,REPLACE(REPLACE(pin,"-",""), ".","")) AS PARCEL, ...
Hope it helps.
First, I would use replace() (twice). Second, I would use try_convert():
SELECT try_convert(bigint,
replace(replace(pin, '-', ''), '.', '')
) as PARCEL,
taxyear, subdivisn, township, propclass, paddress1, paddress2, pcity
INTO [ASSESS].[dbo].[vpams_temp]
FROM [ASSESS].[dbo].[Property]
WHERE parcelstat = 'F' ;
You might want to check if there are other characters in the value:
select pin
from [ASSESS].[dbo].[Property]
where pin like '%[^-0-9.]%';
Why not just:
select cast(replace(replace('11-1111-11111.000','-',''),'.','') as bigint)
simply, use the next code:-
declare #var varchar(100)
set #var = '11-1111-11111.000'
select convert(bigint, replace(replace(#var,'-',''),'.',''))
Result:-
11111111111000
Related
I am using SSIS to transform a raw data row into a transaction. Everything was going well until I added logic for a new field called "SplitPercentage" to the SQL command. The new field simply converts the value to a decimal, for example 02887 would transform into 0.2887.
The new logic works as intended, but now it takes 8 hours to run instead of 5 minutes.
Please see entire original code vs new code here:
Greatly appreciate any help!
New logic resulting in poor performance:
IF TRIM(SUBSTRING(#line, 293, 1)) = 1
BEGIN
SET #SplitPercentage = 1
END
ELSE
BEGIN
SET #SplitPercentage = CAST(''.'' + TRIM(SUBSTRING(#line, 294, 4)) AS decimal(7, 4))
END
While your current code is not ideal, I don't see anything in your new expression (SUBSTRING(), TRIM(), concatenation, CAST) that would account for such a drastic performance hit. I suspect the cause lies elsewhere.
However, I believe your expression can be simplified to eliminate the IF. Given a 5-character field "nnnnn" that you wish to treat as a decimal n.nnnn, you should be able to do this in a single statement using STUFF() to inject the decimal point:
#SplitPercentage = CAST(STUFF(SUBSTRING(#line, 293, 5), 2, 0, '.') AS decimal(7, 4))
The STUFF() injects the decimal point at position 2 (replacing 0 characters). I see no need for the TRIM().
(You would to double up the quotes for use within your Exec ('...') statement.)
Please try to change IF/ELSE block of code as follows:
SET #SplitPercentage = IIF(TRIM(SUBSTRING(#line, 293, 1)) = ''1''
, 1.0000
, CAST(''.'' + TRIM(SUBSTRING(#line, 294, 4)) AS DECIMAL(7, 4)));
A challenge you've run into is "I have a huge dynamic query process that I cannot debug." When I run into these issues, I try to break the problem down into smaller, solvable, set based options.
Reading that wall of code, my psuedocode would be something like
For all the data in Inbound_Transaction_Source by a given Source value (#SourceName)
Do all this data validation, type correction and cleanup by slicing out the current line into pieces
You can then lose the row-based approach by slicing your data up. I favor using CROSS APPLY at this point in my life but a CTE, Dervied Table, whatever makes sense in your head is valid.
Why I favor this approach though, is you can see what you're building, test it, and then modify it without worrying you're going to upset a house of cards.
-- Column ordinal declaration and definition is offsite
SELECT
*
FROM
[dbo].[Inbound_Transaction_Source] AS ITS
CROSS APPLY
(
SELECT
CurrentAgentNo = SUBSTRING(ITS.line, #CurrentAgentStartColumn, 10)
, CurrentCompMemo = SUBSTRING(ITS.line, #CompMemoStartColumn + #Multiplier, 1)
, CurrentCommAmount = SUBSTRING(ITS.line, #CommAmountStartColumn + #Multiplier, 9)
, CurrentAnnCommAmount = SUBSTRING(ITS.line, #AnnCommAmountStartColumn + #Multiplier, 9)
, CurrentRetainedCommAmount = SUBSTRING(ITS.line, #RetainedCommAmountStartColumn + #Multiplier, 9)
, CurrentRetainedSwitch = SUBSTRING(ITS.line, #RetainedSwitchStartColumn + #Multiplier, 9)
-- etc
-- A sample of your business logic
, TransactionSourceSystemCode = SUBSTRING(ITS.line, 308, 3)
)NamedCols
CROSS APPLY
(
SELECT
-- There's some business rules to be had here for first year processing
-- Something special with position 102
SUBSTRING(ITS.line,102 , 1) AS SeniorityBit
-- If department code? is 0079, we have special rules
, TRIM(SUBSTRING(ITS.line,141, 4)) As DepartmentCode
)BR0
CROSS APPLY
(
SELECT
CASE
WHEN NamedCols.TransactionSourceSystemCode in ('LVV','UIV','LMV') THEN
CASE WHEN BR0.SenorityBit = '0' THEN '1' ELSE '0' END
WHEN NamedCols.TransactionSourceSystemCode in ('CMP','FAL') AND BR0.DepartmentCode ='0079' THEN
CASE WHEN BR0.SenorityBit = '1' THEN '0' ELSE '1' END
WHEN NamedCols.TransactionSourceSystemCode in ('UIA','LMA','RIA') AND BR0.SenorityBit > '1' THEN
'1'
WHEN NamedCols.TransactionSourceSystemCode in ('FAL') THEN
'1'
ELSE '0'
END
)FY(IsFirstYear)
WHERE Source = #SourceName
ORDER BY Id;
Why did processing take increase from 5 minutes to 8 hours?
It likely had nothing to do with the change to the dynamic SQL. When an SSIS package run is "taking forever" relative to normal, then preferably while it's still running, look at your sources and destinations and make note of what it happening as it's likely one of the two.
A cursor complicates your life and is not needed once you start thinking in sets but it's unlikely to be the source of the performance problems given than you have a solid baseline of what normal is. Plus, this query is a single table query with a single filter.
Your SSIS package's data flow is probably chip shot Source to Destination Extract and Load or Slurp and Burp with no intervening transformation (as the logic is all in the stored procedure). If that's the case, then the only two possible performance points of contention are the source and destination. Since the source appears trivial, then it's likely that some other process had the destination tied up for those 8 hours. Had you run something like sp_whoisactive on the source and destination, you can identify the process that is blocking your run.
I am converting an access query that updates fields from ALL CAPS to Basically normal First Letter Capital, rest lower case. In access ive used strconv
Ive havent found a similar function in advantage sql db. Ive found upper and lcase, but those dont seem to work for me.
StrConv([City],3)
An import process brings in all caps to a field called City. So City Comes in as CHICAGO, my end result would be Chicago, and this works in access using strconv,3
This statement should work: SELECT Upper( Left( [field], 1 ) ) + Lower( SubString( [field], 2, Length( [field] )-1 ) ) FROM CustomersTbl
Never used advantage but a quick Google suggests something like the following should work.
CONCAT(SUBSTRING(field, 1, 1), LOWER(SUBSTRING(field, 2, LENGTH(field) - 1)))
Advantage doesn't have a StrConv function, but there are a few ways to do the same thing.
I've used a scripting variable in this code segment to emulate the field. You can replace all instances of cityname with the name of your database column (field). Pick the one you like. system.iota is a one-row system database that can be used in Advantage when you don't want to use a real table for testing functions. Of course, you'd replace it with your actual table name in your own code. You can run each of the sample select statements (individually) in the Advantage Database Utility to test them; pick the one that seems the cleanest to you, as they're all the same as far as performance goes. (NOTE: UCase() and UpperCase() are the exact same functions internally; they both exist simply because some users expect to find UCase() and others UpperCase() depending on the programming language they're using. The same is true of LCase() and LowerCase().)
I've intentionally used a mixed bag of UPPER/lower case letters in the variable cityname below to demonstrate that the actual case doesn't matter; the statements provided will produce a proper-cased (first letter upper, rest lower) result.
declare cityname string;
set cityname = 'cHIcAGo';
-- Method 1
select
UCase(Substring(cityname, 1, 1)) + LCASE(SubString(cityname, 2, Length(cityname))) as city
from system.iota
-- Method 2
select
UpperCase(Left(cityname, 1)) + LowerCase(SubString(cityname, 2, Length(cityname)) as city
from system.iota
-- Method 3
select
UCase(Left(cityname, 1)) + LCase(Right(cityname, Length(city) - 1)) as city
from system.iota
I started with this but is it the best way to perform the task?
select
reverse(
substring(reverse(some_field),
charindex('-', reverse(some_field)) + 1,
len(some_field) - charindex('-', reverse(some_field))))
from SomeTable
How does SQL Server treat the
multiple calls to
reverse(some_field)?
Besides a UDF and iterating through
the string looking for charindex
of the '-' and storing the last
index of it, is there a more
efficient way to perform this task in T-SQL?
Note that what I have works, I just am really wondering if it is the best way about it.
Below are some sample values for some_field.
s2-st, s1-st, s3-st, s3-sss-zzz, s4-sss-zzzz
EDIT:
Sample output for this would be...
s1, s2, s3-sss, s3, s4-sss
The solution ErikE wrote is actually getting the end of the string so everything after the last hyphen. I just modified his version to get everything before it instead using a similar method with the left function. Thanks for all of your your help.
select left(some_field, abs(charindex('-', reverse(some_field)) - len(some_field)))
from (select 's2-st' as some_field
union select 's1-st'
union select 's3-st'
union select 's3-sss-zzz'
union select 's4-sss-zzzz') as SomeTable
May I suggest this simplification of your expression:
select right(some_field, charindex('-', reverse(some_field)) - 1)
from SomeTable
Also, there's no harm, as far as I know, in specifying 8000 characters in length with the substring function when you want the rest of the string. As long as it's not varchar(max), it works just fine.
If this is something you have to do all the time, over and over, how about #1 splitting out the data into separate columns and storing it that way, or #2 adding a calculated column with an index on it, which will perform the calculation once at update/insert time and not again later.
Last, I don't know if SQL Server is smart enough to reverse(some_field) only once and inject it into the other instance. When I get some time I'll try to figure it out.
Update
Oops, somehow I got backwards what you wanted. Sorry about that. The new expression you showed can still be simplified a little:
select left(some_field, len(some_field) - charindex('-', reverse(some_field)))
from (
select 's2-st'
union all select 's1-st'
union all select 's3-st'
union all select 's3-sss-zzz'
union all select 's4-sss-zzzz'
union all select 's5'
) X (some_field)
The abs() in your expression was just reversing the sign. So I put + len - charindex instead of + charindex - len and all is well now. It even works for strings without dashes.
One more thing to mention: your UNION SELECTs should be UNION ALL SELECT because without the ALL, the engine has to remove duplicates just as if you'd indicated SELECT DISTINCT. Simply get in the habit of using ALL and you'll be much better off. :)
Not sure about #1, but I would say that you might be better off doing this in code. Is there a reason you have to do it in the database?
Are you experiencing performance problems because of some similar code or is this purely hypothetical.
I am also not sure how SQL Server handles the multiple calls to REVERSE and CHARINDEX.
You can eliminate the last call to CHARINDEX since you want to take everything to the end of the string:
select
reverse(
substring(reverse(some_field),
charindex('-', reverse(some_field)) + 1,
len(some_field)))
from SomeTable
Although I would recommend against it, you could also replace the LEN function call with the size of the column:
select
reverse(
substring(reverse(some_field),
charindex('-', reverse(some_field)) + 1,
1024))
from SomeTable
I am curious how much of a difference either of these changes would make.
The 3 inner reverses are discrete from each other. The outer reverse will reverse anything that is already reversed by the inner ones.
ErikE's approach is best as a pure TSQL solution. You don't need LEN
I'm looking for a way to transform a genuine string into it's hexadecimal value in SQL. I'm looking something that is Informix-friendly but I would obviously prefer something database-neutral
Here is the select I am using now:
SELECT SomeStringColumn from SomeTable
Here is the select I would like to use:
SELECT hex( SomeStringColumn ) from SomeTable
Unfortunately nothing is that simple... Informix gives me that message:
Character to numeric conversion error
Any idea?
Can you use Cast and the fn_varbintohexstr?
SELECT master.dbo.fn_varbintohexstr(CAST(SomeStringColumn AS varbinary))
FROM SomeTable
I'm not sure if you have that function in your database system, it is in MS-SQL.
I just tried it in my SQL server MMC on one of my tables:
SELECT master.dbo.fn_varbintohexstr(CAST(Addr1 AS VARBINARY)) AS Expr1
FROM Customer
This worked as expected. possibly what I know as master.dbo.fn_varbintohexstr on MS-SQL, might be similar to informix hex() function, so possibly try:
SELECT hex(CAST(Addr1 AS VARBINARY)) AS Expr1
FROM Customer
The following works in Sql 2005.
select convert(varbinary, SomeStringColumn) from SomeTable
Try this:
select convert(varbinary, '0xa3c0', 1)
The hex number needs to have an even number of digits. To get around that, try:
select convert(varbinary, '0x' + RIGHT('00000000' + REPLACE('0xa3c','0x',''), 8), 1)
If it is possible for you to do this in the database client in code it might be easier.
Otherwise the error probably means that the built in hex function can't work with your values as you expect. I would double check the input value is trimmed and in the format first, it might be that simple. Then I would consult the database documentation that describes the hex function and see what its expected input would be and compare that to some of your values and find out what the difference is and how to change your values to match that of the expected input.
A simple google search for "informix hex function" brought up the first result page with the sentence: "Must be a literal integer or some other expression that returns an integer". If your data type is a string, first convert the string to an integer. It looks like at first glance you do something with the cast function (I am not sure about this).
select hex(cast SomeStringColumn as int)) from SomeTable
what about:
declare #hexstring varchar(max);
set #hexstring = 'E0F0C0';
select cast('' as xml).value('xs:hexBinary( substring(sql:variable("#hexstring"), sql:column("t.pos")) )', 'varbinary(max)')
from (select case substring(#hexstring, 1, 2) when '0x' then 3 else 0 end) as t(pos)
I saw this here:
http://blogs.msdn.com/b/sqltips/archive/2008/07/02/converting-from-hex-string-to-varbinary-and-vice-versa.aspx
Sorrry, that work only on >MS SQL 2005
OLD Post but in my case I also had to remove the 0x part of the hex so I used the below code. (I'm using MS SQL)
convert(varchar, convert(Varbinary(MAX), YOURSTRING),2)
SUBSTRING(CONVERT(varbinary,Addr1 ) ,1,1) as Expr1
I want to get the MD5 Hash of a string value in SQL Server 2005. I do this with the following command:
SELECT HashBytes('MD5', 'HelloWorld')
However, this returns a VarBinary instead of a VarChar value. If I attempt to convert 0x68E109F0F40CA72A15E05CC22786F8E6 into a VarChar I get há ðô§*à\Â'†øæ instead of 68E109F0F40CA72A15E05CC22786F8E6.
Is there any SQL-based solution?
Yes
I have found the solution else where:
SELECT SUBSTRING(master.dbo.fn_varbintohexstr(HashBytes('MD5', 'HelloWorld')), 3, 32)
SELECT CONVERT(NVARCHAR(32),HashBytes('MD5', 'Hello World'),2)
Use master.dbo.fn_varbintohexsubstring(0, HashBytes('SHA1', #input), 1, 0) instead of master.dbo.fn_varbintohexstr and then substringing the result.
In fact fn_varbintohexstr calls fn_varbintohexsubstring internally. The first argument of fn_varbintohexsubstring tells it to add 0xF as the prefix or not. fn_varbintohexstr calls fn_varbintohexsubstring with 1 as the first argument internaly.
Because you don't need 0xF, call fn_varbintohexsubstring directly.
Contrary to what David Knight says, these two alternatives return the same response in MS SQL 2008:
SELECT CONVERT(VARCHAR(32),HashBytes('MD5', 'Hello World'),2)
SELECT UPPER(master.dbo.fn_varbintohexsubstring(0, HashBytes('MD5', 'Hello World'), 1, 0))
So it looks like the first one is a better choice, starting from version 2008.
convert(varchar(34), HASHBYTES('MD5','Hello World'),1)
(1 for converting hexadecimal to string)
convert this to lower and remove 0x from the start of the string by substring:
substring(lower(convert(varchar(34), HASHBYTES('MD5','Hello World'),1)),3,32)
exactly the same as what we get in C# after converting bytes to string
With personal experience of using the following code within a Stored Procedure which Hashed a SP Variable I can confirm, although undocumented, this combination works 100% as per my example:
#var=SUBSTRING(master.dbo.fn_varbintohexstr(HashBytes('SHA2_512', #SPvar)), 3, 128)
Changing the datatype to varbinary seems to work the best for me.