I have data stored in a column in my SQL Server table like the following:
0x003600380032003200330031002D003400
I know the output for this should be:
682231-4
Can someone help me with the SQL code for the conversion?
Something like:
select converttochar (0x003600380032003200330031002D003400)
Returns: 682231-4
I'm not sure if I am looking at hex or binary conversion or what....
It is difficult to see what the exact encoding issue is based on the sample data.
I imagine Larnu is correct and this is just UCS-2 encoded data that for some reason has had a leading null byte added.
Another interpretation is that this is UCS-2 encoded data encoded with a different endianess - so each double byte needs to be reversed (though then there is still a surplus null byte at the end).
As long as each alternate byte is 0x00 it doesn't much matter which case is correct. To strip out the null bytes and cast to varchar you can also use
SELECT REPLACE(0x003600380032003200330031002D003400, 0x00, '' COLLATE Latin1_General_Bin)
For the limited character set shown in the question these will all have the same effect. If the raw data can have non ASCII characters this approach will likely fall down.
This actually looks like you have an nvarchar with a leading 00 at the start. This is messy, but would work:
SELECT CONVERT(nvarchar(30),CONVERT(varbinary(60),STUFF(CONVERT(VARCHAR(60), 0x003600380032003200330031002D003400,1),3,2,''),1));
This strips out the leading 00 characters, making the value 0x3600380032003200330031002D003400, which is the nvarchar value for N'682231-4'.
It strips the value out by converting the varbinary to a varchar but using the style code 1, this means you have a varchar with the value '0x3600380032003200330031002D003400'; allowing you to perform string manipulations. Then I use STUFF to remove the character and CONVERT with style code 1 to make it a varbinary again.
DB<>Fiddle
Alternatively, you could remove all the null character tuples and convert. This looks way messier as you can't just blanket replace '00' (or rather I don't feel it's "safe" to), so I put the characters into their tuples, and rebuild:
DECLARE #YourBinary varbinary(60) = 0x003600380032003200330031002D003400;
WITH N AS(
SELECT N
FROM (VALUES(NULL),(NULL),(NULL),(NULL),(NULL),(NULL))N(N)),
Tally AS(
SELECT TOP(60) ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) -1 AS I
FROM N N1, N N2, N N3)
SELECT CONVERT(varchar(60),CONVERT(varbinary(60),(SELECT V.SS + ''
FROM Tally T
CROSS APPLY(VALUES(SUBSTRING(CONVERT(varchar(60),#YourBinary,1),(T.I*2)+1,2)))V(SS)
WHERE V.SS NOT IN ('00','')
FOR XML PATH(''),TYPE).value('.','varchar(60)'),1));
This idea might be better though, if some of your values have a leading 00 and others don't. Of course, this solution also assumes the end value is a varchar, not an nvarchar.
SELECT CONVERT(VARCHAR(60), 0x003600380032003200330031002D003400);
Demo
I had also same issue some ago and the used the below statement for convert and my problem resolved that time ..
SELECT CONVERT(VARCHAR(60), 0x003600380032003200330031002D003400);
Please try to run this and see whether your problem is resolved or not?
Related
How would you decode a hex string to get the value in text format by using a select statement?
For example my data in hex is:
4f004e004c005900200046004f00520020004200410043004b002d005500500020004f004e0020004c004500560045004c0020004f004e004500200046004f00520020004300520041004e004500530020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020000000
I want to decode it to get the string value using a select statement.
The value of the above is "ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES"
what I have tried is :
SELECT CAST('4f004e004c005900200046004f00520020004200410043004b002d005500500020004f004e0020004c004500560045004c0020004f004e004500200046004f00520020004300520041004e004500530020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020000000'
AS VARCHAR(30000) CCSID 37) from myschema.atable
The above sql returns the exact same hex string and not the decoded text string of "ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES" what I expected.
Is it possible to do this with a cast? If it is what will the syntax be?
My problem that I have is a system stores text data in a blob field and I want to use a select statement to see what the text data is in the blob field.
Db : Db2 on Ibm
Edit:
I have managed to covert the string to the hex value by using :
select hex(cast('ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES' as varchar(100) ccsid 1208))
FROM myschema.atable
This gives me the string in hex :
4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553
Now somehow I need to do the inverse and get the value.
Thanks.
Edit
Using the answer from Daniel Lema, I tried using the unhex function but my result that I got was :
|+<ßã|êâ ä.í&|+<áîá<|+áã|êäê +áë
Is this something to do with a CSSID? Or how should I convet the above to a readable string?
This is the table field definition if it will help the field with my data in is GDTXFT a BLOB :
I was able to take your shortened hex string and convert is to a valid EBCDIC string.
The problem I ran into is that the original hex code you receive comes in UTF-16LE (Thanks Tom Blodget). IBM's CCSID system does not have a distinction between UTF-16BE and UTF-16LE so I am at a loss there on how to convert it properly.
If it is in UTF-8 as you generated later, the following would work for you. It's not the prettiest but throw it in a couple functions and it will work.
Create or replace function unpivothex (in_ varchar(30000))
returns table (Hex_ char(2), Position_ int)
return
with returnstring (ST , POS )
as
(Select substring(STR,1,2), 1
from table(values in_) as A(STR)
union all
Select nullif(substring(STR,POS+2,2),'00'), POS+2
from returnstring, table(values in_) as A(STR)
where POS+2 <= length(in_)
)
Select ST, POS
from returnstring
;
Create or replace function converthextostring
(in_string char(30000))
returns varchar(30000)
return
(select listagg(char(varbinary_format(B.Hex_),1)) within group(order by In_table.Position_)
from table(unpivothex(upper(in_string))) in_table
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 1208)))) A on In_table.Hex_ = A.Hex_
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 37)))) B on A.Position_ = B.Position_
);
Here is a version if you're not on at least V7R2 TR6 or V7R3 TR2.
Create or replace function converthextostring
(in_string char(30000))
returns varchar(30000)
return
(select xmlserialize(
xmlagg(
xmltext(cast(char(varbinary_format(B.Hex_),1) as char(1) CCSID 37))
order by In_table.Position_)
as varchar(30000))
from table(unpivothex(upper(in_string))) in_table
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 1208)))) A on In_table.Hex_ = A.Hex_
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 37)))) B on A.Position_ = B.Position_
);
I tried the following solution I found published by Marcin Rudzki at Convert HEX value to CHAR on DB2, tested in my own Db2 for LUW v11 with a small modification.
the solution consists on creating a function just as Marcin suggested:
CREATE FUNCTION unhex(in VARCHAR(32000) FOR BIT DATA)
RETURNS VARCHAR(32000)
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC NO EXTERNAL ACTION
BEGIN ATOMIC
RETURN in;
END
To test the solution, lets create an HEXSAMPLE table with a HEXSTRING column loaded with the string representation of a HEX sequence:
INSERT INTO HEXSAMPLE (HEXSTRING) VALUES ('4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553')
Then exec the following query (and here it is different from the original proposal):
SELECT UNHEX(CAST(HEXTORAW(HEXSTRING) AS VARCHAR(2000) FOR BIT DATA)) as TEXT, HEXSTRING FROM HEXSAMPLE
With result:
TEXT HEXSTRING
---------------------------------------- --------------------------------------------------------------------------------
ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES 4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553
I hope someone else can find a more direct solution. Also, if someone can explain why it works, it will be very interesting.
I question why you need to do this...
There's valid reasons to convert a hex string back to it's character equivalent...for instance somebody sends you a 32 byte string UUID and you want it back it it's 16 byte binary form.
But there's no reason ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES should have been transformed to hex.
I suspect you need to post a new question asking why you're not getting readable strings in the first place.
However, in answer to this question... IBM i has an MI function Convert Character to Hex (CVTCH) that is easily called from any ILE langage. You could wrap that function call up into a user defined function in order to use it from SQL.
Note that you'll need to know what the hex string represents, EBCDIC, ASCII or Unicode, because you'll need to be able to tell the system what you've started with. From there there are ways to convert between encoding.
Here's an article that shows how to call the MI function from RPG.
Utilizing MI Functions in RPG Programs
A more modern free form version of the prototype that takes advantage of enhancements to the CCSID keyword might look like
dcl-pr FromHex extproc('cvtch');
charString char(32767) ccsid(*UTF8) options(*varsize);
hexString char(65534) ccsid(*HEX) const options(*varsize);
hexStringLen int(10) value;
end-pr;
With the above prototype, the system will treat the character string that comes back as UTF8 (ccsid 1208). But all I'm doing is telling the system how to interpret the bytes that come back. If the string was actually EBCDIC, I'm going to get garbage.
I think you could even defined the cvtch function directly as an external UDF without needing an ILE wrapper. I'd have to play around with that...
Disregard that idea...cvtch only has parameters, not a return value. Using an ILE wrapper is the best way to move the output parameter to a return value for use as a UDF.
The problem is that your original string is in ASCII format (actually with x'00' byte after each letter), and you have to convert it to EBCDIC.
Below is the solution for latin capital letters only:
select cast(translate(replace(mycol, x'00', x'')
, x'C1C2C3C4C5C6C7C8C9D1D2D3D4D5D6D7D8D9E2E3E4E5E6E7E8E940'
, x'4142434445464748494A4B4C4D4E4F505152535455565758595A20'
) as varchar(500) ccsid 37)
from mytab;
Every ASCII character is translated to the corresponding EBCDIC one.
x'00' symbols are removed.
cast (col_name as varchar(2000) ccsid ascii for sbcs data)
I wanted to double check my logic for a query in SQL Server.
The idea is that I am able to feed the following values and it will make sure the result is a decimal with four trailing digits.
Possible values for #LABORQTY:
1,200
1,200.42
1200 (Integer)
1200.42
1200 (As a String)
1200.42 (As a String)
When the value is a string, it will give the error:
Error converting data type nvarchar to numeric.
Here is my code:
CONVERT(DECIMAL(12, 4), REPLACE(#LABORQTY, ',', ''))
The output each time though should be decimal:
1200.4200
Your question is really confused, but I'll answer according to the following parameters:
#laborqty is a VARCHAR
#laborqty may somehow come to contain any of the following values:
'1200'
'1200.42'
'1,200'
'1,200.42'
In which case CONVERT(DECIMAL(12, 4), REPLACE(#LABORQTY, ',', '')) will indeed produce a decimal with up to 4 digits of fractional precision. Whether your query tool/programming language will output it as 1200.4200 or not is another matter entirely; it might well just output 1200.42 and drop the trailing zeroes
If you're getting Error converting data type varchar to numeric. still, there is some other character data (not comma) in your numeric string
If you definitely want the trailing zeroes, format it into a string before you output
FORMAT(CONVERT(decimal(12,4), '1200.42'), '0.0000')
This will generate a string with 4 trailing zeroes
you can use :
select CAST ( REPLACE( '1,200.4' , ',','') AS decimal(17,4))
SQL Server Table has a column with varchar(250) with data like:
(933.13755119234614, -725.869154930114746)(334.30714385628804, 103.50584506988525) and so on.
I want to compare the first two numbers in the first pair with some external values.
I use the following query which works fine sometimes but sometimes results in error converting varchar to numeric.Where am I going wrong??
select *
from testentry2
where (convert(decimal(10,6),substring(data,2,10)) between 25 and 635 )
AND (convert(decimal(10,6),substring(data,CHARINDEX(',',data)+1,10)) between -99 and -9 ) ;
If you really want the entire numbers, parsing in SQL is pretty unpleasant:
SELECT CONVERT(decimal(10,6),SUBSTRING(data, 2, CHARINDEX(',', data)-2)) AS FirstNumber,
CONVERT(decimal(10,6),SUBSTRING(data, CHARINDEX(',', data)+2, CHARINDEX(')(', data)-CHARINDEX(',', data)-2)) AS SecondNumber,
CONVERT(decimal(10,6),SUBSTRING(data, CHARINDEX(')(', data)+2, CHARINDEX(',',data, CHARINDEX(')(', data))-CHARINDEX(')(', data)-2)) AS ThirdNumber,
CONVERT(decimal(10,6),SUBSTRING(data, CHARINDEX(',', data, CHARINDEX(')(', data))+2, LEN(data)-CHARINDEX(',', data, CHARINDEX(')(', data))-2)) AS FourthNumber
FROM testentry2
You do not need the full numbers for the comparison you want. You can try this:
where (convert(decimal(10,6),substring(data, 2, 5)) between 25 and 635 ) AND
(convert(decimal(10,6),substring(data,CHARINDEX(',', data)+1, 4)) between -99 and -9
The numbers "5" and "4" may be overkill. But the idea is simple. If the first number were 12345678.9, then then first five digits suffice, because 12345 is outside your range.
Perhaps this will help with the formatting issue. Otherwise, you will need to parse the strings, which is not pleasant in SQL Server (although you can find various implementations of split() on the web). As a general rule, you don't want to be storing delimited lists in strings. This data should really be separate columns and rows in another table.
I want to compare varbinary type with byte array. I have tried so far:
DECLARE #data AS NVARCHAR(MAX)='4283'
Select * from table1 Where bindayData=CAST(#data AS VARBINARY)
But this does not work.
I note one strange behaviour of this: when I statically use it like
Select * from table1 Where bindayData=CAST('4283' AS VARBINARY)
then it works fine. But when I declare a variable, it doesn't work.
Please share your ideas.
Thanks,
Naresh Goradara
Try
DECLARE #data AS NVARCHAR(MAX)='4283'
The string constant '4283' is non-unicode in the CAST, one byte per character.
This gives 4 bytes varbinary 0x34323833
When you use NVARCHAR(MAX), then it changed to unicode N'4283'string with 2 bytes per character.
This gives 8 bytes varbinary, something like 0x0034003200380033
Using a style of 2 in the convert function does not truncate the final result. It leaves off the "0x" prefex in essence converting the result to a non-binary result.
Please refer to Microsoft's documentation. There is an example at the bottom:
https://learn.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-ver15
Select *
from table1
Where convert(varchar(max),bindayData,2) like '%4283%'
Just to clarify, by specifying something like VARCHAR(45) means it can take up to max 45 characters? I remember I heard from someone a few years ago that the number in the parenthesis doesn't refer to the number of characters, then the person tried to explain to me something quite complicated which I don't understand and forgot already.
And what is the difference between CHAR and VARCHAR? I did search around a bit and see that CHAR gives you the max of the size of the column and it is better to use it if your data has a fixed size and use VARCHAR if your data size varies.
But if it gives you the max of the size of the column of all the data of this column, isn't it better to use it when your data size varies? Especially if you don't know how big your data size is going to be. VARCHAR needs to specify the size (CHAR don't really need right?), isn't it more troublesome?
You also have to specify the size with CHAR. With CHAR, column values are padded with spaces to fill the size you specified, whereas with VARCHAR, only the actual value you specified is stored.
For example:
CREATE TABLE test (
char_value CHAR(10),
varchar_value VARCHAR(10)
);
INSERT INTO test VALUES ('a', 'b');
SELECT * FROM test;
The above will select "a " for char_value and "b" for varchar_value
If all your values are about the same size, the CHAR is possibly a better choice because it will often require less storage space than VARCHAR. This is because VARCHAR stores both the length of the value and the value itself, whereas CHAR can just store the (fixed-size) value.
The MySQL documentation gives a good explanation of the storage requirements of the various data types.
In particular, for a string of length L, a CHAR(M) datatype will take up (M x c) bytes (where c is the number of bytes required to store a character... this depends on the character set in use).
A VARCHAR(M) will take up (L + 1) or (L + 2) depending on whether M is <=255 or >255.
So, it really depends on how long you expect your strings to be, what the variation in length will be.
NB: The documetation doesn't discuss the impact of character sets on the storage requirements of a VARCHAR type. I've tried to quote it accurately, but my guess is that you would need to multiply the string length by the character byte-width as well to get the storage requirement.
The complicated stuff you don't remember is that the 45 refer to bytes, not chars. It's not the same if you are using a multibyte character encoding. In Oracle you can specify bytes or chars explicitly.
varchar2(45 BYTE)
or
varchar2(45 CHAR)
See Difference between BYTE and CHAR in column datatypes
char and varchar actually becomes irrelevant if you have just 1 variable length field in your table, like a varchar or text. Mysql will automatically change all char to varchar.
The fixed length/size record can give you extra performance, but you can't use any variable length field types. The reason is that it will be quicker and easier for mysql to find the next record.
For example, if you do a SELECT * FROM table LIMIT 10, mysql has to scan the table file for the tenth record. This means finding the end of each record until you find the end of the 10th record. But if your table has fixed length/size records, mysql just needs to know the record size and then skip 10 x #bytes.
If you know a column will contain a small, fixed number of chars use a CHAR, otherwise use a varchar. A CHAR column is padded to the max length.
VARCHAR has a small overhead (4-8 bytes depending on RDBMS), but only uses the overhead + the actual number of chars stored.
For the values you know they are going to be constant, for example for Phone Numbers, Zip Codes etc., It is optimal to use "char" for sure.