Encrypting and Decrypting String Using SQL's EncryptByPassPhrase - sql

I have a need to Encrypt text, transmit the encrypted text, and decrypt the text ... using SQL.
I tried EncryptByPassPhrase which returns a VARBINARY. I convert it to VARCHAR (for easier transmitting), and when I try to convert it back to VARBINARY, it fails. From what I can see, the CONVERT from VarBinary to VarChar, truncates the string, appears like the second half of the VarBinary value gets dropped when converting to VarChar. So when converting back from that VarChar value, it is incomplete and fails.
Here is sample code:
DECLARE #binEncryptedText VARBINARY(MAX) = EncryptByPassPhrase('Password', 'My Text')
DECLARE #strEncryptedText VARCHAR(MAX) = CONVERT(VARCHAR, #binEncryptedText, 2)
DECLARE #binDecryptedText VARBINARY(MAX) = CONVERT(VARBINARY, #strEncryptedText, 2)
DECLARE #strDecryptedText VARCHAR(MAX) = CONVERT(VARCHAR, DecryptByPassPhrase('Password', #binDecryptedText))
SELECT 1 As RowNumber,
#binEncryptedText AS EncryptedBinary,
#strEncryptedText AS EncryptedBinaryAsText
UNION
SELECT 2 As RowNumber,
#binDecryptedText AS EncryptedBinary,
#strDecryptedText AS EncryptedBinaryAsText
As you can see in the screenshot of my results above... In Row #1, the Binary that was converted to Text ... is shorter than the original Binary (truncated roughly halfway). So when converting that truncated text back to Binary, it fails to return the complete value ... which then causes the Decrypt to fail.
How can I get the encrypted text's VarBinary converted to "text" and then have the "text" converted back to the correct VarBinary value so it can be decrypted?

MSSQL has a well hidden Base64 encoder and decoder - within the XML subsystem. Convert the encrypted varbinary to Base64 for transmission, it will become an ASCII string, transmit the string, then decode back to a varbinary. The encoder/decoder is an XPath function, so you need an XML context to run it, but you can provide a dummy one. So the code for encoding goes:
cast('' as xml).value('xs:base64Binary(sql:variable("#b"))', 'varchar(max)')
where #b is your varbinary.
To decode:
cast('' as xml).value('xs:base64Binary(sql:variable("#s"))', 'varbinary(max)')
where #s is the Base64 string.
It's kind of funky that the encoder and the decoder is the same function, but I guess it knows what to do by the datatype of the argument. I haven't used this technique extensively myself, but I've tried it and it worked as expected. MSSQL 2019.
You can also use xs:hexBinary to encode the binary data as hex. Base64 representation is smaller, though; 4 characters for 3 bytes vs. 2 for 1.

So I actually ended up figuring out what my problem was. In SQL, some of my commands were "incomplete" ... in a manner of speaking.
DECLARE #binEncryptedText VARBINARY(MAX) = EncryptByPassPhrase('Password', 'My Text')
DECLARE #strEncryptedText VARCHAR(MAX) = CONVERT(VARCHAR(MAX), #binEncryptedText, 2)
DECLARE #binDecryptedText VARBINARY(MAX) = CONVERT(VARBINARY(MAX), #strEncryptedText, 2)
DECLARE #strDecryptedText VARCHAR(MAX) = CONVERT(VARCHAR(MAX), DecryptByPassPhrase('Password', #binDecryptedText))
SELECT 1 As RowNumber,
#binEncryptedText AS EncryptedBinary,
#strEncryptedText AS EncryptedBinaryAsText
UNION
SELECT 2 As RowNumber,
#binDecryptedText AS EncryptedBinary,
#strDecryptedText AS EncryptedBinaryAsText
As you can see in the code above, in the CONVERT statements ... I needed to add the additional "size" of the returning VARCHAR I was converting to. Previously my convert commands said ...
...CONVERT(VARCHAR, ...
... when they should have said ...
...CONVERT(VARCHAR(MAX), ...
Once I made that change, the returning HEX String was no longer being truncated.

Related

How to cast hex data string to a string db2 sql

How would you decode a hex string to get the value in text format by using a select statement?
For example my data in hex is:
4f004e004c005900200046004f00520020004200410043004b002d005500500020004f004e0020004c004500560045004c0020004f004e004500200046004f00520020004300520041004e004500530020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020000000
I want to decode it to get the string value using a select statement.
The value of the above is "ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES"
what I have tried is :
SELECT CAST('4f004e004c005900200046004f00520020004200410043004b002d005500500020004f004e0020004c004500560045004c0020004f004e004500200046004f00520020004300520041004e004500530020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020002000200020000000'
AS VARCHAR(30000) CCSID 37) from myschema.atable
The above sql returns the exact same hex string and not the decoded text string of "ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES" what I expected.
Is it possible to do this with a cast? If it is what will the syntax be?
My problem that I have is a system stores text data in a blob field and I want to use a select statement to see what the text data is in the blob field.
Db : Db2 on Ibm
Edit:
I have managed to covert the string to the hex value by using :
select hex(cast('ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES' as varchar(100) ccsid 1208))
FROM myschema.atable
This gives me the string in hex :
4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553
Now somehow I need to do the inverse and get the value.
Thanks.
Edit
Using the answer from Daniel Lema, I tried using the unhex function but my result that I got was :
|+<ßã|êâ ä.í&|+<áîá<|+áã|êäê +áë
Is this something to do with a CSSID? Or how should I convet the above to a readable string?
This is the table field definition if it will help the field with my data in is GDTXFT a BLOB :
I was able to take your shortened hex string and convert is to a valid EBCDIC string.
The problem I ran into is that the original hex code you receive comes in UTF-16LE (Thanks Tom Blodget). IBM's CCSID system does not have a distinction between UTF-16BE and UTF-16LE so I am at a loss there on how to convert it properly.
If it is in UTF-8 as you generated later, the following would work for you. It's not the prettiest but throw it in a couple functions and it will work.
Create or replace function unpivothex (in_ varchar(30000))
returns table (Hex_ char(2), Position_ int)
return
with returnstring (ST , POS )
as
(Select substring(STR,1,2), 1
from table(values in_) as A(STR)
union all
Select nullif(substring(STR,POS+2,2),'00'), POS+2
from returnstring, table(values in_) as A(STR)
where POS+2 <= length(in_)
)
Select ST, POS
from returnstring
;
Create or replace function converthextostring
(in_string char(30000))
returns varchar(30000)
return
(select listagg(char(varbinary_format(B.Hex_),1)) within group(order by In_table.Position_)
from table(unpivothex(upper(in_string))) in_table
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 1208)))) A on In_table.Hex_ = A.Hex_
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 37)))) B on A.Position_ = B.Position_
);
Here is a version if you're not on at least V7R2 TR6 or V7R3 TR2.
Create or replace function converthextostring
(in_string char(30000))
returns varchar(30000)
return
(select xmlserialize(
xmlagg(
xmltext(cast(char(varbinary_format(B.Hex_),1) as char(1) CCSID 37))
order by In_table.Position_)
as varchar(30000))
from table(unpivothex(upper(in_string))) in_table
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 1208)))) A on In_table.Hex_ = A.Hex_
join table(unpivothex(hex(cast('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ' as char(53) CCSID 37)))) B on A.Position_ = B.Position_
);
I tried the following solution I found published by Marcin Rudzki at Convert HEX value to CHAR on DB2, tested in my own Db2 for LUW v11 with a small modification.
the solution consists on creating a function just as Marcin suggested:
CREATE FUNCTION unhex(in VARCHAR(32000) FOR BIT DATA)
RETURNS VARCHAR(32000)
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC NO EXTERNAL ACTION
BEGIN ATOMIC
RETURN in;
END
To test the solution, lets create an HEXSAMPLE table with a HEXSTRING column loaded with the string representation of a HEX sequence:
INSERT INTO HEXSAMPLE (HEXSTRING) VALUES ('4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553')
Then exec the following query (and here it is different from the original proposal):
SELECT UNHEX(CAST(HEXTORAW(HEXSTRING) AS VARCHAR(2000) FOR BIT DATA)) as TEXT, HEXSTRING FROM HEXSAMPLE
With result:
TEXT HEXSTRING
---------------------------------------- --------------------------------------------------------------------------------
ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES 4F4E4C5920464F52204241434B2D5550204F4E204C4556454C204F4E4520464F52204352414E4553
I hope someone else can find a more direct solution. Also, if someone can explain why it works, it will be very interesting.
I question why you need to do this...
There's valid reasons to convert a hex string back to it's character equivalent...for instance somebody sends you a 32 byte string UUID and you want it back it it's 16 byte binary form.
But there's no reason ONLY FOR BACK-UP ON LEVEL ONE FOR CRANES should have been transformed to hex.
I suspect you need to post a new question asking why you're not getting readable strings in the first place.
However, in answer to this question... IBM i has an MI function Convert Character to Hex (CVTCH) that is easily called from any ILE langage. You could wrap that function call up into a user defined function in order to use it from SQL.
Note that you'll need to know what the hex string represents, EBCDIC, ASCII or Unicode, because you'll need to be able to tell the system what you've started with. From there there are ways to convert between encoding.
Here's an article that shows how to call the MI function from RPG.
Utilizing MI Functions in RPG Programs
A more modern free form version of the prototype that takes advantage of enhancements to the CCSID keyword might look like
dcl-pr FromHex extproc('cvtch');
charString char(32767) ccsid(*UTF8) options(*varsize);
hexString char(65534) ccsid(*HEX) const options(*varsize);
hexStringLen int(10) value;
end-pr;
With the above prototype, the system will treat the character string that comes back as UTF8 (ccsid 1208). But all I'm doing is telling the system how to interpret the bytes that come back. If the string was actually EBCDIC, I'm going to get garbage.
I think you could even defined the cvtch function directly as an external UDF without needing an ILE wrapper. I'd have to play around with that...
Disregard that idea...cvtch only has parameters, not a return value. Using an ILE wrapper is the best way to move the output parameter to a return value for use as a UDF.
The problem is that your original string is in ASCII format (actually with x'00' byte after each letter), and you have to convert it to EBCDIC.
Below is the solution for latin capital letters only:
select cast(translate(replace(mycol, x'00', x'')
, x'C1C2C3C4C5C6C7C8C9D1D2D3D4D5D6D7D8D9E2E3E4E5E6E7E8E940'
, x'4142434445464748494A4B4C4D4E4F505152535455565758595A20'
) as varchar(500) ccsid 37)
from mytab;
Every ASCII character is translated to the corresponding EBCDIC one.
x'00' symbols are removed.
cast (col_name as varchar(2000) ccsid ascii for sbcs data)

Conversion of Image type to Text

I'm having some trouble converting the below Image data type to text;
0x1F8B0800FC7E1F3000FF758FCB6EC2301045F748FCC3FC00D524518A0AAB967655D10D5D7AE324E360C9B1A3B179B4887FAF431254FAD8F83173E6DE3B27C14125425AAF2F47D9D6499AA742B16B021D03888A94C29350CE865098E9243E51287FD0DEC3236B6996E74B3111AA7115B18595DBB12686373A0CBDA867B511AADC4AF61452D87C34851B27B371F20AE0ADC6052B9D711C1308A60A45CD441645617684CB5FA534CF3B6DB12B13D14AAE446BA4B6DD2A4116D93D42BF438AF0BE2558EB929D772AC0CB3106B035C186781FDDD7529BC21DE36D651DFF5BE9A1744D6B2850052DBB92BCD7B686A607C97776D3C92648EE081916223A428A389FE1C30C73C07C91CD17E9BC075757B17FD12CEFD1F56831FA52354C0CFD588A213D346E3FB67088A33F099CEA3BD00CE080DC21C2EB530F3E531FE60772AB52FD0D7DD7397F01A585816657020000
I have tried casting as varbinary and then casting that to nvarchar, but get what looks like Unicode characters;
select top 1 cast(cast(body as varbinary(max)) as nvarchar(max)) from dbo.messagedata
How can I convert the above Image data into readable text?

How to create human-readable identifier with more than 64 bits of entropy on IBM Netezza?

I'm working on IBM Netezza/PureData and I want to add an identifier column to a table containing billions of new records every day, so that I can track each record as it travels through different tables and systems. I want this identifier to have more than 64 bits of entropy, as the column may contain (in time) more than 10^12 different records, and I want to avoid hash collisions. According to this table, 64 bits is not enough to avoid hash collisions with this number of records.
Not enough bits
So, on Netezza I can easily do
select hash8(123456) as id
which will return a 64-bit number (BIGINT):
-1789169473613552245
This is perfectly readable, but it has only 64-bits of entropy.
Not human readable
I can also do:
select hash(123456) as id
to create a 128-bit hash on Netezza. This has more than enough entropy, but it becomes an unreadable mess of unicode characters:
oð8^GþåíOpJ
I'm afraid this will cause trouble when I start combining this date with tables from other systems.
Enough bits and human readable?
So instead, I would like to create a human-readable identifier by, for example, converting this 128-bit unicode string to a base-62 string, containing only alphanumeric characters (0-9, a-z, A-Z). Something like:
6KMPOATg6Y5TbuEZlD59Dp
Any ideas on how to do this? Ideally with only (Netezza) SQL-code or functions...
Since you seem to have the SQL extensions available (hash() being one of the Netezza SQL extension functions), you could try doing 'rawtohex()' on the output of your hash()
e.g.
select rawtohex(hash(123456));
This gives a nice HEX string representation of the hashed data:
'E10ADC3949BA59ABBE56E057F20F883E'
MS SQL has base-64 support, in XML datatype.
declare #source varbinary(max), #encoded varchar(max), #decoded varbinary(max)
set #source = convert(varbinary(max), 'Hello Base64')
set #encoded = cast('' as xml).value('xs:base64Binary(sql:variable("#source"))', 'varchar(max)')
set #decoded = cast('' as xml).value('xs:base64Binary(sql:variable("#encoded"))', 'varbinary(max)')
select
convert(varchar(max), #source) as source_varchar,
#source as source_binary,
#encoded as encoded,
#decoded as decoded_binary,
convert(varchar(max), #decoded) as decoded_varchar
From http://blog.falafel.com/t-sql-easy-base64-encoding-and-decoding/

Varbinary and image conversion

I have an Access front end that links to a SQL Server backend.
There are 3 fields in a table that I am trying to convert to text from the backend:
o_name varbinary(2000)
O_PropertyBinary1 varbinary(2000)
O_PropertyBinary2 image
I can convert the o_name field using:
convert(varchar([max]),[O_Name])
and that works fine.
e.g. 4153534554 = ASSET
However, what can I use for the other two fields, as it seems I can't convert an image field and converting the O_PropertyBinary1 comes out with garbage characters.
The output is depended on the stored data an the appropriate conversion.
If the stored data is binary e.g. Bitmaps, converting to text will never give a usable result.
If data stored is text, it could be Varchar or NVarchar and kind conversion is depending.
in the example below VC_VB2NVarchar and VC_IMG2NVarchar would display your described garbage characters
Declare #tab Table(nvc NVarchar(100),vc Varchar(100)
,img image,vb VarBinary(200),img2 image,vb2 VarBinary(200))
Insert into #tab (nvc,vc) Values ('123456789','123456789')
Update #tab set vb=Convert(VarBinary(200),nvc),img=Convert(Image,Convert(Varbinary(max),nvc))
,vb2=Convert(VarBinary(200),vc),img2=Convert(Image,Convert(Varbinary(max),vc))
Select nvc,vc
,CONVERT(Nvarchar(100),vb) as NVC_VB2NVarchar
,CONVERT(Varchar(200),vb) as NVC_VB2Varchar
,CONVERT(Nvarchar(100),Convert(VarBinary(max),img)) as NVC_IMG2NVarchar
,CONVERT(Varchar(200),Convert(VarBinary(max),img)) as NVC_IMG2Varchar
,CONVERT(Nvarchar(100),vb2) as VC_VB2NVarchar
,CONVERT(Varchar(200),vb2) as VC_VB2Varchar
,CONVERT(Nvarchar(100),Convert(VarBinary(max),img2)) as VC_IMG2NVarchar
,CONVERT(Varchar(200),Convert(VarBinary(max),img2)) as VC_IMG2Varchar
from #Tab

How to compare varbinary in where clause in SQL Server

I want to compare varbinary type with byte array. I have tried so far:
DECLARE #data AS NVARCHAR(MAX)='4283'
Select * from table1 Where bindayData=CAST(#data AS VARBINARY)
But this does not work.
I note one strange behaviour of this: when I statically use it like
Select * from table1 Where bindayData=CAST('4283' AS VARBINARY)
then it works fine. But when I declare a variable, it doesn't work.
Please share your ideas.
Thanks,
Naresh Goradara
Try
DECLARE #data AS NVARCHAR(MAX)='4283'
The string constant '4283' is non-unicode in the CAST, one byte per character.
This gives 4 bytes varbinary 0x34323833
When you use NVARCHAR(MAX), then it changed to unicode N'4283'string with 2 bytes per character.
This gives 8 bytes varbinary, something like 0x0034003200380033
Using a style of 2 in the convert function does not truncate the final result. It leaves off the "0x" prefex in essence converting the result to a non-binary result.
Please refer to Microsoft's documentation. There is an example at the bottom:
https://learn.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-ver15
Select *
from table1
Where convert(varchar(max),bindayData,2) like '%4283%'