Sql - Bulk crypto hash generation - sql

Question

We're trying to implement change detection in out ETL process.
So we decided to get the cryptographic hash using
SET a.[HASH] = (SELECT
master.dbo.fn_varbintohexsubstring(0, HashBytes('md5', (SELECT TOP 1 * FROM customer_demographics_staging b WHERE b.customer_no = a.customer_no FOR XML RAW)), 1, 0))
FROM customer_demographics_staging a
For a table with 700k records and about 140 columns (we are yet to determine the changing columns), the query run for about half an hour before we canceled it.
Is there anyway, apart from reducing the number of queries we can improve this?

Answer 1

A couple of things. If the data type of the HASH column is varbinary(20), you don't need to concern yourself with converting the MD5 hash to a string; just store the hash bytes. To that end though, if you want to use a cryptographic hash for change detection, I'd use an inline table-valued function to get it. Here's an example that I cobbled together using AdventureWorks:
ALTER TABLE [HumanResources].[Employee] ADD [Hash] VARBINARY(20) NULL;
GO
CREATE FUNCTION dbo.CalculateHash(#EmployeeID AS INT)
RETURNS TABLE
AS
RETURN
SELECT e.[BusinessEntityID], HASHBYTES('md5', (
SELECT *
FROM [HumanResources].[Employee] AS [e2]
WHERE [e2].[BusinessEntityID] = e.[BusinessEntityID]
FOR XML RAW
)) AS [Hash]
FROM [HumanResources].[Employee] AS [e]
WHERE [e].[BusinessEntityID] = #EmployeeID
go
SELECT TOP 10 [e].*, ch.[Hash]
FROM [HumanResources].[Employee] AS [e]
CROSS APPLY dbo.[CalculateHash]([e].[BusinessEntityID]) AS [ch]
GO
That said, if it were me, I wouldn't bother with MD5 at all and just use the CHECKSUM() function (perhaps as a persisted computed column in the table). It supports taking multiple columns natively (so you don't incur the overhead of serializing the row to XML).

Answer 2

In line with what [Ben Thul] already said, I too tend to rely on BINARY_CHECKSUM() simply because its ease of use.
I'll agree that this function returns "but an int" which is 8 bytes while e.g MD5 will return a varbinary(16) which is twice as much of bytes so you get the square (not the double!) of 'result-space' meaning you'll end up with an incredibly much smaller chance on collisions. But paranoid me would like to add that even so, an exact match of MD5 values does not mean you also have the same (input) values!
In all honesty, I use the function only to eliminate differences. If the result of the checksum (or hash) is different then you can be 100% certain that the values are different too. If they are identical then you should still check the source-values in their entirety to see if there are no 'false matches'.
Your use-case seems to be the other way around: you want to find the ones that are different by eliminating the ones that are identical and short-cutting the latter by looking at the hash-code only. To be honest, I'm not a fan of the approach simply because you risk running into a collision causing a 'changed' record in your staging table to get the exact same hash value as the old one and thus being ignored when you want to copy the changes. Again, the chances are incredibly small, but like I said, I'm paranoid when it comes to this =)
If you'd wish to continue down this track nevertheless, some remarks:
HashBytes only supports an input of 8000 bytes. Given the overhead added by the XML syntax you might run into trouble with those 140 columns
I don't see any (good) reason to convert the result of HashBytes to something else before writing it to the table
Although FOR XML is pretty fast, wouldn't CONCAT be just as fast while at the same time resulting in a 'smaller' result (cf point 1)? I'll agree that it brings its own set of issues like when field1,field2,field3 are "hello", "world" "" would result in the same thing as "hello", "", "world" =/ You could get around this by CONCAT-ing the LEN() of each field too... not sure how much gain we'd have left though =)
I'm guessing you already have it, but is there an index, preferably unique and clustered, on the customer_no field in the staging table?