advice on improving query performance ~ 1 day - sql

Question

I have 5 tables - Each have tens of thousands of records
1 main/very important table (TABLE A)
2 other tables (TABLES B/C) that still important but not as important as table
2 side tables (TABLES D/E)that hold primary keys between A<=>B and A<=>C i.e. only have two columns each
The 3 main tables have ~140 columns each, all have the same column names
The purpose of my query is to perform column level matching between all the tables A<=>D<=>B and A<=>E<=>C in one query
The final query will have about 286 columns (two ID columns from each main table,
select tableA.ID1 as [TABLEAID1],
tableA.ID2 as [TABLEAID2],
tableB.ID1 as [TABLEBID1],
tableB.ID2 as [TABLEBID2],
tableC.ID1 as [TABLECID1],
tableC.ID2 as [TABLECID2],
fn_TESTMatcher(tableA.[postCode], tableB.[postCode],) as
[TABLEAB.postCode.RESULT],
fn_TESTMatcher(tableA.[CityCode], tableB.[CityCode],) as
[TABLEAB.CityCode.RESULT],
.
.
. x238 more 'fn_TESTMatcher(...) as xyz' columns
.
INTO #Results
From tableA WITH (NOLOCK)
FULL JOIN tableD WITH (NOLOCK) ON tableA.ID1 = tableD.A
) FULL JOIN tableB WITH (NOLOCK) ON tableD.B = tableB.ID1
) FULL JOIN tableE WITH (NOLOCK) ON tableA.ID1 = tableE.A
) FULL JOIN tableC WITH (NOLOCK) ON tableE.B = tableC.ID
fn_TESTMatcher is a function, it is fed the same column from two main tables, then it removes/replaces special characters/abbreviations, and then tries to match them, if they match it returns a bit '1', if not then a bit '0'.
at the moment it takes about a day to run (i can't really time it with some sort of query timer), I can comment out all the columns except for the last and run it and its fairly quick, but i dont think i can just scale that up
Does anyone have some advice? My first assumption is to start googling on what indexes are and ...maybe.. apply it to the ID1 of every table although I'm a bit hesitant on a) messing up my tables and b) adding an index that ends up being useless
===========================================
update 2: table structure wise all the columns for all the main tables are varchars, length 100-250 characters, where ID (primary key) is not nullable
With the two side tables, they just have two columns, both varchar, 100 character limit (they're both foreign keys). The most important table's ID in this is not nullable
for functions, i technically have two:
FUNCTION [dbo].[fn_TESTStripCharacters]
(
#String NVARCHAR(MAX) ,
#MatchExpression VARCHAR(255)
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
DECLARE #expres VARCHAR(50) = '%[~,#,#,^,_,+,-,$,%,&,/,|,\,*,(,),.,!,`,:,<,>,?]%'
WHILE PATINDEX( #expres, #String ) > 0
SET #String = REPLACE(REPLACE(REPLACE( #String, SUBSTRING( #String, PATINDEX( #expres, #String ), 1 ),''),';',''),'-','')
RETURN #String
END
and second function
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER FUNCTION [dbo].[fn_TESTMatcher](#Field1 NVARCHAR(MAX), #Field2
NVARCHAR(MAX))
RETURNS BIT
BEGIN
SET #Field1 = UPPER(LTRIM(RTRIM(REPLACE(dbo.fn_TESTStripCharacters(#Field1,#SpecialCharacters),'-',''))))
SET #Field2 = UPPER(LTRIM(RTRIM(REPLACE(dbo.fn_TESTStripCharacters(#Field2,#SpecialCharacters),'-',''))))
SET #Field1 = REPLACE(#Field1,' RD ',' ROAD ')
SET #Field2 = REPLACE(#Field2,' RD ',' ROAD ')
SET #Field1 = REPLACE(#Field1,' ST ',' STREET ')
SET #Field2 = REPLACE(#Field2,' ST ',' STREET ')
SET #Field1 = REPLACE(#Field1,' ','')
SET #Field2 = REPLACE(#Field2,' ','')
RETURN
CASE WHEN #Field1=#Field2
THEN '1'
ELSE '0'
END
END
=============================
update 2
Example table data - assuming the same two records exist in all 3 tables (not always the case )
TableA (main + most important table):
ID1 ID2 postCode, cityCode, ................
10001 1221 IG11PJ London ................
10230 1022 IG22PJ Nottingham ................
tableB (slightly less important table)
ID1 ID2 postCode, cityCode, ................
10031 1011 IG1 1PJ london ................
10980 982 IG2 2PJ nottingham ................
tableC (slightly less important table)
ID1 ID2 postCode, cityCode, ................
10551 1011 iG1 1pj london ................
20980 982 iG2 2pJ nottingham ................
tableD (side table)
A B
10001 10031
10230 10980
table E (side table)
A B
10001 10551
10230 20980

Answer 1

If Tables A, B, and C should be identical save for formatting differences I would suggest you create 3 CTEs, the first selecting TableA ID and a HASHBYTES of all other columns (columns will need to be cast to char/varchar so any formatting and replacing can take place there), the second CTE the same for Table B and the third for Table C.
Then just match the HASHBYTES values.
As has already been said though, without sample data, table structures, DDL for the function etc. we are just guessing.
Sean and Milney both make very good points regarding scalar vs inline table function and use of NOLOCK

Answer 2

I see these as a task that does not belong in one query. I would create a new set of tables (or these tables is you have a backup/don't need to preserve the data) and then perform your data cleaning steps into those new tables.
Once you have happy the data has been normalized then do a single query to compare the tables.
Trying to put it all in one query gives no advantage and you don't step wise make progress. For example, if you find you forgot to strip spaces out of once field you have to redo EVERYTHING. If you make new tables with the "cleaned" data you can incrementally invest time on cleaning the data (which is clearly the slow part of this process) till the data is perfect and then run your quick comparison. Forgot something -- it is a relatively quick update and run.

Answer 3

Instead of all the headaches you are running through, making copies of everything, and then trying to parse based on functions that can not be optimized, I would suggest the following. You state you have a column that gets stripped of special characters. I would add a "CleanKey" column for each table and represented column. Then, via a possible table trigger, or before the add/save, pre-clean that value into the "CleanKey" column and you are done. Then have an index on THOSE "Clean" columns and do a direct join.
Since the rest of the system does not know of these "Clean" columns, you can add the columns, clean them out of whatever function you have and not worry about duplicating or otherwise ruining other data.
Yes, it may take a bit to pre-"Clean" these columns, but then its DONE. Your query should be fast after that.

Answer 4

I would agree with the others that cleaning these string values would be a good idea. But since you still need to accomplish that and I absolutely hate loops and scalar functions with a passion I decided to roll up an inline table valued function instead of these two nested scalar functions. I am not using any loops here and the performance might surprise you.
I am using a tally or numbers table for this. I like to keep one of these around as view. Here is the code for the view I use.
create View [dbo].[cteTally] as
WITH
E1(N) AS (select 1 from (values (1),(1),(1),(1),(1),(1),(1),(1),(1),(1))dt(n)),
E2(N) AS (SELECT 1 FROM E1 a, E1 b), --10E+2 or 100 rows
E4(N) AS (SELECT 1 FROM E2 a, E2 b), --10E+4 or 10,000 rows max
cteTally(N) AS
(
SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM E4
)
select N from cteTally
GO
Then you can use this tally table to derive a set based approach to accommodate the business rules you have for deciding if two values match. You also don't need comma delimiters in here. In your example you had a comma nearly every other character in the list of values to remove. A single instance of each character is sufficient.
create function [dbo].[fn_TESTMatcher_Sean]
(
#Field1 nvarchar(max)
, #Field2 nvarchar(max)
, #CharsToRemove nvarchar(max)
) returns table as
RETURN
with MyValues1 as
(
select substring(#Field1, N, 1) as MyChar
, t.N
from cteTally t
where N <= len(#Field1)
and charindex(substring(#Field1, N, 1), #CharsToRemove) = 0
)
, MyValues2 as
(
select substring(#Field2, N, 1) as MyChar
, t.N
from cteTally t
where N <= len(#Field2)
and charindex(substring(#Field2, N, 1), #CharsToRemove) = 0
)
select convert(bit, case when mv1.MyResult = mv2.MyResult then 1 else 0 end) as IsMatch
from
(
select distinct MyResult =
replace(
replace(replace(STUFF((select MyChar + ''
from MyValues1 mv2
order by mv2.N
FOR XML PATH(''),TYPE).value('.','NVARCHAR(MAX)'), 1, 0, '')
, ' RD ', ' ROAD ')
, ' ST ', ' STREET ')
, ' ', '')
from MyValues1 mv
) mv1
cross join
(
select distinct MyResult =
replace(
replace(replace(STUFF((select MyChar + ''
from MyValues2 mv2
order by mv2.N
FOR XML PATH(''),TYPE).value('.','NVARCHAR(MAX)'), 1, 0, '')
, ' RD ', ' ROAD ')
, ' ST ', ' STREET ')
, ' ', '')
from MyValues2 mv
) mv2
;
Give this a shot and let me know if this works in your environment.
For example:
select *
from fn_TESTMatcher_Sean('123 any st rd or something', '123 any street road or something', '%[~,##^_+-$%&/|\*().!`:<>?]%')
The above returns 1 because they are a match under the rules you defined.