remove duplicates from almost similar records in SQL - sql

Question

I have a table with three columns: Name, Address, City. This table is around a million records long. The name and address fields can probably have duplicates.
An example of duplicate names are:
XYZ foundation Coorporation
XYZ foundation Corp
XYZ foundation Co-orporation
Or another example
XYZ Center
XYZ Ctr
An example of duplication in addresses would be
60909 East 34TH STREET BAY #1
60909 East 34TH ST. BAY #1
60909 East 34TH ST. BAY 1
As you can see, the name and address fields are duplicates, but only to the human eye, because we understand abbreviations and short forms. How do I build this into a select statement in SQL Server? If not SQL Server, is there another way to scan and remove such duplicates?

Answer 1

The approach that I used is better suited for surnames, but I used it for company names as well. Most likely it will not work well for addresses.
Stage 1
Add a column to the table that stores a "normalized" company name. In my case I've written a function that populates the column via a trigger. The function has a set of rules, like this:
adds one space in the front and one in the back
replaces single char symbols ~`!##$%^&*()=_+[]{}|;':",.<>? with space (all except / -)
replaces multi-char tokens with space: T/A C/- P/L
replaces single char symbols -/ with space
replaces multi-char tokens with space: PTY PTE INC INCORPORATED LTD LIMITED CO COMPANY MR DR THE AND 'TRADING AS' 'TRADE AS' 'OPERATING AS'
replaces CORPORATION with CORP
trim all leading and trailing spaces
replace multiple consecutive spaces with single space
Note: when dealing with multi-char tokens surround them with spaces
I looked through my data and made these rules up. Adjust them for you case.
Stage 2
I used the so-called Jaro-Winkler metric to calculate the distance between two normalized company names. I implemented the function that calculates this metric in CLR.
In my case my goal was to check for duplicates as a new entry is added to the system. The user enters the company name, program normalizes it and calculates the Jaro-Winkler distance between the given name and all existing names. The closer the distance to 1, the closer the match. The user saw existing records ordered by the relevance and could decide whether the company name that he just entered already exists in the database, or he still wanted to create a new one.
There exist other metrics that try to perform fuzzy search, like Levenshtein distance. Most likely, you'll have to use different metrics for names and addresses, because the types of mistakes are significantly different for them.
SQL Server has built-in functions to do fuzzy search, but I didn't use them and I'm not sure if they are available in standard editions or only enterprise, e.g. CONTAINSTABLE
Returns a table of zero, one, or more rows for those columns
containing precise or fuzzy (less precise) matches to single words and
phrases, the proximity of words within a certain distance of one
another, or weighted matches.
Note
When I was looking into this topic I came to the conclusion that all these metrics (Jaro-Winkler, Levenstein, etc.) look for simple mistypes, like a missed/extra letter or two letters swapped. In my and your cases this approach as-is would perform poorly, because you effectively have a dictionary of contractions first and then on top of that there can be simple mistypes. That's why I ended up doing it in two stages - normalization and then applying the fuzzy search metric.
To make a list of rules that I mentioned above I made a dictionary of all words that appear in my data. Essentially, take each Name and split it into multiple rows by space. Then group by found tokens and count how many times they appear. Manually look through the list of tokens. This list should not be too long when you remove rare tokens from it. Hopefully common words and contractions would be easy to spot. I would imagine that the word Corporation and "Corp" would appear many times, as opposed to the actual company name XYZ. Those odd mistypes like "Coorporation" should be picked up by the fuzzy metric later.
In a similar way make a separate dictionary for Addresses, where you would see that Street and St. appear many times. For addresses you can "cheat" and get a list of common words from the index of some city map (street/st, road/rd, highway/hwy, grove/gv, etc.)
This is my implementation of the Jaro-Winkler metric:
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
public partial class UserDefinedFunctions
{
/*
The Winkler modification will not be applied unless the percent match
was at or above the WeightThreshold percent without the modification.
Winkler's paper used a default value of 0.7
*/
private static readonly double m_dWeightThreshold = 0.7;
/*
Size of the prefix to be concidered by the Winkler modification.
Winkler's paper used a default value of 4
*/
private static readonly int m_iNumChars = 4;
[Microsoft.SqlServer.Server.SqlFunction(DataAccess = DataAccessKind.None, SystemDataAccess = SystemDataAccessKind.None, IsDeterministic = true, IsPrecise = true)]
public static SqlDouble StringSimilarityJaroWinkler(SqlString string1, SqlString string2)
{
if (string1.IsNull || string2.IsNull)
{
return 0.0;
}
return GetStringSimilarityJaroWinkler(string1.Value, string2.Value);
}
private static double GetStringSimilarityJaroWinkler(string string1, string string2)
{
int iLen1 = string1.Length;
int iLen2 = string2.Length;
if (iLen1 == 0)
{
return iLen2 == 0 ? 1.0 : 0.0;
}
int iSearchRange = Math.Max(0, Math.Max(iLen1, iLen2) / 2 - 1);
bool[] Matched1 = new bool[iLen1];
for (int i = 0; i < Matched1.Length; ++i)
{
Matched1[i] = false;
}
bool[] Matched2 = new bool[iLen2];
for (int i = 0; i < Matched2.Length; ++i)
{
Matched2[i] = false;
}
int iNumCommon = 0;
for (int i = 0; i < iLen1; ++i)
{
int iStart = Math.Max(0, i - iSearchRange);
int iEnd = Math.Min(i + iSearchRange + 1, iLen2);
for (int j = iStart; j < iEnd; ++j)
{
if (Matched2[j]) continue;
if (string1[i] != string2[j]) continue;
Matched1[i] = true;
Matched2[j] = true;
++iNumCommon;
break;
}
}
if (iNumCommon == 0) return 0.0;
int iNumHalfTransposed = 0;
int k = 0;
for (int i = 0; i < iLen1; ++i)
{
if (!Matched1[i]) continue;
while (!Matched2[k])
{
++k;
}
if (string1[i] != string2[k])
{
++iNumHalfTransposed;
}
++k;
// even though length of Matched1 and Matched2 can be different,
// number of elements with true flag is the same in both arrays
// so, k will never go outside the array boundary
}
int iNumTransposed = iNumHalfTransposed / 2;
double dWeight =
(
(double)iNumCommon / (double)iLen1 +
(double)iNumCommon / (double)iLen2 +
(double)(iNumCommon - iNumTransposed) / (double)iNumCommon
) / 3.0;
if (dWeight > m_dWeightThreshold)
{
int iComparisonLength = Math.Min(m_iNumChars, Math.Min(iLen1, iLen2));
int iCommonChars = 0;
while (iCommonChars < iComparisonLength && string1[iCommonChars] == string2[iCommonChars])
{
++iCommonChars;
}
dWeight = dWeight + 0.1 * iCommonChars * (1.0 - dWeight);
}
return dWeight;
}
};

Answer 2

You could look for a more customized solution, together with the DIFFERENCE function for instance. (see: DIFFERENCE function, SQL Server)
Is it possible for Name and City to be logically similar yet different, as well?
Since there's a lot of room for variations here and only you have access to the real data, only you can check what works and what kind of exceptions you basically have there.
But hopefully this will get you started.
-- Creating the test set
DECLARE #TESTTABLE TABLE (Name VARCHAR(256), City VARCHAR(256), Address VARCHAR(256))
INSERT INTO #TESTTABLE VALUES ('Billy bob' ,'New York' ,'Baker street 125')
INSERT INTO #TESTTABLE VALUES ('Billy bob' ,'New York' ,'Baker street 120')
INSERT INTO #TESTTABLE VALUES ('Billy bob' ,'New York' ,'Baker st 125')
INSERT INTO #TESTTABLE VALUES ('Billy bob' ,'New York' ,'Mallroad 1')
INSERT INTO #TESTTABLE VALUES ('James Dean' ,'Washington DC' ,'Primadonna road 15 c 100')
INSERT INTO #TESTTABLE VALUES ('James Dean' ,'Washington DC' ,'Primadonna r 15')
INSERT INTO #TESTTABLE VALUES ('Got Nuttin' ,'Philly' ,'Mystreet 1500') -- Doesn't show, since no real duplicates
And then, after the test data, the actual query.
-- The query
;WITH CTE AS
(SELECT DISTINCT SRC.RN, T1.*, DIFFERENCE(T1.Address, T2.Address) DIFF_FACTOR
FROM #TESTTABLE T1
JOIN #TESTTABLE T2 ON T1.Name = T1.Name AND T2.City = T1.City AND T1.Address <> T2.Address
JOIN (SELECT DENSE_RANK() OVER (ORDER BY Name, City) RN, Name, City FROM #TESTTABLE T3 GROUP BY Name, City HAVING COUNT(*) > 1) SRC
ON SRC.City = T1.City AND SRC.Name = T1.Name)
SELECT DISTINCT RN, Name, City, COUNT(DISTINCT C.Address) Address_CT
, STUFF((SELECT ','+B.Address
FROM CTE B
WHERE B.RN = C.RN AND B.DIFF_FACTOR = C.DIFF_FACTOR
ORDER BY B.Address ASC
FOR XML PATH('')),1,1,'') AllAdresses
, DIFF_FACTOR
FROM CTE C
WHERE DIFF_FACTOR > 1 -- Comment this row to see that 'Mallroad 1' was considered to be too different from the rest, so this filter prevents us from considering that in the result set
GROUP BY RN, Name, City, DIFF_FACTOR
ORDER BY RN ASC, DIFF_FACTOR DESC
That is probably not the most effective - or accurrate - way to go about doing this, but it's a good place to start and to show what can be done. If there's a chance for Name and City to also be different but duplicates to human eyes, you could modify the query to match any two identical column values, comparing the third. But it gets really difficult to automate comparisons in cases where you have one identifying column, and both of the others can be different from one another to varying degrees.
I suspect you need to make several queries to first sort out the biggest mess, and eventually find the last most evasive "duplicates" by hand, a few at a time.