Best Practice For Lookup Tables In SQL - sql

Question

I am fairly new to SQL so apologies if this questions sounds odd.
I keep coming across the issue of poor data. As an example London could be stored as LON, London UK, London England etc. Prior to using SQL I had lots of Excel lookup tables where in the first column I would have the original, and in the 2nd helper column I would have the corrected version. As an example:
Name Name_1
London, UK London
Lon London
LON London
London London
London, England London
LND London
Is there a straightforward way of doing this in SQL? I am currently trying to create lookup tables and then use joins. This is becoming tricky as I do not always have corrections for every instance, so in most scenarios my lookup tables have fewer items than the tables I'm joining them to.
I've been teaching myself about stored procedures and I wondered if this could solve the problem. Trouble is my searching on the subject of lookup tables is coming up empty.
Any advice or pointers would be gratefully received, even if it is just to say that this cannot be done.
Thanks as always for you help and apologies for the long post.

Answer 1

You can join to the lookup table and preferrable use the value given there. If not found, use the original:
SELECT t1.FirstName, LookupField = ISNULL(t2.Name_1, t1.LookupField)
FROM People as t1
LEFT INNER JOIN TableLookupCities as t2 ON t1.LookupField = t2.Name
Make sure, that for each name there is at most one match in TableLookupCities, or else the join will produce multiple results. Create a unique index on TableLookupCities.Name:
CREATE UNIQUE (CLUSTERED) INDEX djgndkg ON TableLookupCities (Name) INCLUDE (Name_1)

Answer 2

You don't have to do any of the other things, just return the original if you don't have a translation for it.
SELECT
t1.FirstName,
t1.LookupField,
case when t2.Name_1 is null
then t1.lookupfield
else t2.name_1 end Name_1
FROM People as t1
LEFT INNER JOIN TableLookupCities as t2
ON t1.LookupField = t2.Name

Answer 3

As mentioned above, bad data is it's own problem. Data cleaning is an industry in itself, so you've got a huge range of options for this kind of problem, from the simple and straightforward, to the elaborate tries to fix everything bells and whistles. What's "best" depends on your situation and needs.
Certainly it's possible to continue to expand this lookup table to cater for a growing number of standard errors/variations, but if this is a constant flow of information there's a maintenance overhead. This may well be adequate for your needs, so don't shy away from it just because there's fancier alternatives.
It's quite common place to trade the reliability of manual human intervention for the scalability of automated approaches; this is much easier to maintain and grow, but (depending on the nature of your problem) may make mistakes.
Eg 1. Use a pattern based approach (Contains, LIKE, RegEx) to finds something that seems like a reasonable fit. That maybe fine in some situations, such as when Name_1 is a static, well understood list, so you can make sure the results are usually going to good enough.
+ easy to setup/understand
+ more flexible than full list
- still takes a bit of maintenance
- hopeless in complex/poorly understood situations
Eg 2. In the more general case, you can use the text search capabilities offered by the database to "score" how good a match one value is against the other, and pick the best matching option. Again this isn't fool proof or safe in all contexts, and it's a bit more work to setup, but it's a lot more robust. This is a bit more performance intensive, so the size of the datasets involved, the timescales your working to, and the infrastructure available are also considerations.
+ fairly good success rates
- slower setup
- larger performance overheads
eg 3. Another option would be something more domain specific. In this case, it's spatial data, so you could use a 3rd party geocoding service as a means of validation.
+ high success rate
+ capable of dealing with huge ranges of values,
- may incur additional costs
- hardest/slowest to setup

Answer 4

The bottom line... bad data is bad data and it takes a lot of work to either use bad data or clean bad data or both.
UPDATE AFTER CLARIFICATION
Build your own ETL (Extract, Transform, Load) process to handle all the variant incoming data. Your ETL process will most likely get modified with each new batch of data you receive because you will have to trap for new "Bad Data" variants.
Import the data into an ALL VARCHAR table
Run the ETL Process
Good Data goes into real data tables
Bad data goes into an exception table
Repeat
Modify ETL Process
Run ETL Process
Until No More Exceptions
-- End Update
If you use LEFT JOIN you can identify the missing values fairly easily.
SELECT
t1.FirstName,
t1.LookupField,
t2.Name_1
FROM People as t1
LEFT INNER JOIN TableLookupCities as t2
ON t1.LookupField = t2.Name
Anywhere t2.Name_1 returns a NULL you know you need to add that "LookupField" to your lookup table. Here is a good book for learning database design Database Design for Mere Mortals
-- Group By to Find Missing Unique Values
t1.LookupField,
t2.Name_1
FROM People as t1
LEFT INNER JOIN TableLookupCities as t2
ON t1.LookupField = t2.Name
GROUP BY
t1.LookupField,
t2.Name_1