Check for similarity in string using Postgres - sql

Question

I cannot disclose the actual data so I am just using an example. I have two tables one is a dictionary table which has the IDs for the titles. The second table is new data that's coming into the database and doesn't have IDs I need to update the ID for the new data by checking in the dictionary table if I have something similar already in there or else update the dictionary with the new value and get a new ID for it and update the same for the new data. The Expected ID column in the second table is what I expect them to be updated as.
ID Title
-- ------------------------
1 Aliens
2 The Hunger Games
3 John Wick
4 Alien vs Predator
ID Title Expected ID
----------------------------------------------------------------
null The Hunger Games: Mockingjay Part I 2
null The Hunger Games: Mockingjay Part II 2
null John Wick (2014) 3
null John Wick Chapter 2 3
null Alien 1
null Aliens 1
null Alien 3 1
null Lord of the Rings 5 (New ID generated)

Answer 1

This seems like a task for pg_trgm. It provides you with the % operator, which returns true if the two strings are "close enough". You can tune what close enough means via changing pg_trgm.similarity_threshold. You can create an index to accelerate this operation. The higher similarity_threshold is set, the more acceleration you are likely to achieve.
If I take you first shown table as foo1 and all the distinct titles from your desired output as foo2, then this query gives reasonable results:
select *, foo1.title <-> foo2.title as distance from foo2 left join foo1 on foo1.title % foo2.title;
title | id | title | distance
--------------------------------------+--------+-------------------+-----------
The Hunger Games: Mockingjay Part I | 2 | The Hunger Games | 0.5142857
The Hunger Games: Mockingjay Part II | 2 | The Hunger Games | 0.5277778
John Wick (2014) | 3 | John Wick | 0.3333333
John Wick Chapter 2 | 3 | John Wick | 0.5
Alien | 1 | Aliens | 0.375
Alien | 4 | Alien vs Predator | 0.6666666
Aliens | 1 | Aliens | 0
Alien 3 | 1 | Aliens | 0.5
Alien 3 | 4 | Alien vs Predator | 0.7
Lord of the Rings | (null) | (null) | (null)
If you want one a single output row for each foo2, showing the "best" match only, then you would use a LEFT JOIN LATERAL:
select *, a.title <-> foo2.title as distance from foo2 left join lateral
(select * from foo1 where foo1.title % foo2.title order by foo1.title <-> foo2.title limit 1) a
on true;
title | id | title | distance
--------------------------------------+--------+------------------+-----------
The Hunger Games: Mockingjay Part I | 2 | The Hunger Games | 0.5142857
The Hunger Games: Mockingjay Part II | 2 | The Hunger Games | 0.5277778
John Wick (2014) | 3 | John Wick | 0.3333333
John Wick Chapter 2 | 3 | John Wick | 0.5
Alien | 1 | Aliens | 0.375
Aliens | 1 | Aliens | 0
Alien 3 | 1 | Aliens | 0.5
Lord of the Rings | (null) | (null) | (null)
How to replace the NULL in the "id" column (where there are no close-enough matches) with a newly generated id is a separate question, and you should ask separate questions separately.
For any realistically sized datasets, it is unlikely you would be able to just blindly accept whatever a query like the above produces, at least not if you want high quality results. Rather, you could have the computer generate something like the above as recommendations, and then offer them (in a convenient interface) for a human to ratify, reject, or investigate further.