Poor regex performance for word matching on postgres - sql

Question

I have a list of blocked phrases and I want to match for the existence of those phrases in user inputed text, but performance is very bad.
I am using this query :
SELECT value FROM blocked_items WHERE lower(unaccent( 'my input text' )) ~* ('[[:<:]]' || value || '[[:>:]]') LIMIT 1;
After my investigation I found out that world boundries [[:<:]] and [[:>:]] perform very badly knowing that blocked_items has 24k records in it.
For instance when I try to run this one:
SELECT value FROM blocked_items WHERE lower(unaccent( 'my input text ' )) ilike ('%' || value || '%') LIMIT 1;
it's very fast compared to the first one. The problem is that I need to keep the test for word boundries.
This check is performed frequently on a large program so the performance is very important for me.
Do you guys have any suggestions for making this faster?
EXPLIAN ANALYZE screenshot

Answer 1

Since you know that the LIKE (~~) query is fast and the RegEx (~) query is slow, the easiest solution is to combine both conditions (\m / \M are equivalent to [[:<:]] / [[:>:]]):
SELECT value FROM blocked_items
WHERE lower(unaccent('my input text')) ~~ ('%'||value||'%')
AND lower(unaccent('my input text')) ~ ('\m'||value||'\M')
LIMIT 1;
This way the fast query condition filters out most of the rows and then the slow condition discards the remaining ones.
I am using the faster case sensitive operators, assuming that value is already normalized. If that is not the case, drop the (then redundant) lower() and use the case sensitive versions as in the original queries.
On my test set with 370k rows that speeds up the query from 6s (warm) to 90ms:
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.00..1651.85 rows=1 width=10) (actual time=89.702..89.702 rows=1 loops=1)
-> Seq Scan on blocked_items (cost=0.00..14866.61 rows=9 width=10) (actual time=89.701..89.701 rows=1 loops=1)
Filter: ((lower(unaccent('my input text'::text)) ~~ (('%'::text || value) || '%'::text)) AND (lower(unaccent('my input text'::text)) ~ (('\m'::text || value) || '\M'::text)))
Rows Removed by Filter: 153281
Planning Time: 0.097 ms
Execution Time: 89.717 ms
(6 rows)
However we are still doing a full table scan and performance will vary based on the position in the table.
Ideally we can answer the query in near constant time by using an index.
Let's rewrite the query to use Text Search Functions and Operators:
SELECT value FROM blocked_items
WHERE to_tsvector('simple', unaccent('my input text'))
## phraseto_tsquery('simple', value)
LIMIT 1;
First we split the input into search vectors and then check if the blocked phrase matches those vectors.
This takes about 440ms for the test query – a step back from our combined query:
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.00..104.01 rows=1 width=10) (actual time=437.761..437.761 rows=1 loops=1)
-> Seq Scan on blocked_items (cost=0.00..192516.05 rows=1851 width=10) (actual time=437.760..437.760 rows=1 loops=1)
Filter: (to_tsvector('simple'::regconfig, unaccent('my input text'::text)) ## phraseto_tsquery('simple'::regconfig, value))
Rows Removed by Filter: 153281
Planning Time: 0.063 ms
Execution Time: 437.772 ms
(6 rows)
Since we can't use tsvector ## tsquery for indexing of tsquery, we can rewrite the query again to check if the blocked phrase is a subquery of the input phrase using the tsquery #> tsquery Text Search Operator, which can then be indexed using tsquery_ops from the GiST Operator Classes:
CREATE INDEX blocked_items_search ON blocked_items
USING gist (phraseto_tsquery('simple', value));
ANALYZE blocked_items; -- update query planner stats
SELECT value FROM blocked_items
WHERE phraseto_tsquery('simple', unaccent('my input text'))
#> phraseto_tsquery('simple', value)
LIMIT 1;
The query can now use an index scan and takes 20ms with the same data.
Since GiST is a lossy index, query times can vary a bit depending on how many rechecks are required:
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=0.54..4.23 rows=1 width=10) (actual time=19.215..19.215 rows=1 loops=1)
-> Index Scan using blocked_items_search on blocked_items (cost=0.54..1367.01 rows=370 width=10) (actual time=19.214..19.214 rows=1 loops=1)
Index Cond: (phraseto_tsquery('simple'::regconfig, value) <# phraseto_tsquery('simple'::regconfig, unaccent('my input text'::text)))
Rows Removed by Index Recheck: 4028
Planning Time: 0.093 ms
Execution Time: 19.236 ms
(6 rows)
One great advantage of using full text search is that you can now use language specific matching of words by using search configurations (regconfig).
The above queries are all using the default 'simple' regconfig to match the behavior of the original query. By switching to 'english' you can also match variations of the same word like cat and cats (stemming) and common words without significance like the or my will be ignored (stop words).