Spacy tokenizer rule for exceptions that contain whitespace? - spacy

Question

When I create a pipeline with the default tokenizer for say English, I can then call the method for adding a special case:
tokenizer.add_special_case("don't", case)
The tokenizer will happily accept a special case that contains whitespace:
tokenizer.add_special_case("some odd case", case)
but it appears that does not actually change the behavior of the tokenizer or will never match?
More generally, what is the best way of extending an existing tokenizer so that the some patterns which normally would result in multiple tokens only create one token? For example something like [A-Za-z]+\([A-Za-z0-9]+\)[A-Za-z]+ should not result in three tokens because of the parentheses but in a single token, e.g. for asdf(a33b)xyz while the normal English rules should still apply if that pattern does not match.
Is this something that can be done somehow by augmenting an existing tokenizer or would I have to first tokenize, then find entities that match the corresponding token patterns and then merge the entity tokens?

Answer 1

As you found, Tokenizer.add_special_case() doesn't work for handling tokens that contain whitespace. That's for adding strings like "o'clock" and ":-)", or expanding e.g. "don't" to "do not".
Modifying the prefix, suffix and infix rules (either by setting them on an existing tokenizer or creating a new tokenizer with custom parameters) also doesn't work since those are applied after whitespace splitting.
To override the whitespace splitting behavior, you have four options:
Merge after tokenization. You use Retokenizer.merge(), or possibly merge_entities or merge_noun_chunks. The relevant documentation is here:
https://spacy.io/usage/linguistic-features#retokenization and https://spacy.io/api/pipeline-functions#merge_entities and https://spacy.io/api/pipeline-functions#merge_noun_chunks
This is your best bet for keeping as much of the default behavior as possible.
Subclass Tokenizer and override __call__. Sample code:
from spacy.tokenizer import Tokenizer
def custom_tokenizer(nlp):
class MyTokenizer(Tokenizer):
def __call__(self, string):
# do something before
doc = super().__call__(string)
# do something after
return doc
return MyTokenizer(
nlp.vocab,
prefix_search=nlp.tokenizer.prefix_search,
suffix_search=nlp.tokenizer.suffix_search,
infix_finditer=nlp.tokenizer.infix_finditer,
token_match=nlp.tokenizer.token_match,
)
# usage:
nlp.tokenizer = custom_tokenizer(nlp)
Implement a completely new tokenizer (without subclassing Tokenizer). Relevant docs here: https://spacy.io/usage/linguistic-features#custom-tokenizer-example
Tokenize externally and instantiate Doc with words. Relevant docs here: https://spacy.io/usage/linguistic-features#own-annotations
To answer the second part of your question, if you don't need to change whitespace splitting behavior, you have two other options:
Add to the default prefix, suffix and infix rules. The relevant documentation is here: https://spacy.io/usage/linguistic-features#native-tokenizer-additions
Note from https://stackoverflow.com/a/58112065/594211: "You can add new patterns without defining a custom tokenizer, but there's no way to remove a pattern without defining a custom tokenizer."
Instantiate Tokenizer with custom prefix, suffix and infix rules. The relevant documentation is here: https://spacy.io/usage/linguistic-features#native-tokenizers
To get the default rules, you read the existing tokenizer's attributes (as shown above) or use the nlp object’s Defaults. There are code samples for the latter approach in https://stackoverflow.com/a/47502839/594211 and https://stackoverflow.com/a/58112065/594211.

Answer 2

Use token match for combining multiple tokens to single one