Tokenization process: Allow zero-width 'infix' token (French tokenization)

[raphael0202]

Feature request

I've started to work on the tokenization of French texts in SpaCy. In French, the apostrophe ' is a breaking character and is used for elision (e.g: le avion is written l'avion, je ai is written j'ai,...). There are a few exceptions to this rule, like aujourd'hui that should not be splitted.

As elision is very common in French, it is not feasible to list all these elisions as exceptions in TOKENIZER_EXCEPTIONS. It would be much better to consider ' as a word breaking character in fr/punctuations.py, and add r'(?<=[{a}]\')(?=[{a}])'.format(a=ALPHA) to TOKENIZER_INFIXES.
However, in Tokenizer._attach_tokens, spaCy raises an error the regex matches a zero-width infix token.

In order for this to work, we should consider that if infix_start == infix_end (in tokenizer.pyx, line 292), we shouldn't add an infix token.

[honnibal]

That sounds like a reasonable approach.

One quick suggestion though: Would it be possible to enumerate the prefixes, and put them in the prefix regex?

[raphael0202]

Yes of course. I can work on this and submit a PR if you wish.

[ines]

Thanks for the PR. Just merged it and added regression tests – all works fine and as expected 👍 I think this will be a pretty useful feature for other languages as well.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.