check for abstracts cut-and-pasted from the PDF

[mjpost]

Ingestion (perhaps via normalize_anth) should check for hyphenations that indicate an abstract has been cut-and-pasted from a PDF, e.g., "pro- vides". We could probably mostly correct this automatically, though note that there are situations where the hyphen should be retained and only the space deleted (e.g., "top- selling").

Here's an example.

[davidweichiang]

I believe cut-and-paste could also result in strange characters like LATIN SMALL LIGATURE FI (U+FB01) but I'm not sure if I've seen that.

[davidweichiang]

Maybe there should be a preventative step like printing a message in the final submission form saying 'Your abstract will appear in the Anthology exactly as entered here. If you copy and paste it from a PDF, please check it carefully, particularly for word breaks like "pro- vides".'

The second step would be detection; I guess the heuristic should be something like "look for strings of the form 'x- y' where 'xy' is a dictionary word but either 'x' or 'y' is not." If we can build a reliable detector, perhaps we could even ask Softconf to run it for us at submission time.

If you want to automatically correct, maybe the heuristic needs to be better than that. If the LaTeX is provided, we can use the LaTeX. Otherwise, I suppose we could train a CRF to do it, but I'm not sure if that is worth the time.

[davidweichiang]

Similar to issue #643, this affects the conference website, handbook, etc., as well. So it might be preferable to deal with this earlier than ingestion time.

[mjpost]

I'd love to see this in START, which is the only place it could be fixed by the author.

Fixing automatically at ingestion time sounds fine so long as we use the dictionary approach. Maintaining a CRF or having to dig up LaTeX sounds time-consuming on the part of the ingester.

[mjpost]

This could be a good thing to add to the proposed sanity-check.py script in ACLPUB.