Ingest RANLP 2019

[mjpost]

Also updated latex_to_unicode.py to remove \cite from abstracts (maybe @davidweichiang wants to have a look at that: 14a75ba).

[mjpost]

The current code transforms \cite{key} to \cite key and then deletes \cite, which is not what we want.

Since we cannot (easily) algorithmically distinguish parenthetical and text citations (and people do not always use \citet and \citep correctly), I think I am in favor of just leaving them as they are. Unless you register disagreement, I’ll update the code and the PR to do that.

Noting this in #666—Softconf should also look for \cite.

[akoehn]

My position on LaTeX code in abstracts: We cannot (and should not) do post-processing and guess what the authors wanted. The submission sites clearly ask for plaintext and we should handle it as such.

If the authors cannot be bothered to do the cleanup themselves, so be it. The only helpful thing would be a warning by softcfonc à la “You seem to have entered LaTeX code into this field. We expect plain text and will not perform any post-processing.” or similar (I don’t know whether there is any post-processing going on currently).

[mjpost]

Agreed. I like things to be right but chasing this long tail of manual corrections is too tiring.

[mjpost]

Okay, this is ready for review now I believe. I updated latex_to_unicode to (a) restore the space between control code and open brace and (b) no longer remove remaining control codes.

[mjpost]

The change here is that LaTeX commands not handled by the above are left in place instead of removed. We also restore the space that was inserted to ease handling. So \nonexistentcommand will be left as-is, as will \cite{citekey}. The view is that it cannot be our responsibility to ensure these are all correct.

[davidweichiang]

The rationale behind deleting all TeX commands was that things like \textsc, \underline, etc. I think really would be appropriate to delete. I agree that \cite is bad to delete, but think other commands might be better to delete. I can't remember if there were other problematic commands like \cite.

[mjpost]

The argument for a default policy of non-deletion is that it is a mistake to have stray latex in here at all, but when we can't automatically fix it (with the current list of cases we handle), it's better to leave the mistake exposed than to silently remove it, since removing it prevents later (possibly manual) correction.

What about turning \cite{x} into CITATION(x)? This removes the LaTeX and also simplifies the regex code since I don't have to worry about carving out an exception when we remove all stray codes at the end (assuming we don't change that, since it's the current default).

[davidweichiang]

Is the suggestion in your second paragraph to do \cite{x} -> CITATION(X) and then to remove all remaining TeX commands? I think I prefer that. I would actually suggest \cite{x} -> (CITATION) which might read slightly better, e.g., "The sky is blue (CITATION)" rather than "The sky is blue CITATION(smith:1900)".

I remember another gray-area case: I have definitely seen \footnote used in abstracts.

[mjpost]

This is done.

I think footnotes should just be removed, and added that, too.

[davidweichiang]

Are you worried about cases like \footnote{This is \emph{really} important.}?

[mjpost]

Clearly not :) But I independently ran into this while reprocessing EMNLP (#736), where an (obvious) common instance is \footnote{\url{...}}.

We really need to add unit tests so we know we're handling these issues correctly.

[davidweichiang]

Oh yes, that's an extremely common case. The lazy solution would be to delete \footnote but turn its argument into normal text.

[davidweichiang]

The non-lazy solution would be to add \footnote to the list of LaTeX commands, so that it gets parsed correctly with its argument. If you're sure this is the right way to go, I can try to make the modification. Am I allowed to push to your branch?

Something similar should probably be done for \cite and friends, although there are so many variants of this command that it would clutter up the list of commands...

[mjpost]

Yes, please do.

[davidweichiang]

Because of line 153, the space after \cite is not optional.

[mjpost]

True, I added it for robustness against future changes.

[mjpost]

(Could be removed)

[davidweichiang]

Another reason for handling \cite and \footnote outside of latex_to_unicode is that they aren't related to Unicode.

[davidweichiang]

Hopefully that works!

[mjpost]

I'm merging in #736 since it has more test cases for the latex conversion. So far they have turned up the following problems:

• \href{...}{...}
• The use of parenthetical custom newcommand which when deleted leave empty parens (e.g., in the past (\past) becomes in the past ())

I'll look at handling these later today.

I also wonder if we should convert footnotes to parenthetical notes instead of removing, but I'm not sure that I'm going to do this.

[davidweichiang]

It should be straightforward to delete \href along with its first argument, leaving the second argument.

If authors are putting macros in abstracts, I'm not sure what we can do about it. At some point in the future, we might be able to scrape the abstract straight from the PDF.

[davidweichiang]

The above commit changes \href{http://url.com}{text} to text, and it also turns off changing curly braces to fixed-case inside abstracts.

[mjpost]

@davidweichiang can you run this code and provide the output?

#!/usr/bin/env python3

import latexcodec, codecs
import sys

s = "the ''generative step'' into"
s = codecs.decode(s, "ulatex+utf8")

print(s)

I get the ”generative step” into with versions 1.0.7 and 2.0.0 of latexcodec.

[davidweichiang]

@mjpost I get the same output, which is correct, isn't it?

[mjpost]

Shouldn't it be the “generative step” into? The output has two right curly quotes.

[davidweichiang]

But so does the LaTeX; it's the author's mistake.

[mjpost]

Ah right.