RoBERTa on SuperGLUE's 'Reading Comprehension with Commonsense Reasoning' task

[dirkgr]

ReCoRD is one of the tasks of the SuperGLUE benchmark. The task is to re-trace the steps of Facebook's RoBERTa paper (https://arxiv.org/pdf/1907.11692.pdf) and build an AllenNLP config that reads the ReCoRD data and fine-tunes a model on it. We expect scores in the range of their entry on the SuperGLUE leaderboard.

This is a span prediction task. You can use the existing TransformerQA model and dataset reader, or write your own in the style of TransformerClassificationTT. Or you write an AllenNLP model that's a thin wrapper around the Huggingface SQuAD model. All of these are valid approaches. To tie your components together, we recommend the IMDB experiment config as a starting point.

[dirkgr]

It looks like this can be cast as a QA model, and thus does not need any model code. It will need

1. a dataset reader that produces data in the format that TransformerQA expects
2. a model config that combines the new dataset reader with the TransformerQA model

[gabeorlanski]

I would like to work on this issue.

[dirkgr]

Let us know if you run into any trouble!

[gabeorlanski]

I have a couple of quick questions that arose while implementing the reader:

• Should I implement both a generic and a transformer reader? (squad.py vs transformer_squad.py)

• For the entities and answers spans, I assume I need to extract the token str itself rather than return the indices it is at. Is that correct?

• Since there are multiple questions per paragraph, should I return a single instance per query or a single instance per paragraph?

[dirkgr]

• No, just the equivalent of transformer_squad.py is enough.
• I don't remember exactly, but I know for sure that the QA model expects indices/spans as labels, since it also predicts those. When it generates the output, the model finds the tokens associated with the indices and makes a proper string out of them. It's possible that we compare that final string with the gold string (instead of the gold indices) to get a final score.
• A single instance per question. The paragraphs will be duplicated.

In general, the output of this reader has to follow the guidelines of the transformer_squad reader more or less exactly, so that the model works with it. If you run into a snag where that's just not possible, then we'll have to write a different model, so we can deal with different formats. But I think it should be fine.

[dirkgr]

I updated the description of this task to recommend the new Tango framework.

[gabeorlanski]

@dirkgr I actually finished this some time ago (https://github.com/gabeorlanski/allennlp-readers-development) but forgot to make a pull request. Would it be better to make the PR with what I have (obviously ported to a fork of allennlp_models) first, or should I try to make it Tango compatible?

[dirkgr]

If you want to try doing it with Tango, please do! Tango is still very new, and you'd be the first one outside of AI2 trying it. It might be a little rough, but I would love your feedback, especially because you could bring an outside perspective.

[dirkgr]

If you don't have a ton of time though, I'd also be quite happy with your original version.