Ent Eval

This repository contains the code for EntEval EntEval: A Holistic Evaluation Benchmark for Entity Representations (EMNLP 2019).

The structure of this repo:

• enteval: the Entity Evaluation framework

Pretrained ELMo baseline model and ELMo hyperlink model can be downloaded from https://drive.google.com/file/d/1hwE2Po_mmypgc3QsrpBRXGr3nyT1WIka/view?usp=sharing

The EntEval evaluation dataset can be downloaded from https://drive.google.com/file/d/1oEWXb7u81JaYQFycVTxow5anhWEG4KFz/view?usp=sharing, please download it and untar it in the EntEval main directory.

Evaluation example code (You will need to set the elmo path)

example/eval_elmo.py

Note that in this evaluation script, we evaluate per layer ELMo and averaged ELMo layers, different from the paper impelementation. In the paper, the weights of each layer are trained as model parameters.

The code is tested under the following environment/versions:

• Python 3.6.2
• PyTorch 1.0.0
• numpy 1.16.0

Some code in this repo is adopted from SentEval.

Pretraining

The pretraining WikiEnt data can be downloaded from https://drive.google.com/drive/folders/1q3csyFdSQNiN6dMK19ahrmHQwijBAY38?usp=sharing v1 is the version we used in the paper, v2 is the updated version

Experiments

Our experiment results are as follows:

	CAP	CERP	EFP	ET	ESR	ER	ED	Average
GloVe	71.9	52.6	67.0	10.3	50.9	40.8	41.2	47.8
BERT Base mix	80.6	65.6	74.8	32.0	28.8	42.2	50.6	53.5
BERT Large mix	79.1	66.9	76.7	32.3	32.6	48.8	54.3	55.8
5.5B ELMo	80.2	61.2	75.8	35.6	60.3	46.8	51.6	58.8
Hyperlinking ELMo baseline	78.0	59.6	71.5	31.3	61.6	46.5	48.5	56.7
Hyperlinking ELMo	76.9	59.9	72.4	32.2	59.7	45.7	49.0	56.5
Hyperlinking ELMo without context	73.5	59.4	71.1	33.2	53.3	44.6	48.9	54.9
Hyperlinking ELMo with entity mention	76.2	60.4	70.9	33.6	49.0	42.9	49.3	54.6

Reference

@inproceedings{mchen-enteval-19,
  author    = {Mingda Chen and Zewei Chu and Yang Chen and Karl Stratos and Kevin Gimpel},
  title     = {EntEval: A Holistic Evaluation Benchmark for Entity Representations},
  booktitle = {Proc. of {EMNLP}},
  year      = {2019}
}