set vector of merged entity

[svlandeg]

Closes #5082

Description

When using the merge_entities pipe, the vector of the merged token is now appropriately set to the vector of the original span (i.e. the average of the original tokens). Before it was just a zero vector.

Types of change

enhancement

Checklist

• I have submitted the spaCy Contributor Agreement.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[adrianeboyd]

@svlandeg: do you think this is something the retokenizer should handle more generally?

[svlandeg]

Yea, perhaps? I was doubting where to put it. You think it would be better moved to the _merge function?

[adrianeboyd]

I think it might make more sense to have it near .tensor here:

spaCy/spacy/tokens/_retokenize.pyx

Lines 197 to 202 in 2281c47

	# Resize the doc.tensor, if it's set. Let the last row for each token stand
	# for the merged region. To do this, we create a boolean array indicating
	# whether the row is to be deleted, then use numpy.delete
	if doc.tensor is not None and doc.tensor.size != 0:
	doc.tensor = _resize_tensor(doc.tensor,
	[(m[0].start, m[0].end) for m in merges])

I was a little worried initially that it would be tricky to add new vectors if the table is full, but I see that it gets resized automatically.

One downside if it's not optional is that the size of the vectors could keep growing and growing, e.g., in a situation like #5083? I'm not sure this is a major concern, but I was just looking at all the places that can grow...

[svlandeg]

Ok I moved it. The issue of the growing Vocab probably needs a more generic solution that is out-of-scope for this PR...