[WIP] Phrases optimizations

[gojomo]

• avoids an unnecessary duplicate giant dictionary in common case of a single add_vocab() call
• adds new Phraser helper class & supporting options on existing methods

Phraser takes a Phrases and does a single (time-consuming) pass to discover all the Phrases that it would want to create, saving those into a much much smaller (and somewhat faster) helper object. This can be saved & used separately.

Needs more testing.

[piskvorky]

Note to self: use the faster detection loop in Python.

@lev good student project: implement the inner loops as C (via Cython) optional extension (optional ala word2vec extension).

[piskvorky]

@gojomo @lev ready to merge? What's missing?

[tmylk]

Created #918 asking for volunteers to create tests

[tmylk]

Merged in #954

[tmylk]

@gojomo Should Phrases be deprecated? The existing warning about its slowness doesn't seem to deter users.

[gojomo]

@tmylk It's necessary to use a Phrases to construct a Phraser. Phraser takes a bunch of extra time to create, but then is slightly faster but much more compact. (The main benefit is memory, not speed, and certainly not speed-to-first-use, which is worse.)

Also, a Phraser essentially locks in the effective min_count and threshold values at the time of its creation – so if you want to try other values, you need to go back to a Phrases with the full frequency data.

So Phrases can't quite be deprecated or hidden... perhaps some renaming/refactoring and updated docs/examples would guide people to the best options for them. There's also the potential for using more memory-efficient approximate counts, to use less memory during initial Phrases analysis. I believe one or two folks have started down that road before, but their PRs never reached mergeable state. If there's someone with the interest/skills to tackle that, they might want to, at the same time, try to rationalize the interfaces/names for better clarity.

[piskvorky]

@tmylk a fast C/Cython implementation of Phrases, ideally multicore, would be a great student project.

The entire "training" is essentially incrementing a counter, no reason it shouldn't be as fast as your input iterator provides.

Plus, phrases (collocations) are not going anywhere, so it's a stable module to invest more time into.