Optimize FastText.load_fasttext_model

[mpenkov]

Should fix #1261

[menshikh-iv]

Thanks @mpenkov, what's still missing

• ngram byte-based func (FB port) in "glued" form I guess (to avoid "".join().split())
• more tests (esp. for previous note)
• final measurements (by time) for loading (before VS after VS FB impl) in 2 variants
  • load model and retrieve vector to word
  • load model and start training
• backward compatibility fix for fastText fixes in 3.7 break compatibility with old models #2341 Fixed by Fix backward compatibility issue: loading FastTextKeyedVectors using KeyedVectors (missing attribute compatible_hash) #2349

[horpto]

stupid question: what if self.trainables does not have syn1neg attr at all, so can model train ?

[mpenkov]

Yes. I don't see any other code that sets syn1neg to None. So, the new code uses that value to mean "cannot continue training".

If trainables does not have syn1neg at all, it is possible to start training.

[mpenkov]

I benchmarked the model loading in this PR against 3.7.0 using:

from gensim.models import FastText
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(filename)s:%(lineno)s - %(message)s')
m = FastText.load_fasttext_format("cc.ru.300.bin")

Before: 13 min
After: 2min 20s

370.prof.gz

improv.prof.gz

[menshikh-iv]

Awesome, thanks @mpenkov and @horpto 👍

[mpenkov]

We're still considerably slower than the FB app:

(improv.venv) mpenkov@hetrad2:~$ cat words.txt 
команда
маленьких
друзей
возит
грузы
всех
быстрей
(improv.venv) mpenkov@hetrad2:~$ cat bench.py 
from gensim.models import FastText
import sys

m = FastText.load_fasttext_format("cc.ru.300.bin", full_model=False)
for line in sys.stdin:
    word = line.rstrip()
    print(word, m.wv[word])
(improv.venv) mpenkov@hetrad2:~$ time cat words.txt | python bench.py > /dev/null

real    2m31.777s
user    1m50.864s
sys     0m41.336s
(improv.venv) mpenkov@hetrad2:~$ time cat words.txt | fasttext print-word-vectors cc.ru.300.bin > /dev/null                                                                                                   

real    0m14.301s
user    0m2.812s
sys     0m11.480s

[piskvorky]

@mpenkov how much of that time is loading vs access? (in gensim and in fb)

EDIT: n/m, I see there are just a few words accessed, so this must be all loading.

[menshikh-iv]

@piskvorky you are right, all of it is loading, retrieve a vector by word works fast.

Note: I guess reason in "retrieve vectors for vocab" (adjust_vector mostly) on loading, FB doesn't do that (construct all vectors on-the-fly, don't precompute vocab), but we do.