[MRG] Implement 'Translation Matrix'

[robotcator]

This PR adds translation matrix feature to gensim model.

[gojomo]

Very excited to see this! Note that it may be useful for more than just sets of word-vectors – for example, if you were to Doc2Vec-train a batch of new documents, but include a bunch (100s? 1000s? TBD) of reference documents from a prior larger run, you could presumably translate the new doc-vecs back into the older/larger space using those reference docs to learn the back-mapping.

The comments & documentation need a lot of review/improvement for clarity/completeness. (Variable and method names might benefit from some careful consideration, too.) When it's thought to be ready for merging – passing tests and feature complete – I'll be happy to give it a close top-to-bottom read.

[robotcator]

@gojomo Thanks, happy to have you help. You mean the translation matrix not only restrict to translate the word, but can apply to the document translation? Am I right?

[gojomo]

Not document 'translation' as in from one natural-language to another, but from alternate Doc2Vec runs on different sets of (same-language) documents - but featuring enough overlapping texts to learn the mapping.

[robotcator]

You mean when I were to train on a batch of new documents, and get the new doc vector. And There are also many reference documents, we can use those to learn to back-mapping. But I don't quite understand the reference documents( in same language), can you explain in more detail? Thanks.

[gojomo]

Let's say you train doc-vecs for 10 million documents. Then you get a batch of 500,000 more.

If you train the old model on just the new 500K, the model is improving for those – but drifting away from modelling the older docs. (If some docs/doc-types are not re-presented, in interleaved fashion, the model will lose its 'tuning' for them.)

Alternatively, if you train a new model on just the new 500K, the doc-vectors from this new model will have no essential relationship with the older model. (There's no absolute vector positioning from any training session - the exact same document can wind up in very different places based on the inherent randomness of initialization and training. It's only the relative positions of doc-vectors from the same combined training that is meaningful.)

So, I'm proposing it could be effective to train the new 500K, with some small set of "reference point" documents from the older set mixed-in. (Let's say for now: 10,000 docs.) When the training of the all-new model is done, those 10K ref docs will have points in the "new" space that render them comparable to the 500K new docs. These points will quite probably be very unlike their points in the older space. But, using this TranslationMatrix, those 10K points can be used to learn a mapping back to the older (10M doc) space, and then transform all 500K new docs into the original, older space for comparison to any of the now 10.5M documents.

[robotcator]

Thank for you explaining. I understand what you said now. The "reference documents" from the older set will be mixed into the new training set, those can be seen as pivot to learn the "TranslationMatrix". It is worth trying.

[robotcator]

Hi, @gojomo recently, I have doing the experiment about produce new document vectors via the translation matrix which learned by the back-mapping method. If I train on new 500K document , with 10k of "reference point" documents taken from the 10 million docs. Then, by using TranslationMatrix, we can transform all 500K new docs into the original, older space. So we got 500k more document vector in the original space. But Is there any need measure/evaluate the new docs' vector in older space (perform better)? Or we just use this method to produce more doc vectors rather than train the entire corpus(10.5 million document )again?

[piskvorky]

Hi guys, what's the context here? Can you please add some description of the purpose of this PR?

What are its use-cases, from gensim users' POV?

[robotcator]

@piskvorky This PR is to implement the translation matrix. If needed, I will add more description in this PR.

[gojomo]

@robotcator - Yes, an experiment to evaluate the usefulness of such reference-projected doc-vectors would be good – especially in comparison to inferring-doc-vectors. (Maybe inference works better. Or maybe if there are a lot of new words in the incremental docs, the back-projection works better... it'd certainly be able to bootstrap word-vectors, in the original space, that inference alone would not.) But, I don't have a proposed experiment design ready – my suggestion was conjectural.

[robotcator]

Hi, I use the IMDB sentiment dataset to train two models, one with 15k documents and the other with 25k documents, all other parameters are the same. The code is below(some are from the notebook)

import gensim
from gensim.models.doc2vec import TaggedDocument
from gensim.models import Doc2Vec
from collections import namedtuple
from gensim import utils

def read_sentimentDocs():
    SentimentDocument = namedtuple('SentimentDocument', 'words tags split sentiment')

    alldocs = []  # will hold all docs in original order
    with utils.smart_open('alldata-id.txt', encoding='utf-8') as alldata:
        for line_no, line in enumerate(alldata):
            tokens = gensim.utils.to_unicode(line).split()
            words = tokens[1:]
            tags = [line_no] # `tags = [tokens[0]]` would also work at extra memory cost
            split = ['train','test','extra','extra'][line_no//25000]  # 25k train, 25k test, 25k extra
            sentiment = [1.0, 0.0, 1.0, 0.0, None, None, None, None][line_no//12500] # [12.5K pos, 12.5K neg]*2 then unknown
            alldocs.append(SentimentDocument(words, tags, split, sentiment))

    train_docs = [doc for doc in alldocs if doc.split == 'train']
    test_docs = [doc for doc in alldocs if doc.split == 'test']
    doc_list = alldocs[:]  # for reshuffling per pass

    print('%d docs: %d train-sentiment, %d test-sentiment' % (len(doc_list), len(train_docs), len(test_docs)))

    return train_docs, test_docs, doc_list

train_docs, test_docs, doc_list = read_sentimentDocs()

import multiprocessing
from random import shuffle

cores = multiprocessing.cpu_count()
# PV-DM w/concatenation - window=5 (both sides) approximates paper's 10-word total window size
model = Doc2Vec(dm=1, dm_concat=1, size=100, window=5, negative=5, hs=0, min_count=2, workers=cores)

small_train_docs = train_docs[:15000]

# train for small corpus
model.build_vocab(small_train_docs)
for epoch in xrange(50):
    shuffle(small_train_docs)
    model.train(small_train_docs, total_examples=len(small_train_docs), epochs=1)
# save the model
model.save("small_doc_15000_iter50.bin")


# train for large corpus
model.build_vocab(train_docs)
for epoch in xrange(50):
    shuffle(train_docs)
    model.train(train_docs, total_examples=len(train_docs), epochs=1)
# save the model
model.save("large_doc_25000_iter50.bin")

I use those 15k documents as reference document to mix in the new 15k documents, so I got the model2 with 25k documents. Then using those 15k documents and their corresponding doc vectors in model1 and model2, we can train a translation matrix. The code is below:

# coding=utf-8
from gensim.models import Doc2Vec
from sklearn.linear_model import LogisticRegression
from collections import namedtuple

import numpy as np

model1 = Doc2Vec.load("small_doc_15000_iter50.bin")
model2 = Doc2Vec.load("large_doc_25000_iter50.bin")

l = model1.docvecs.count
l2 = model2.docvecs.count
m1 = np.array([model1.docvecs[i] for i in range(l)])
m2 = np.array([model2.docvecs[i] for i in range(l)])

# train the translation matrix
tm = np.linalg.lstsq(m2, m1, -1)[0]
# back-mapping for the 15k documents
model1_15k_docvec = []
for i in range(l, l2):
    x = model2.docvecs[i]
    y = np.dot(x, tm)
    model1_15k_docvec.append(y)

The problem is how to evaluate the 15k document vector learned by the method, I think we can train a classifier to see whether those vector works. But we need a benchmark for comparison. The code is rough but did I explain the problem clear?@menshikh-iv

[menshikh-iv]

Remove spaces in the filename Translation_Matrix_Revisit.ipynb

[menshikh-iv]

Spaces before/after + (here and anywhere)

[menshikh-iv]

Need more explanation between cells (what happens)

[menshikh-iv]

Please use plotly for all viz

[robotcator]

done

[menshikh-iv]

Also please add a description before each image (what I see now).

[menshikh-iv]

E251 unexpected spaces around keyword / parameter equals (here and anywhere)

[menshikh-iv]

E226 missing whitespace around arithmetic operator (here and anywhere)

[robotcator]

Done

[menshikh-iv]

Why only "sanity check"? Please add another tests (similar with translation tests from previous class)?

[robotcator]

This is a little different with the translation of word. We build the mapping F between the target model and original model(old). Given a document vector x from the target model, we infer the vector by multiple F with x to get a new vector for the original vector. By now, we didn't have a good evaluation metric to assess the vector. So I only give the "sanity check".

[menshikh-iv]

You can calculate cosine distance between vector from model and vector from translation

[menshikh-iv]

Please add example with viz for it (to translation matrix notebook)

[menshikh-iv]

You can calculate cosine distance between vector from model and vector from translation

[menshikh-iv]

Please add this test to translation_matrix.ipynb (as another section) and remove Translation_Matrix_Revisit.ipynb

[menshikh-iv]

In general, LGTM for me (after fixing last comments), have you any suggestions @gojomo @piskvorky?

[gojomo]

Could note that this is potentially more useful than just word-to-word translation; it'll work for any two sets of named-vectors where there are some paired-guideposts to learn the transformation.

[gojomo]

This API that supplies word_pair twice - both at construction & a later necessary train() step – seems odd to me, and in contrast to other gensim practice (where training can be combined with initialization, if training data is supplied.)

[robotcator]

Done.

[gojomo]

Wording like 'list of lexicon' unclear: is index2word a list of strings?

[robotcator]

Yes, index2word is a list of words(strings).

[gojomo]

Capitalization, verb-tense: 'Normalize...'

[gojomo]

As a list of many word-pairs, a more natural name for this parameter would be plural word_pairs.

[robotcator]

fixed

[gojomo]

This word_pair argument, which I noted seemed odd above, doesn't seem to be consulted at all in the following code. (Only the word-pairs supplied at initialization have any effect.)

[robotcator]

As you said in the second comment, if training was combined with initialization, then this parameter would be used in train method.

[gojomo]

Looking at the code body of the train() method, I see no references to this word_pair parameter. Hence, the parameter is unused and (unless the code changes) unnecessary here. If the intent was that this be an alternate way to supply the list-of-word-pairs, the code would have to change to reflect that intent.

[robotcator]

Okay, I change the code where the word_pairs is optional parameter in the initialization, and if provided, then train method will run followed.

[gojomo]

source_space

[robotcator]

Thanks, I will fix all the grammar errors and typos here and below.

[gojomo]

Capitalization, tense: "Map…"

[gojomo]

Every one of these word-pairs is in the training mappings. A 'truer' test of 'translation' functionality would verify that using these known signpost-equivalences creates a mapping that's useful for getting other word-to-word translations approximately right.

[gojomo]

This could be a loop driven by test_word_pair.

[gojomo]

The constant stream of tiny tweak change-notifications, without any deeper review of the docs/naming issues I'd mentioned way-back, made me think this work was still far from review-readiness.

I just now see it was it marked for [MRG] and merged in the last 10 hours.

I think the API, naming, comments, and testing are all still a bit off. I've added some comments, but all of the code-doc-comments should get additional style/clarity proofreading.

[robotcator]

@gojomo Thank you for reviewing, there still need more improvements. And I will fix the problem according to your comments.

[robotcator]

@gojomo Hi, I open a new PR to fix the comment according to you. I change the train method to make it compulsory and in the __init__ method to optional, but If the word_pairs is provided, the train method will also run. And I will make change in the notebook later.