Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix _is_single from Phrases for case when corpus is numpy array of tokens #1987

Merged
merged 3 commits into from
Mar 20, 2018
Merged

Fix _is_single from Phrases for case when corpus is numpy array of tokens #1987

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
16e988f
fix _is_single element-wise comparison error
rmalouf Mar 20, 2018
f97910e
fix mode
rmalouf Mar 20, 2018
4854bc5
add test for corpus as numpy array
rmalouf Mar 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gensim/models/phrases.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def _is_single(obj):
if isinstance(peek, string_types):
# It's a document, return the iterator
return True, obj_iter
if temp_iter == obj:
if temp_iter is obj:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add a clear test for the case when this doesn't work correctly without this fix (and now it works).
Also, return mode of file please (755 -> 644).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

# Checking for iterator to the object
return False, obj_iter
else:
Expand Down
18 changes: 17 additions & 1 deletion gensim/test/test_phrases.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@

import six

import numpy as np

from gensim.utils import to_unicode
from gensim.models.phrases import SentenceAnalyzer, Phrases, Phraser
from gensim.models.phrases import pseudocorpus, original_scorer
Expand Down Expand Up @@ -148,7 +150,7 @@ def gen_sentences(self):


class PhrasesCommon:
""" Tests that need to be run for both Prases and Phraser classes."""
""" Tests that need to be run for both Phrases and Phraser classes."""

def setUp(self):
self.bigram = Phrases(
Expand Down Expand Up @@ -230,6 +232,20 @@ def testBigramConstructionFromGenerator(self):
break
self.assertTrue(bigram1_seen and bigram2_seen)

def testBigramConstructionFromArray(self):
"""Test Phrases bigram construction building when corpus is a numpy array"""
bigram1_seen = False
bigram2_seen = False

for s in self.bigram[np.array(self.sentences)]:
if not bigram1_seen and self.bigram1 in s:
bigram1_seen = True
if not bigram2_seen and self.bigram2 in s:
bigram2_seen = True
if bigram1_seen and bigram2_seen:
break
self.assertTrue(bigram1_seen and bigram2_seen)

def testEncoding(self):
"""Test that both utf8 and unicode input work; output must be unicode."""
expected = [u'survey', u'user', u'computer', u'system', u'response_time']
Expand Down