Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added EnsembleLda for stable LDA topics #2980

Merged
merged 211 commits into from
Jul 22, 2021
Merged

Added EnsembleLda for stable LDA topics #2980

merged 211 commits into from
Jul 22, 2021

Conversation

sezanzeb
Copy link
Contributor

@sezanzeb sezanzeb commented Oct 14, 2020 •
edited by piskvorky
Loading

Reopened #2282 because I lost access to the repo there.

I'll merge the up to date develop now and I'm not sure if the required documentation was provided, I'll check that (#2673).

sezanzeb and others added 30 commits December 1, 2018 19:27
added EnsembleLda
7b73db9
Merge branch 'develop' of https://github.com/RaRe-Technologies/gensim …
241c17e 
…into EnsembleLda
Merge branch 'master' of https://github.com/rare-technologies/gensim …
51945e4 
…into EnsembleLda
improvements to add_model, various small changes to comments and code
a67d5db
pandas -> numpy: group by label and mean
e27be0a
pandas -> numpy: generate_stable_topics
83de2dd
pandas -> numpy: distance matrix creation
2af1658
pandas -> numpy: CBDBSCAN
100bbf0
fixes for automated checks
aff3287
improvements on logs, comments and variable naming. Changed save func…
a545ddf 
…tion to simply pickling the whole thing.
minor fix in log message format
d1a6854
added tests
3650895
fixed test
00a06e9
removed some dead leftover pandas code from test
f5f1c9c
removed pathlib from test
c32ddad
tests work in python2 locally now
dab067f
Merge branch 'master' of https://github.com/rare-technologies/gensim …
dcc77ef 
…into EnsembleLda
updated ensemble test reference model
eb9ea27
passing tox8
6b0dc77
improved determinism of methods
6dc6001
improved order of assertions
3ec31e7
trying to achieve higher precision with float64 to avoid some sorting…
7afd192 
… differences across architectures
better approach for comparing with pretrained model
16d0357
potentially fixing the tests on windows
01b68e4
potentially fixing the tests on windows
9314cb4
Merge branch 'develop' of https://github.com/RaRe-Technologies/gensim …
b282393 
…into EnsembleLda
changed citation of opinosis
0b7febc
tox8 test passing after small change on opinosis comments/citation
60a717d
@aloosley
Moving max_random_state inside the model as a private variable.
2ff60ca
@aloosley
removed whitespace
d36fe43
@sezanzeb
Copy link
Contributor Author

sezanzeb commented Jul 3, 2021 •
edited
Loading

Didn't work out on Friday and also probably not today, but we will be on it shortly again and finish up what is still open now

@sezanzeb
Copy link
Contributor Author

sezanzeb commented Jul 4, 2021

Tests should pass now.

We definitely think that dataclasses and typing would create a much cleaner codebase even though they were not directly asked for in the review, but doing it correctly would result in somewhat large changes. Therefore we decided to leave out such changes for a future PR.

We are looking forward to contributing this to version 4.1 and are waiting for your feedback now.

@sezanzeb
Copy link
Contributor Author

sezanzeb commented Jul 5, 2021 •
edited
Loading

@aloosley added two simple dataclasses Topic and Cluster now, it is already a lot better and wasn't much work after all. I think this is allright

tests are failing because

Reading package lists...
E: Failed to fetch https://dl.bintray.com/sbt/debian/InRelease  403  Forbidden [IP: 52.88.131.165 443]
E: The repository 'https://dl.bintray.com/sbt/debian  InRelease' is not signed.

whatever https://dl.bintray.com/sbt/debian is, it is not accessible. Maybe this is a temporary issue

@mpenkov
Copy link
Collaborator

mpenkov commented Jul 17, 2021

Note to self: need to work around bintray being sunset.

https://status.bintray.com/

@mpenkov
update sbt install step
c35fb01
@mpenkov
minor refactoring
71b33dd 
Decoupled multiprocessing code from EnsembleLda class.  This reduces the
length of the class by several hundred lines, making it slightly easier
to understand.

Added _generate_topic_models_worker function to clarify distinction
between single-process and multi-process code.

Fixed flake8 problem (l is an ambiguous variable name)

Adjusted _teardown function (removed i parameter, it's only for logs)

Moved _MAX_RANDOM_STATE to module level
@mpenkov
roll back change to docs/src/Makefile
444c190
@mpenkov
Copy link
Collaborator

mpenkov commented Jul 18, 2021

@aloosley @sezanzeb Finally got a chance to sit down and have a final look at this.

Everything looks good, with the exception that the EnsembleLda class was still a bit too busy: on top of the actual model functionality, it's dealing with ton of multiprocessing stuff. This makes it difficult to understand what's going on. I mentioned this earlier but it may have been lost in the rest of the comments.

Anyway, to help things move along, I made the changes myself. I hope you guys don't mind. All the EnsembleLda tests pass locally (they're still running in CI as I write this) so I don't think I broke anything, but just in case I did, can you please have a glance at the changes here: 71b33dd?

@mpenkov
re-raise caught exception instead of raising a new one
f00aca8 
we don't want to hide the details of the problem
@aloosley
Copy link
Contributor

@aloosley @sezanzeb Finally got a chance to sit down and have a final look at this.

Everything looks good, with the exception that the EnsembleLda class was still a bit too busy: on top of the actual model functionality, it's dealing with ton of multiprocessing stuff. This makes it difficult to understand what's going on. I mentioned this earlier but it may have been lost in the rest of the comments.

Anyway, to help things move along, I made the changes myself. I hope you guys don't mind. All the EnsembleLda tests pass locally (they're still running in CI as I write this) so I don't think I broke anything, but just in case I did, can you please have a glance at the changes here: 71b33dd?

Taking a look at the commit, 90% is the excavation of logic from the EnsembleLda class into module level functions. This all looks fine to me and tests pass for me locally as well. @sezanzeb, any objections?

@mpenkov mpenkov merged commit 76579b3 into piskvorky:develop Jul 22, 2021
@mpenkov
Copy link
Collaborator

mpenkov commented Jul 22, 2021

Alright, finally merged this. @sezanzeb @aloosley Thank you for this awesome contribution and your patience!

@sezanzeb
Copy link
Contributor Author

sezanzeb commented Jul 22, 2021 •
edited
Loading

Thank you so much @mpenkov and @piskvorky for your continuous interest in this! I hope the community will find good use for it :)

@piskvorky piskvorky changed the title EnsembleLda Added EnsembleLda for stable LDA topics Jul 22, 2021
@aloosley
Copy link
Contributor

aloosley commented Aug 1, 2021

Alright, finally merged this. @sezanzeb @aloosley Thank you for this awesome contribution and your patience!

Many thanks @mpenkov and @piskvorky for believing in this stable topic modeling idea (turn @sezanzeb's thesis project) and working with us to get it out there for the world to use more easily.

@sezanzeb sezanzeb mentioned this pull request Oct 2, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aloosley aloosley aloosley left review comments

@mpenkov mpenkov mpenkov requested changes

@piskvorky piskvorky piskvorky requested changes

Assignees
No one assigned
Labels
interesting PR ⭐ Interesting PR topic, but not ready (need much work to finish)
Projects
None yet
Milestone
4.2.0
Development

Successfully merging this pull request may close these issues.
4 participants
@sezanzeb @piskvorky @aloosley @mpenkov