Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary iterator in Language.pipe #5101

Merged
merged 1 commit into from
Mar 8, 2020

Conversation

adrianeboyd
Copy link
Contributor

Description

Remove iterator over raw_texts with iterator.tee() in Language.pipe that is never consumed and consumes memory unnecessarily.

Addresses part of issue in #5083. I think the use of iterator.tee() may still be causing memory problems with multiprocessing, but this should be an improvement for .pipe(n_process=1).

Types of change

Checklist

  • I have submitted the spaCy Contributor Agreement.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

Remove iterator over `raw_texts` with `iterator.tee()` in
`Language.pipe` that is never consumed and consumes memory
unnecessarily.
@svlandeg svlandeg added enhancement Feature requests and improvements feat / pipeline Feature: Processing pipeline and components perf / memory Performance: memory use labels Mar 4, 2020
@honnibal honnibal merged commit 993758c into explosion:master Mar 8, 2020
@svlandeg svlandeg mentioned this pull request Mar 16, 2020
3 tasks
@svlandeg svlandeg added bug Bugs and behaviour differing from documentation and removed enhancement Feature requests and improvements labels Mar 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bugs and behaviour differing from documentation feat / pipeline Feature: Processing pipeline and components perf / memory Performance: memory use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants