Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow indexing large vectors [ENHANCEMENT] #60

Closed
freakeinstein opened this issue Jan 15, 2020 · 1 comment
Closed

Allow indexing large vectors [ENHANCEMENT] #60

freakeinstein opened this issue Jan 15, 2020 · 1 comment
Assignees
@freakeinstein
Labels
enhancement New feature or request JS JavaScript (JS) is a lightweight interpreted or JIT-compiled programming language(-github). medium priority Python Python is a dynamically typed programming language(-github).

Comments

@freakeinstein
Copy link
Member

Currently, there is a limit (which got introduced as a side effect of bulk retrieval logic) in vector dimension throttled by gRPC request limit. Currently, once the document count hit vecount a bulk retrieval from document database is performed, which in turn blows JS heap memory as well as gRPC data limit.

Proposed fixes:

  • move bulk loading logic to vecstore module
  • enable mini batch reading from disk
@freakeinstein freakeinstein added enhancement New feature or request medium priority JS JavaScript (JS) is a lightweight interpreted or JIT-compiled programming language(-github). Python Python is a dynamically typed programming language(-github). labels Jan 15, 2020
@freakeinstein freakeinstein self-assigned this Jan 15, 2020
@freakeinstein
Copy link
Member Author

Code is rewritten. Bug is irrelevant and covered. closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@freakeinstein freakeinstein

Labels
enhancement New feature or request JS JavaScript (JS) is a lightweight interpreted or JIT-compiled programming language(-github). medium priority Python Python is a dynamically typed programming language(-github).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@freakeinstein