Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pinecone Document Store - minimal implementation #81

Merged
merged 42 commits into from
Dec 22, 2023
Merged

Pinecone Document Store - minimal implementation #81

merged 42 commits into from
Dec 22, 2023

Conversation

anakin87
Copy link
Member

@anakin87 anakin87 commented Dec 6, 2023 •
edited
Loading

Part of #128

Minimal implementation.

  • filtering and Embedding retriever will be implemented in future PRs
  • (tests are slow and probably improvable)

@anakin87 anakin87 mentioned this pull request Dec 6, 2023
Closed
@anakin87 anakin87 self-assigned this Dec 6, 2023
@anakin87 anakin87 requested a review from masci December 20, 2023 09:03
@anakin87 anakin87 changed the title Add Pinecone Document Store Pinecone Document Store - minimal implementation Dec 20, 2023
@anakin87 anakin87 marked this pull request as ready for review December 20, 2023 09:03
@anakin87 anakin87 requested a review from a team as a code owner December 20, 2023 09:03
@anakin87 anakin87 requested review from julian-risch and removed request for a team and julian-risch December 20, 2023 09:03
masci
masci suggested changes Dec 21, 2023
View reviewed changes
integrations/pinecone/pyproject.toml Outdated Show resolved Hide resolved
integrations/pinecone/src/pinecone_haystack/document_store.py Outdated Show resolved Hide resolved
integrations/pinecone/src/pinecone_haystack/document_store.py Show resolved Hide resolved
integrations/pinecone/src/pinecone_haystack/document_store.py Outdated Show resolved Hide resolved
integrations/pinecone/src/pinecone_haystack/document_store.py Outdated Show resolved Hide resolved
integrations/pinecone/tests/test_count.py Outdated Show resolved Hide resolved
integrations/pinecone/tests/test_document_store.py Outdated Show resolved Hide resolved
integrations/pinecone/tests/test_document_store.py Show resolved Hide resolved
integrations/pinecone/tests/test_write.py Outdated Show resolved Hide resolved
integrations/pinecone/tests/test_write.py Outdated Show resolved Hide resolved
@anakin87 anakin87 requested a review from masci December 22, 2023 14:29
anakin87
anakin87 commented Dec 22, 2023
View reviewed changes
integrations/pinecone/pyproject.toml
Comment on lines +51 to +53
# Pinecone tests are slow (require HTTP requests), so we run them in parallel
# with pytest-xdist (https://pytest-xdist.readthedocs.io/en/stable/distribution.html)
test = "pytest -n auto --maxprocesses=3 {args:tests}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest-xdist works for parallelization.
I'm limiting the processes to 3, otherwise Pinecone fails with "too many requests"
Including the filters, tests last 5-6 minutes.

masci
masci approved these changes Dec 22, 2023
View reviewed changes
Copy link
Contributor

@masci masci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@anakin87 anakin87 merged commit fbdb9a0 into main Dec 22, 2023
5 checks passed
@anakin87 anakin87 deleted the add_pinecone branch December 22, 2023 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@silvanocerza silvanocerza silvanocerza left review comments

@masci masci masci approved these changes

Assignees

@anakin87 anakin87

Labels
topic:CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@anakin87 @masci @silvanocerza @vrunm