Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: extend pipeline.add_component to support stores #5261

Merged
merged 48 commits into from
Jul 17, 2023

Conversation

ZanSara
Copy link
Contributor

@ZanSara ZanSara commented Jul 3, 2023

Related Issues

Proposed Changes:

  • Changes the API of v2 Pipelines with respect to connecting docstores with components (see the issue for usage examples). The old API left all responsibility to the components, while the new API uses the pipeline as an intermediary for the connection.
  • Creates two mixins to help the implementation of components requiring docstore(s): StoreMixin and MultiStoreMixin.
  • Adapt MemoryRetriever to this new connection style
  • Fix all tests and add new ones

How did you test it?

  • local test run
  • ci

Notes for the reviewer

Let's focus on how the API feels like both on the pipeline side and on the component side.

Checklist

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Jul 3, 2023
@coveralls
Copy link
Collaborator

coveralls commented Jul 3, 2023

Pull Request Test Coverage Report for Build 5575173381

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.06%) to 45.542%

Files with Coverage Reduction New Missed Lines %
preview/pipeline.py 2 95.0%
nodes/prompt/invocation_layer/hugging_face.py 9 91.47%
Totals Coverage Status
Change from base Build 5543286756: 0.06%
Covered Lines: 10562
Relevant Lines: 23192

💛 - Coveralls

@ZanSara ZanSara marked this pull request as ready for review July 4, 2023 08:02
@ZanSara ZanSara requested a review from a team as a code owner July 4, 2023 08:02
@ZanSara ZanSara requested review from vblagoje and removed request for a team and vblagoje July 4, 2023 08:02
@ZanSara ZanSara marked this pull request as draft July 5, 2023 09:29
@ZanSara ZanSara mentioned this pull request Jul 12, 2023
"""

_store: Optional[Store] = None
supported_stores: List[Type[Store]] = [Store] # type: ignore # (see https://github.com/python/mypy/issues/4717)
Copy link
Contributor

@silvanocerza silvanocerza Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must not be part of the public interface of a component.

Suggested change
supported_stores: List[Type[Store]] = [Store] # type: ignore # (see https://github.com/python/mypy/issues/4717)
_supported_stores: List[Type[Store]] = [Store] # type: ignore # (see https://github.com/python/mypy/issues/4717)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this to be public API since all components will need to set it explicitly. I'll leave it without the underscore.

Copy link
Contributor

@silvanocerza silvanocerza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments, I very much prefer if we force component creators to be explicit when defining which stores their component supports.

@ZanSara ZanSara merged commit 8f3fe85 into main Jul 17, 2023
54 checks passed
@ZanSara ZanSara deleted the v2-docstores-connections branch July 17, 2023 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change docstore connection in pipelines preview
3 participants