-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: extend
pipeline.add_component
to support stores (#5261)
* add protocol and adapt pipeline * change API in pipeline.add_component * adapt pipeline tests * adapt memoryretriever * additional checks * separate protocol and mixin * review feedback & update tests * pylint * Update haystack/preview/document_stores/protocols.py Co-authored-by: Silvano Cerza <[email protected]> * Update haystack/preview/document_stores/memory/document_store.py Co-authored-by: Silvano Cerza <[email protected]> * docstring of Store * adapt memorydocumentstore * fix tests * remove direct inheritance * pylint * Update haystack/preview/document_stores/mixins.py Co-authored-by: Silvano Cerza <[email protected]> * Update test/preview/components/retrievers/test_memory_retriever.py Co-authored-by: Silvano Cerza <[email protected]> * Update test/preview/components/retrievers/test_memory_retriever.py Co-authored-by: Silvano Cerza <[email protected]> * Update test/preview/components/retrievers/test_memory_retriever.py Co-authored-by: Silvano Cerza <[email protected]> * Update test/preview/components/retrievers/test_memory_retriever.py Co-authored-by: Silvano Cerza <[email protected]> * Update test/preview/components/retrievers/test_memory_retriever.py Co-authored-by: Silvano Cerza <[email protected]> * test names * revert suggestion * private self._stores * move asserts out * remove protocols * review feedback * review feedback * fix tests * mypy * review feedback * fix tests & other details * naming * mypy * fix tests * typing * partial review feedback * move .store to input dataclass * Revert "move .store to input dataclass" This reverts commit 53f624b. * disable reusing components with stores * disable sharing components with docstores * Update mixins.py * black * upgrade canals & fix tests --------- Co-authored-by: Silvano Cerza <[email protected]>
- Loading branch information
1 parent
adfabdd
commit 8f3fe85
Showing
8 changed files
with
539 additions
and
124 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
from haystack.preview.document_stores.protocols import Store, DuplicatePolicy | ||
from haystack.preview.document_stores.mixins import StoreAwareMixin | ||
from haystack.preview.document_stores.memory.document_store import MemoryDocumentStore | ||
from haystack.preview.document_stores.errors import StoreError, DuplicateDocumentError, MissingDocumentError |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
from typing import List, Optional, Type | ||
|
||
|
||
from haystack.preview.document_stores.protocols import Store | ||
|
||
|
||
class StoreAwareMixin: | ||
""" | ||
Adds the capability of a component to use a single document store from the `self.store` property. | ||
To use this mixin you must specify which document stores to support by setting a value to `supported_stores`. | ||
To support any document store, set it to `[Store]`. | ||
""" | ||
|
||
_store: Optional[Store] = None | ||
supported_stores: List[Type[Store]] # type: ignore # (see https://github.com/python/mypy/issues/4717) | ||
|
||
@property | ||
def store(self) -> Optional[Store]: | ||
return self._store | ||
|
||
@store.setter | ||
def store(self, store: Store): | ||
if not isinstance(store, Store): | ||
raise ValueError("'store' does not respect the Store Protocol.") | ||
if not any(isinstance(store, type_) for type_ in type(self).supported_stores): | ||
raise ValueError( | ||
f"Store type '{type(store).__name__}' is not compatible with this component. " | ||
f"Compatible store types: {[type_.__name__ for type_ in type(self).supported_stores]}" | ||
) | ||
self._store = store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.