Feast Integration #322
Closed
Feast Integration #322
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
kumare3
reviewed
Jul 7, 2021
| # * - ``get_historical_features()`` | ||
| # - Enrich an entity dataframe with historical feature values for either training or batch scoring. | ||
| @task | ||
| def store_offline(parquet_file: FlyteFile, repo_path: str) -> (str, str): |
There was a problem hiding this comment.
Can we just create it like a pre-existing task plugin? Where given a parquet file the task will automatically upload the data to Feast Offline feature store?
There was a problem hiding this comment.
@kumare3: What do we do about this? When I write a plugin, the input will not just be a parquet file, we'll have to take features, primary key, etc. Similarly, when retrieving the offline features, the user has to give the primary key and datetime values. The same applies to online features as well.
kumare3
reviewed
Jul 7, 2021
| } | ||
| ) | ||
|
|
||
| retrieval_job = fs.get_historical_features( |
There was a problem hiding this comment.
where are these historical features stored?
There was a problem hiding this comment.
For now, locally.
project: feature_engineering
registry: data/registry.db
provider: local
online_store:
path: data/online_store.db
kumare3
reviewed
Jul 7, 2021
| # One key difference between the online store and data source is that only the latest feature values are stored per entity key. No historical values are stored. | ||
| # Our dataset has two such entries with the same ``Hospital Number`` but different time stamps. Only data point with the latest timestamp is picked from the online store. | ||
| @task | ||
| def store_online(repo_path: str) -> str: |
There was a problem hiding this comment.
this too can be a tasktype already right?
FeastOnlineStoreTask
kumare3
reviewed
Jul 7, 2021
| @task | ||
| def store_online(repo_path: str) -> str: | ||
| store = FeatureStore(repo_path=repo_path) | ||
| store.materialize( |
There was a problem hiding this comment.
why is the time hard coded?
There was a problem hiding this comment.
We can take inputs from the user. But then, it can be days, hours, minutes, ... etc. We'll have to ask the user to give four inputs: two specifying the start and end time format, and the other two specifying their respective values.
Signed-off-by: Samhita Alla <[email protected]>
Signed-off-by: Samhita Alla <[email protected]>
eapolinario
added a commit
that referenced
this pull request
Sep 16, 2021
Signed-off-by: Eduardo Apolinario <[email protected]>
kumare3
pushed a commit
that referenced
this pull request
Sep 20, 2021
* Initial version Signed-off-by: Eduardo Apolinario <[email protected]> * Add venv to dockerfile Signed-off-by: Eduardo Apolinario <[email protected]> * Rename feast integration dir Signed-off-by: Eduardo Apolinario <[email protected]> * Configure minio in the image Signed-off-by: Eduardo Apolinario <[email protected]> * Refactoring + retrieve offline features Signed-off-by: Eduardo Apolinario <[email protected]> * Remove all_together Signed-off-by: Eduardo Apolinario <[email protected]> * Attempt to add s3 credentials to image Signed-off-by: Eduardo Apolinario <[email protected]> * Fix s3 endpoint Signed-off-by: Eduardo Apolinario <[email protected]> * custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Transform FeatureView prior to executing queries Signed-off-by: Eduardo Apolinario <[email protected]> * Set PYTHONPATH Signed-off-by: Eduardo Apolinario <[email protected]> * Set PYTHONPATH to multiple values Signed-off-by: Eduardo Apolinario <[email protected]> * Remove "custom_provider" from path Signed-off-by: Eduardo Apolinario <[email protected]> * Replace minio endpoint Signed-off-by: Eduardo Apolinario <[email protected]> * Print env vars Signed-off-by: Eduardo Apolinario <[email protected]> * Set FEAST_S3_ENDPOINT_URL while building feature store Signed-off-by: Eduardo Apolinario <[email protected]> * Remove minio credentials from image Signed-off-by: Eduardo Apolinario <[email protected]> * Add aws env vars Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mention to local provider Signed-off-by: Eduardo Apolinario <[email protected]> * Remove piping of registry object Signed-off-by: Eduardo Apolinario <[email protected]> * Create random path via FlyteContext Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Remove piping of registry object" This reverts commit ccdf326. Signed-off-by: Eduardo Apolinario <[email protected]> * Clean up feature description and remove debugging statements Signed-off-by: Eduardo Apolinario <[email protected]> * Add tasks up to `train_model` Signed-off-by: Eduardo Apolinario <[email protected]> * Rename workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Comment use of custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Rename workflow Signed-off-by: Eduardo Apolinario <[email protected]> * fix error in training Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> * Add TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Import feature_eng tasks directly Signed-off-by: Eduardo Apolinario <[email protected]> * Add store_online task Signed-off-by: Eduardo Apolinario <[email protected]> * Copy remote file to a local file and replace batch_source in materialize Signed-off-by: Eduardo Apolinario <[email protected]> * Add some debugging statements and fix local execution parameter Signed-off-by: Eduardo Apolinario <[email protected]> * Add remaining steps to workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements and put replacement of remote files back in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Add more logging Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements again Signed-off-by: Eduardo Apolinario <[email protected]> * Add workflow return type Signed-off-by: Eduardo Apolinario <[email protected]> * Include a directory prefix in the model filename Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused overrides in custom provider and comment use of localize_feature_view Signed-off-by: Eduardo Apolinario <[email protected]> * Add type transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Pipe _Feature_Store to all interactions with feast Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unnecessary override in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Rearrange initialization of FeatureStore for better legibility Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Remove unnecessary override in custom provider" This reverts commit 2808ba0. Signed-off-by: Eduardo Apolinario <[email protected]> * Use create_node to enforce order Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused function Signed-off-by: Eduardo Apolinario <[email protected]> * Guard env vars behind a check Signed-off-by: Eduardo Apolinario <[email protected]> * Expose inputs to workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Task to build FeatureStore Signed-off-by: Eduardo Apolinario <[email protected]> * Do not guard env vars behind a check Signed-off-by: Eduardo Apolinario <[email protected]> * Experiment with converted_df Signed-off-by: Eduardo Apolinario <[email protected]> * Comments Signed-off-by: Eduardo Apolinario <[email protected]> * Remove commented code from type transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused portion of sandbox.config Signed-off-by: Eduardo Apolinario <[email protected]> * Remove TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Remove registry parameter from local execution Signed-off-by: Eduardo Apolinario <[email protected]> * No need for type transformers Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mentions to type transformers Signed-off-by: Eduardo Apolinario <[email protected]> * Copy README.rst from #322 Signed-off-by: Eduardo Apolinario <[email protected]> * Step 3 of guide on adding a new integration Signed-off-by: Eduardo Apolinario <[email protected]> * Remove extraneous print statement and turn comments into docstrings in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Comments on README.rst Signed-off-by: Eduardo Apolinario <[email protected]> * Fix link to feast Signed-off-by: Eduardo Apolinario <[email protected]> * Fix serialization of feast_integration dir Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Samhita Alla <[email protected]>
samhita-alla
commented
Oct 14, 2021
Comment on lines
+72
to
+94
| @reference_task( | ||
| project="flytesnacks", | ||
| domain="development", | ||
| name="feast_integration.feature_eng_tasks.mean_median_imputer", | ||
| version="v1", | ||
| ) | ||
| def mean_median_imputer( | ||
| dataframe: pd.DataFrame, | ||
| imputation_method: str, | ||
| ) -> FlyteSchema: | ||
| ... | ||
|
|
||
|
|
||
| @reference_task( | ||
| project="flytesnacks", | ||
| domain="development", | ||
| name="feast_integration.feature_eng_tasks.univariate_selection", | ||
| version="v1", | ||
| ) | ||
| def univariate_selection( | ||
| dataframe: pd.DataFrame, num_features: int, data_class: str, feature_view_name: str | ||
| ) -> pd.DataFrame: | ||
| ... |
pingsutw
pushed a commit
to pingsutw/flyte-monorepo
that referenced
this pull request
Apr 4, 2023
* Initial version Signed-off-by: Eduardo Apolinario <[email protected]> * Add venv to dockerfile Signed-off-by: Eduardo Apolinario <[email protected]> * Rename feast integration dir Signed-off-by: Eduardo Apolinario <[email protected]> * Configure minio in the image Signed-off-by: Eduardo Apolinario <[email protected]> * Refactoring + retrieve offline features Signed-off-by: Eduardo Apolinario <[email protected]> * Remove all_together Signed-off-by: Eduardo Apolinario <[email protected]> * Attempt to add s3 credentials to image Signed-off-by: Eduardo Apolinario <[email protected]> * Fix s3 endpoint Signed-off-by: Eduardo Apolinario <[email protected]> * custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Transform FeatureView prior to executing queries Signed-off-by: Eduardo Apolinario <[email protected]> * Set PYTHONPATH Signed-off-by: Eduardo Apolinario <[email protected]> * Set PYTHONPATH to multiple values Signed-off-by: Eduardo Apolinario <[email protected]> * Remove "custom_provider" from path Signed-off-by: Eduardo Apolinario <[email protected]> * Replace minio endpoint Signed-off-by: Eduardo Apolinario <[email protected]> * Print env vars Signed-off-by: Eduardo Apolinario <[email protected]> * Set FEAST_S3_ENDPOINT_URL while building feature store Signed-off-by: Eduardo Apolinario <[email protected]> * Remove minio credentials from image Signed-off-by: Eduardo Apolinario <[email protected]> * Add aws env vars Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mention to local provider Signed-off-by: Eduardo Apolinario <[email protected]> * Remove piping of registry object Signed-off-by: Eduardo Apolinario <[email protected]> * Create random path via FlyteContext Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Remove piping of registry object" This reverts commit ccdf3264bb5b428eb2b474d8422a21a5bb82b0b5. Signed-off-by: Eduardo Apolinario <[email protected]> * Clean up feature description and remove debugging statements Signed-off-by: Eduardo Apolinario <[email protected]> * Add tasks up to `train_model` Signed-off-by: Eduardo Apolinario <[email protected]> * Rename workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Comment use of custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Rename workflow Signed-off-by: Eduardo Apolinario <[email protected]> * fix error in training Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> * Add TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Import feature_eng tasks directly Signed-off-by: Eduardo Apolinario <[email protected]> * Add store_online task Signed-off-by: Eduardo Apolinario <[email protected]> * Copy remote file to a local file and replace batch_source in materialize Signed-off-by: Eduardo Apolinario <[email protected]> * Add some debugging statements and fix local execution parameter Signed-off-by: Eduardo Apolinario <[email protected]> * Add remaining steps to workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements and put replacement of remote files back in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Add more logging Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements again Signed-off-by: Eduardo Apolinario <[email protected]> * Add workflow return type Signed-off-by: Eduardo Apolinario <[email protected]> * Include a directory prefix in the model filename Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused overrides in custom provider and comment use of localize_feature_view Signed-off-by: Eduardo Apolinario <[email protected]> * Add type transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Pipe _Feature_Store to all interactions with feast Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unnecessary override in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Rearrange initialization of FeatureStore for better legibility Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Remove unnecessary override in custom provider" This reverts commit 2808ba07b2ba73f77d2f6c1a08aba0c2cdccea97. Signed-off-by: Eduardo Apolinario <[email protected]> * Use create_node to enforce order Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused function Signed-off-by: Eduardo Apolinario <[email protected]> * Guard env vars behind a check Signed-off-by: Eduardo Apolinario <[email protected]> * Expose inputs to workflow Signed-off-by: Eduardo Apolinario <[email protected]> * Task to build FeatureStore Signed-off-by: Eduardo Apolinario <[email protected]> * Do not guard env vars behind a check Signed-off-by: Eduardo Apolinario <[email protected]> * Experiment with converted_df Signed-off-by: Eduardo Apolinario <[email protected]> * Comments Signed-off-by: Eduardo Apolinario <[email protected]> * Remove commented code from type transformer Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused portion of sandbox.config Signed-off-by: Eduardo Apolinario <[email protected]> * Remove TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Remove registry parameter from local execution Signed-off-by: Eduardo Apolinario <[email protected]> * No need for type transformers Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mentions to type transformers Signed-off-by: Eduardo Apolinario <[email protected]> * Copy README.rst from flyteorg/flytesnacks#322 Signed-off-by: Eduardo Apolinario <[email protected]> * Step 3 of guide on adding a new integration Signed-off-by: Eduardo Apolinario <[email protected]> * Remove extraneous print statement and turn comments into docstrings in custom provider Signed-off-by: Eduardo Apolinario <[email protected]> * Comments on README.rst Signed-off-by: Eduardo Apolinario <[email protected]> * Fix link to feast Signed-off-by: Eduardo Apolinario <[email protected]> * Fix serialization of feast_integration dir Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Samhita Alla <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Samhita Alla [email protected]
Feast (Feature Store) is an operational data system for managing and serving machine learning features to models in production.
Integration between Flyte and Feast can help users take their models and features from prototyping all the way to production cost-effectively and efficiently.