Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Updated snowflake template #3130

Merged
merged 1 commit into from
Aug 24, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions sdk/python/feast/templates/snowflake/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
A quick view of what's in this repository:

* `data/` contains raw demo parquet data
* `example_repo.py` contains demo feature definitions
* `driver_repo.py` contains demo feature definitions
* `feature_store.yaml` contains a demo setup configuring where data sources are
* `test_workflow.py` showcases how to run all key Feast commands, including defining, retrieving, and pushing features.

Expand All @@ -14,6 +14,6 @@ You can run the overall workflow with `python test_workflow.py`.
2. Setup CI/CD + dev vs staging vs prod environments to automatically update the registry as you change Feast feature definitions. See [docs](https://docs.feast.dev/how-to-guides/running-feast-in-production#1.-automatically-deploying-changes-to-your-feature-definitions).
3. (optional) Regularly scheduled materialization to power low latency feature retrieval (e.g. via Airflow). See [Batch data ingestion](https://docs.feast.dev/getting-started/concepts/data-ingestion#batch-data-ingestion)
for more details.
4. (optional) Deploy feature server instances with `feast serve` to expose endpoints to retrieve online features.
- See [Python feature server](https://docs.feast.dev/reference/feature-servers/python-feature-server) for details.
- Use cases can also directly call the Feast client to fetch features as per [Feature retrieval](https://docs.feast.dev/getting-started/concepts/feature-retrieval)
4. (optional) Deploy feature server instances with `feast serve` to expose endpoints to retrieve online features.
- See [Python feature server](https://docs.feast.dev/reference/feature-servers/python-feature-server) for details.
- Use cases can also directly call the Feast client to fetch features as per [Feature retrieval](https://docs.feast.dev/getting-started/concepts/feature-retrieval)
5 changes: 5 additions & 0 deletions sdk/python/feast/templates/snowflake/bootstrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@ def bootstrap():
execute_snowflake_statement(
conn, f'DROP TABLE IF EXISTS "{project_name}_feast_driver_hourly_stats"'
)
execute_snowflake_statement(
conn,
f'ALTER WAREHOUSE IF EXISTS "{snowflake_warehouse}" RESUME IF SUSPENDED',
)

write_pandas(
conn,
driver_df,
Expand Down
58 changes: 0 additions & 58 deletions sdk/python/feast/templates/snowflake/driver_repo.py

This file was deleted.

Empty file.
Original file line number Diff line number Diff line change
@@ -1,39 +1,28 @@
project: my_project
# By default, the registry is a file (but can be turned into a more scalable SQL-backed registry)
# On GCP/AWS, minimally you should create a GCS/S3 bucket for a remote file registry
# See https://docs.feast.dev/getting-started/concepts/registry for details
registry: data/registry.db
provider: gcp
registry: registry.db
provider: local
offline_store:
type: snowflake.offline
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
type: snowflake.offline
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
batch_engine:
type: snowflake.engine
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
online_store:
type: snowflake.online
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
# Use GCP Datastore
# See https://docs.feast.dev/reference/online-stores/datastore
#online_store:
# type: datastore
# project_id: my_gcp_project
# namespace: my_datastore_namespace
# Use AWS DynamoDB
# See https://docs.feast.dev/reference/online-stores/dynamodb
#online_store:
# type: dynamodb
# region: %AWS_REGION%
# Use Redis
# See https://docs.feast.dev/reference/online-stores/redis
#online_store:
# type: redis
# connection_string: "localhost:6379"
type: snowflake.online
account: SNOWFLAKE_DEPLOYMENT_URL
user: SNOWFLAKE_USER
password: SNOWFLAKE_PASSWORD
role: SNOWFLAKE_ROLE
warehouse: SNOWFLAKE_WAREHOUSE
database: SNOWFLAKE_DATABASE
entity_key_serialization_version: 2
28 changes: 0 additions & 28 deletions sdk/python/feast/templates/snowflake/feature_store.yaml

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@


def run_demo():
store = FeatureStore(repo_path=".")
store = FeatureStore(repo_path="./feature_repo")
print("\n--- Run feast apply to setup feature store on Snowflake ---")
subprocess.run(["feast", "apply"])
command = "cd feature_repo; feast apply"
subprocess.run(command, shell=True)

print("\n--- Historical features for training ---")
fetch_historical_features_entity_df(store, for_batch_scoring=False)
Expand Down Expand Up @@ -49,7 +50,8 @@ def run_demo():
fetch_online_features(store, use_feature_service=True)

print("\n--- Run feast teardown ---")
subprocess.run(["feast", "teardown"])
command = "cd feature_repo; feast teardown"
subprocess.run(command, shell=True)


def fetch_historical_features_entity_df(store: FeatureStore, for_batch_scoring: bool):
Expand Down