feat: Add mysql as online store (feast-dev#3190)

* update remove debug Signed-off-by: hao-affirm <[email protected]> * add interface Signed-off-by: hao-affirm <[email protected]> * fix lint Signed-off-by: hao-affirm <[email protected]> * fix lint Signed-off-by: hao-affirm <[email protected]> * update sql Signed-off-by: hao-affirm <[email protected]> * update ci reqs Signed-off-by: hao-affirm <[email protected]> * remove pip index Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * fix unit test issue Signed-off-by: hao-affirm <[email protected]> * fix lint issue Signed-off-by: hao-affirm <[email protected]> * add entity_key_serialization_version Signed-off-by: hao-affirm <[email protected]> * update doc Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> * fix makefile typo Signed-off-by: hao-affirm <[email protected]> * move to func Signed-off-by: hao-affirm <[email protected]> * format Signed-off-by: hao-affirm <[email protected]> Signed-off-by: hao-affirm <[email protected]>
franciscojavierarceo · Sep 22, 2022 · cb8db84 · cb8db84
1 parent a2dc0d0
commit cb8db84
Show file tree

Hide file tree

Showing 16 changed files with 473 additions and 0 deletions.
diff --git a/Makefile b/Makefile
@@ -231,6 +231,27 @@ test-python-universal-postgres-online:
 				not test_snowflake" \
  			sdk/python/tests
 
+ test-python-universal-mysql-online:
+	PYTHONPATH='.' \
+		FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.mysql_repo_configuration \
+		PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.mysql \
+		FEAST_USAGE=False \
+		IS_TEST=True \
+		python -m pytest -n 8 --integration \
+ 			-k "not test_universal_cli and \
+ 				not test_go_feature_server and \
+ 				not test_feature_logging and \
+				not test_reorder_columns and \
+				not test_logged_features_validation and \
+				not test_lambda_materialization_consistency and \
+				not test_offline_write and \
+				not test_push_features_to_offline_store and \
+				not gcs_registry and \
+				not s3_registry and \
+ 				not test_universal_types and \
+				not test_snowflake" \
+ 			sdk/python/tests
+
 test-python-universal-cassandra:
 	PYTHONPATH='.' \
 	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -94,6 +94,7 @@
   * [DynamoDB](reference/online-stores/dynamodb.md)
   * [PostgreSQL (contrib)](reference/online-stores/postgres.md)
   * [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md) 
+  * [MySQL (contrib)](reference/online-stores/mysql.md)
 * [Providers](reference/providers/README.md)
   * [Local](reference/providers/local.md)
   * [Google Cloud Platform](reference/providers/google-cloud-platform.md)

diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md
@@ -34,3 +34,7 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
 [cassandra.md](cassandra.md)
 {% endcontent-ref %}
 
+{% content-ref url="mysql.md" %}
+[mysql.md](mysql.md)
+{% endcontent-ref %}
+
diff --git a/docs/reference/online-stores/mysql.md b/docs/reference/online-stores/mysql.md
@@ -0,0 +1,53 @@
+# MySQL online store (contrib)
+
+## Description
+
+The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.
+
+* Only the latest feature values are persisted
+
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: mysql
+    host: DB_HOST
+    port: DB_PORT
+    database: DB_NAME
+    user: DB_USERNAME
+    password: DB_PASSWORD
+```
+{% endcode %}
+
+The full set of configuration options is available in [MySQLOnlineStoreConfig](https://rtd.feast.dev/en/master/#feast.infra.online_stores.contrib.mysql.MySQLOnlineStoreConfig).
+
+## Functionality Matrix
+
+The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
+Below is a matrix indicating which functionality is supported by the Mys online store.
+
+| | Mys |
+| :-------------------------------------------------------- | :-- |
+| write feature values to the online store                  | yes |
+| read feature values from the online store                 | yes |
+| update infrastructure (e.g. tables) in the online store   | yes |
+| teardown infrastructure (e.g. tables) in the online store | yes |
+| generate a plan of infrastructure changes                 | no  |
+| support for on-demand transforms                          | yes |
+| readable by Python SDK                                    | yes |
+| readable by Java                                          | no  |
+| readable by Go                                            | no  |
+| support for entityless feature views                      | yes |
+| support for concurrent writing to the same key            | no  |
+| support for ttl (time to live) at retrieval               | no  |
+| support for deleting expired data                         | no  |
+| collocated by feature view                                | yes |
+| collocated by feature service                             | no  |
+| collocated by entity key                                  | no  |
+
+To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.mysql_online_store.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.mysql_online_store.rst
@@ -0,0 +1,21 @@
+feast.infra.online\_stores.contrib.mysql\_online\_store package
+===============================================================
+
+Submodules
+----------
+
+feast.infra.online\_stores.contrib.mysql\_online\_store.mysql module
+--------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.mysql_online_store.mysql
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.online_stores.contrib.mysql_online_store
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
@@ -9,6 +9,7 @@ Subpackages
 
    feast.infra.online_stores.contrib.cassandra_online_store
    feast.infra.online_stores.contrib.hbase_online_store
+   feast.infra.online_stores.contrib.mysql_online_store
 
 Submodules
 ----------
@@ -29,6 +30,14 @@ feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.online\_stores.contrib.mysql\_repo\_configuration module
+--------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.mysql_repo_configuration
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 feast.infra.online\_stores.contrib.postgres module
 --------------------------------------------------
 

diff --git a/sdk/python/feast/infra/online_stores/contrib/mysql_online_store/README.md b/sdk/python/feast/infra/online_stores/contrib/mysql_online_store/README.md
@@ -0,0 +1,81 @@
+# Mysql Online Store
+Mysql is not included in current [Feast](https://github.com/feast-dev/feast) roadmap, this project intends to add Mysql support for Online Store.
+We create a table <project_name>_<feature_view_name> which gets updated with data on every materialize call
+
+
+#### Create a feature repository
+
+```shell
+feast init feature_repo
+cd feature_repo
+```
+
+#### Edit `feature_store.yaml`
+
+set `online_store` type to be `mysql`
+
+```yaml
+project: feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+    type: mysql
+    host: 127.0.0.1       # mysql endpoint, default to 127.0.0.1
+    port: 3306            # mysql port, default to 3306
+    user: test            # mysql user, default to test
+    password: test        # mysql password, default to test
+    database: feast       # mysql database, default to feast
+```
+
+#### Apply the feature definitions in `example.py`
+
+```shell
+feast -c feature_repo apply
+```
+##### Output
+```
+Registered entity driver_id
+Registered feature view driver_hourly_stats_view
+Deploying infrastructure for driver_hourly_stats_view
+```
+
+### Materialize Latest Data to Online Feature Store (Mysql)
+```
+$ CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S") 
+$ feast -c feature_repo materialize-incremental $CURRENT_TIME
+```
+#### Output
+```
+Materializing 1 feature views from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30 into the mysql online store.
+
+driver_hourly_stats_view from 2022-04-16 15:30:39+05:30 to 2022-04-19 15:31:04+05:30:
+100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 120.59it/s]
+```
+
+### Fetch the latest features for some entity id
+```python
+from pprint import pprint
+from feast import FeatureStore
+
+store = FeatureStore(repo_path=".")
+feature_vector = store.get_online_features(
+    features=[
+        "driver_hourly_stats:conv_rate",
+        "driver_hourly_stats:acc_rate",
+        "driver_hourly_stats:avg_daily_trips",
+    ],
+    entity_rows=[
+        {"driver_id": 1004},
+        {"driver_id": 1005},
+    ],
+).to_dict()
+pprint(feature_vector)
+
+```
+#### Output
+```
+{'acc_rate': [0.01390857808291912, 0.4063614010810852],
+ 'avg_daily_trips': [69, 706],
+ 'conv_rate': [0.6624961495399475, 0.7595928311347961],
+ 'driver_id': [1004, 1005]}
+```
diff --git a/sdk/python/feast/infra/online_stores/contrib/mysql_online_store/__init__.py b/sdk/python/feast/infra/online_stores/contrib/mysql_online_store/__init__.py