feat: Add Hazelcast as an online store (#3523)

* feat: Add Hazelcast as an online store Signed-off-by: mehmettokgoz <[email protected]> * Fix template bugs and address review comments. Signed-off-by: mehmettokgoz <[email protected]> --------- Signed-off-by: mehmettokgoz <[email protected]>
feast-dev · Mar 25, 2023 · b05d50b · b05d50b
1 parent 8dd7217
commit b05d50b
Show file tree

Hide file tree

Showing 23 changed files with 1,073 additions and 0 deletions.
diff --git a/Makefile b/Makefile
@@ -259,6 +259,27 @@ test-python-universal-cassandra:
 	python -m pytest -x --integration \
 	sdk/python/tests
 
+test-python-universal-hazelcast:
+	PYTHONPATH='.' \
+		FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.hazelcast_repo_configuration \
+		PYTEST_PLUGINS=sdk.python.tests.integration.feature_repos.universal.online_store.hazelcast \
+		FEAST_USAGE=False \
+		IS_TEST=True \
+		python -m pytest -n 8 --integration \
+ 			-k "not test_universal_cli and \
+ 				not test_go_feature_server and \
+ 				not test_feature_logging and \
+				not test_reorder_columns and \
+				not test_logged_features_validation and \
+				not test_lambda_materialization_consistency and \
+				not test_offline_write and \
+				not test_push_features_to_offline_store and \
+				not gcs_registry and \
+				not s3_registry and \
+ 				not test_universal_types and \
+				not test_snowflake" \
+ 			sdk/python/tests
+
 test-python-universal-cassandra-no-cloud-providers:
 	PYTHONPATH='.' \
 	FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.contrib.cassandra_repo_configuration \

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -97,6 +97,7 @@
   * [Cassandra + Astra DB (contrib)](reference/online-stores/cassandra.md) 
   * [MySQL (contrib)](reference/online-stores/mysql.md)
   * [Rockset (contrib)](reference/online-stores/rockset.md)
+  * [Hazelcast (contrib)](reference/online-stores/hazelcast.md)
 * [Providers](reference/providers/README.md)
   * [Local](reference/providers/local.md)
   * [Google Cloud Platform](reference/providers/google-cloud-platform.md)

diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md
@@ -46,3 +46,8 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
 [rockset.md](rockset.md)
 {% endcontent-ref %}
 
+{% content-ref url="hazelcast.md" %}
+[hazelcast.md](hazelcast.md)
+{% endcontent-ref %}
+
+
diff --git a/docs/reference/online-stores/hazelcast.md b/docs/reference/online-stores/hazelcast.md
@@ -0,0 +1,59 @@
+# Hazelcast online store
+
+## Description
+
+Hazelcast online store is in alpha development.
+
+The [Hazelcast](htpps://hazelcast.com) online store provides support for materializing feature values into a Hazelcast cluster for serving online features in real-time.
+In order to use Hazelcast as online store, you need to have a running Hazelcast cluster. You can create a cluster using Hazelcast Viridian Serverless. See this [getting started](https://hazelcast.com/get-started/) page for more details.
+
+* Each feature view is mapped one-to-one to a specific Hazelcast IMap
+* This implementation inherits all strengths of Hazelcast such as high availability, fault-tolerance, and data distribution.
+* Secure TSL/SSL connection is supported by Hazelcast online store.
+* You can set TTL (Time-To-Live) setting for your features in Hazelcast cluster. 
+
+Each feature view corresponds to an IMap in Hazelcast cluster and the entries in that IMap corresponds to features of entities.
+Each feature value stored separately and can be retrieved individually.
+
+## Getting started
+
+In order to use Hazelcast online store, you'll need to run `pip install 'feast[hazelcast]'`. You can then get started with the command `feast init REPO_NAME -t hazelcast`.
+
+
+## Examples
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+  type: hazelcast
+  cluster_name: dev
+  cluster_members: ["localhost:5701"]
+  key_ttl_seconds: 36000
+```
+
+## Functionality Matrix
+
+|                                                           | Hazelcast |
+| :-------------------------------------------------------- |:----------|
+| write feature values to the online store                  | yes       |
+| read feature values from the online store                 | yes       |
+| update infrastructure (e.g. tables) in the online store   | yes       |
+| teardown infrastructure (e.g. tables) in the online store | yes       |
+| generate a plan of infrastructure changes                 | no        |
+| support for on-demand transforms                          | yes       |
+| readable by Python SDK                                    | yes       |
+| readable by Java                                          | no        |
+| readable by Go                                            | no        |
+| support for entityless feature views                      | yes       |
+| support for concurrent writing to the same key            | yes       |
+| support for ttl (time to live) at retrieval               | yes       |
+| support for deleting expired data                         | yes       |
+| collocated by feature view                                | no        |
+| collocated by feature service                             | no        |
+| collocated by entity key                                  | yes       |
+
+To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
+
diff --git a/...python/docs/source/feast.infra.online_stores.contrib.hazelcast_online_store.rst b/...python/docs/source/feast.infra.online_stores.contrib.hazelcast_online_store.rst
@@ -0,0 +1,21 @@
+feast.infra.online\_stores.contrib.hazelcast\_online\_store package
+===================================================================
+
+Submodules
+----------
+
+feast.infra.online\_stores.contrib.hazelcast\_online\_store.hazelcast\_online\_store module
+-------------------------------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.hazelcast_online_store.hazelcast_online_store
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.online_stores.contrib.hazelcast_online_store
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
@@ -8,6 +8,7 @@ Subpackages
    :maxdepth: 4
 
    feast.infra.online_stores.contrib.cassandra_online_store
+   feast.infra.online_stores.contrib.hazelcast_online_store
    feast.infra.online_stores.contrib.hbase_online_store
    feast.infra.online_stores.contrib.mysql_online_store
    feast.infra.online_stores.contrib.rockset_online_store
@@ -23,6 +24,14 @@ feast.infra.online\_stores.contrib.cassandra\_repo\_configuration module
    :undoc-members:
    :show-inheritance:
 
+feast.infra.online\_stores.contrib.hazelcast\_repo\_configuration module
+------------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.hazelcast_repo_configuration
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 feast.infra.online\_stores.contrib.hbase\_repo\_configuration module
 --------------------------------------------------------------------
 

diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py
@@ -641,6 +641,7 @@ def materialize_incremental_command(ctx: click.Context, end_ts: str, views: List
             "hbase",
             "cassandra",
             "rockset",
+            "hazelcast",
         ],
         case_sensitive=False,
     ),

diff --git a/sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/README.md b/sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/README.md
@@ -0,0 +1,89 @@
+# Hazelcast Online Store
+
+This contribution makes it possible to use [Hazelcast](https://hazelcast.com/) as online store for Feast.
+
+Once the Hazelcast client configuration is given inside `feature_store.yaml` file, everything else
+is handled as with any other online store: schema creation, read/write from/to Hazelcast and remove operations.
+
+## Quick usage
+
+The following refers to the [Feast quickstart](https://docs.feast.dev/getting-started/quickstart) page. 
+Only the Step 2 is different from this tutorial since it requires you to configure your Hazelcast online store.
+
+### Creating the feature repository
+
+The easiest way to get started is to use the Feast CLI to initialize a new
+feature store. Once Feast is installed, the command
+
+```
+feast init FEATURE_STORE_NAME -t hazelcast
+```
+
+will interactively help you create the `feature_store.yaml` with the
+required configuration details to access your Hazelcast cluster.
+
+Alternatively, you can run `feast init -t FEATURE_STORE_NAME`, as described
+in the quickstart, and then manually edit the `online_store` section in
+the `feature_store.yaml` file as detailed below.
+
+The following steps (setup of feature definitions, deployment of the store,
+generation of training data, materialization, fetching of online/offline
+features) proceed exactly as in the general Feast quickstart instructions.
+
+#### Hazelcast setup
+
+In order to use [Hazelcast](https://hazelcast.com) as online store, you need to have a running Hazelcast cluster. 
+You can create a cluster using Hazelcast Viridian Serverless easily or deploy one on your local/remote machine. 
+See this [getting started](https://hazelcast.com/get-started/) page for more details.
+
+Hazelcast online store provides capability to connect local/remote or Hazelcast Viridian Serverless cluster.
+Following is an example to connect local cluster named "dev" running on port 5701 with TLS/SSL enabled.
+
+```yaml
+[...]
+online_store:
+    type: hazelcast
+    cluster_name: dev
+    cluster_members: ["localhost:5701"]
+    ssl_cafile_path: /path/to/ca/file
+    ssl_certfile_path: /path/to/cert/file
+    ssl_keyfile_path: /path/to/key/file
+    ssl_password: ${SSL_PASSWORD} # The password will be read form the `SSL_PASSWORD` environment variable.
+    key_ttl_seconds: 86400 # The default is 0 and means infinite.
+```
+
+If you want to connect your Hazelcast Viridian cluster instead of local/remote one, specify your configuration as follows:
+
+```yaml
+[...]
+online_store:
+    type: hazelcast
+    cluster_name: YOUR_CLUSTER_ID
+    discovery_token: YOUR_DISCOVERY_TOKEN
+    ssl_cafile_path: /path/to/ca/file
+    ssl_certfile_path: /path/to/cert/file
+    ssl_keyfile_path: /path/to/key/file
+    ssl_password: ${SSL_PASSWORD} # The password will be read form the `SSL_PASSWORD` environment variable.
+    key_ttl_seconds: 86400 # The default is 0 and means infinite.
+```
+
+#### TTL configuration
+
+TTL is the maximum time in seconds for each feature to stay idle in the map.
+It limits the lifetime of the features relative to the time of the last read or write access performed on them. 
+The features whose idle period exceeds this limit are expired and evicted automatically. 
+A feature is idle if no get or put is called on it. 
+Valid values are integers between 0 and Integer.MAX_VALUE.
+Its default value is 0, which means infinite.
+
+```yaml
+[...]
+online_store:
+    [...]
+    key_ttl_seconds: 86400
+```
+
+### More info
+
+You can learn about Hazelcast more from the [Hazelcast Documentation](https://docs.hazelcast.com/home/).
+
diff --git a/sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/__init__.py b/sdk/python/feast/infra/online_stores/contrib/hazelcast_online_store/__init__.py