Add support for Redis to new Feast

[woop]

Currently Feast only supports Redis as part of a Kubernetes deployment (Feast 0.9). We'd like to make it possible to use Redis as part of the new Feast 0.10 architecture, which would allow teams to migrate from Feast 0.9 to Feast 0.10. We also want to ensure that Feast Serving is fully decoupled from Feast Core, making it easier to operate Feast on K8s.

Proposal

1. Decouple Feast Serving from Feast Core. Instead, make it possible to read Feature Views, Feature Tables, and Entities directly from the Protobuf feature registry that the Feast SDK uses.
2. Add support for feature ingestion from the Feast SDK directly to Redis.
3. Update our Helm charts to provide a Spark-free and Core-free Helm installation. Basically only Feast Serving and Redis. This should be behind a flag to not disrupt existing users.
4. Add a Feast k8s provider for reading/writing to Redis from the Feast Python SDK. Reads will happen through Feast Serving. Reads will happen directly from Redis if using the Python SDK. From Go or Java it will go through Feast Serving for the time being.

Further Extension

Push Based Ingestion

As an extension to the above proposal we could also add support for push based ingestion to Redis through Feast Serving (Java), following an approach outlined feast-dev/feast-java-old#6 and #1461

We'd then let the Feast SDK write to Redis through Feast Serving using gRPC. The benefit to this approach is that teams can also use the push API for streaming ingestion.

Spark

We'd be able to move our existing Spark users over to this new architecture by moving a launcher (like the Spark on K8s Feast launcher) into the Feast SDK under a new provider.

[tedhtchang]

@woop How will the Feast Serving read the local registry ?Or I guess for k8s deployment, the Feast Serving will only read Protobuf directly from gcs backed registry.

[woop]

@woop How will the Feast Serving read the local registry? Or I guess for k8s deployment, the Feast Serving will only read Protobuf directly from gcs backed registry.

For K8s you will have to use the object store registry. I mean in theory you could use a volume mount, but we won't test for or support that.

[adchia]

@achals Think we can likely close this?

[achals]

@adchia I think it's best to wait until we cut a release of feast-serving