Clean up Feast configuration

[woop]

Creating this card to clean up the Feast Core and Feast Serving configuration files. The configuration files have the following problems

• No clear separation of static Spring or JVM configuration from user level configuration
• Duplicate configuration points for the same thing (BigQuery, or Dataflow configuration)
• Lack of documentation of the options that are configurable.
• Lack of documentation on where new configuration should be placed.
• [EDIT] Adding more scope to this refactor. Configuration should also be easy to define and pass to stores. Validation should happen at the storage layer, not in Feast Core or Feast Serving. This change should make it easier to support stores as interfaces, possibly with factories building the stores/retrievers based on configuration.

This issue tracks the refactor, cleanup, and documentation to solve the above.

[mrzzy]

Not sure if this should be a separate issue, but this is definitely a pain point. There is almost no documentation on feast.dev the configuring and using Dataflow runner except that it is faster and more reliable than DirectRunner.

Getting Dataflow to work:

• Had to reverse engineer the config file from core/src/main/java/feast/core/config/JobConfig.java. Also make sure to a set region from dataflow regional endpoints
• Finding out from Google Docs that I needed to service account credentials via GOOGLE_APPLICATION_CREDENTIALS
• Figure from the logs that service account needed Storage Bucket permissions in a addition to Dataflow permissions
• and needed to enable Dataflow API & Cloud Resource Manager API

[woop]

Yea, that is definitely a major pain point. Thanks for pointing it out and being so detailed.

[woop]

Throwing this into the ring as well. The following BQ configuration parameters must be set for BQ retrieval to work.

bigquery-initial-retry-delay-secs: 
bigquery-total-timeout-secs: 21600

[woop]

Two more notes.

• We can generate configuration metadata from @ConfigurationProperties annotated classes (like FeastProperties). This produces a JSON with keys/descriptions/defaults for all feast keys in application.yml. We could easily convert that into markdown and publish it to docs.feast.dev. I
• We should ensure that our JavaDocs are up to date. It's currently not being generated.

[ches]

• We can generate configuration metadata from @ConfigurationProperties annotated classes (like FeastProperties)

The account which someone apparently created to catch accidental mentions like that one is my favorite thing on the Internet today.

Returning to my seat in the peanut gallery now.

[woop]

• We can generate configuration metadata from @ConfigurationProperties annotated classes (like FeastProperties)

The account which someone apparently created to catch accidental mentions like that one is my favorite thing on the Internet today.

Returning to my seat in the peanut gallery now.

Just noticed it, hahahahahahha

[woop]

Keeping this open for the time being until #533 is done.

[woop]

I believe this issue is done enough. Let's open a new one if needed.