Review and Improve Current Documentation for Helm Chart

[kaxil]

Comments from #10752 (comment):

• the values.yaml file is missing comments for most values
• more general explanations are pretty sparse (it seems we currently only have short paragraphs in the README)
• examples of common environments are missing (GKE, EKS, On-Prem)
• provide upgrade guides for major/minor version bumps of the chart

All of this should be initially included in README.md and ideally we should have some sort of documentation for Helm Chart here: https://airflow.apache.org/docs/ (maybe https://airflow.apache.org/docs/helm-chart/stable)

We can also autogenerate some docs based on values.schema.json as mentioned in #11077

[sryabkov]

@kaxil Is there an agreement on how to tackle documentation for the chart? It the plan to manually edit/comment values.yaml, values.schema.json and README.md for now? Use artifacthub.io to host chart documentation, which would be based on values.schema.json and artifacthub annotations, and link to that documentation from README.md? Something else?

[potiuk]

I believe the best plan is to document values.schema.json better and generate nice documentation from it. I think @mik-laj contributed to an external tool (can't remember which) to make a nice documentation generated out of that.

UPDATE: Yeah - @kaxil already wrote about it in the description I see. And I agree with the description that some of the docs that cannot be auto-generated from the schema should be put in README.md

[mik-laj]

@kaxil I. I will create a documentation package in docs/helm-chart so that it is easy to publish that documentation on a website.

[mik-laj]

I think it is worth focusing on the following so that users can set up a minimal production environment:

• Images - What images are used by this Helm Chart and what is their purpose? How to set up secrets to be able to pull images?
• Logging - How to configure logging for tasks? A short section to remind the user that by default this doesn't work because we have a distributed environment.
• Database - How to configure a connection to the SQL database? So that users do not try to use the default configuration as it is intended for testing only. It is worth mention that in addition to PostgresSQL, we also support MySQL. Additional points can be obtained for the description of the configuration of a mutual-SSL connection.
• PG Bouncer -How and Why to use it?
• Network - How to make the service available to the end user? Short description with links to relevant Kubernetes documentation.

I think it's worth describing other things as well, but it's not necessary to publish the Helm Chart:

• How to use Kerberos and how we protect the keytab from user code.
• how to set up elasticsearch logging?
• How to ensure an identity for the environment in major cloud providers eg. on GCP, we will use Workload Identiity.
• StatsD - How and Why to use it?

[kaxil]

Thanks @mik-laj for #14643

and I agree with your comment too around minimal production environment: #14303 (comment)

[kaxil]

Docs here: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html