[AIRFLOW-6436] Create & Automate docs on Airflow Configs

[kaxil]

This PR aims to automate the creation of Config docs and add pre-commit hooks that would also serve as tests.

Preview:

• Layout 1: https://near-cherry.surge.sh/configurations-ref.html
• Layout 2: https://flowery-ray.surge.sh/configurations-ref.html (Similar to our CLI doc)

Also, add a structure to the documentation by adding the following sections:

• Description
• Example
• Type
• Default
• Version the config was added in (for new configs) (Example: link)
  

for each config

A user would only need to modify at a single place: config.yml, changes in default_airflow.cfg would be auto-generated using the pre-commit hook.

---

Link to JIRA issue: https://issues.apache.org/jira/browse/AIRFLOW-6436

• Description above provides context of the change
• Commit message starts with [AIRFLOW-NNNN], where AIRFLOW-NNNN = JIRA ID*
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

(*) For document-only changes, no JIRA issue is needed. Commit message starts [AIRFLOW-XXXX].

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[kaxil]

This PR needs a bit of polishing but its overall logic or structure would remain same.

[codecov-io]

Codecov Report

Merging #7015 into master will decrease coverage by 0.42%.
The diff coverage is 4.44%.

@@            Coverage Diff             @@
##           master    #7015      +/-   ##
==========================================
- Coverage   84.85%   84.42%   -0.43%     
==========================================
  Files         679      680       +1     
  Lines       38536    38725     +189     
==========================================
- Hits        32698    32692       -6     
- Misses       5838     6033     +195

Impacted Files	Coverage Δ
airflow/utils/config_yaml_to_cfg.py	0% <0%> (ø)
airflow/configuration.py	91.43% <33.33%> (-1.22%)	⬇️
airflow/kubernetes/volume_mount.py	44.44% <0%> (-55.56%)	⬇️
airflow/kubernetes/volume.py	52.94% <0%> (-47.06%)	⬇️
airflow/kubernetes/pod_launcher.py	45.25% <0%> (-46.72%)	⬇️
airflow/executors/sequential_executor.py	56% <0%> (-32%)	⬇️
airflow/kubernetes/refresh_config.py	50.98% <0%> (-23.53%)	⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py	78.75% <0%> (-20%)	⬇️
airflow/utils/helpers.py	74.5% <0%> (-7.19%)	⬇️
airflow/utils/dag_processing.py	80.95% <0%> (-7.05%)	⬇️
... and 60 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f777f7...ef62156. Read the comment docs.

[potiuk]

Love it. I will take a look latere. Akls @dimberman -> i think we might want to merge your PR #6983 with this one.

[dimberman]

This is fantastic. 10/10 @kaxil!

[kaxil]

Going to follow this up with some polishing up PRs.

So, please feel free to add comments on this PR, I will take care of all the comments and fix them in the follow-up PR

[ryw]

Very nice :)

[mik-laj]

Should we really do this when loading this module? Maybe you can create a function that will only be called in specific cases? If it is an executable file then it should have the following code at the end of the file

if __name__ == '__main__':
    do_some_magic();

Should it also be in the airflow directory? In my opinion, it looks better in the tests or scripts directory.

[kaxil]

Agree, updating it in #7036

[denjas]

Hi!
Latest commit - 9b5dbf9#diff-bbf16e7665ac448883f2ceeb40db35cdR444
in airflow 1.10.8 (1.10.9)
does not allow changing worker concurrency through CLI interface