-
Notifications
You must be signed in to change notification settings - Fork 159
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Move airflow conf fetch call to setting.py
- Loading branch information
1 parent
214813b
commit 97db047
Showing
8 changed files
with
113 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,21 @@ | ||
import os | ||
import tempfile | ||
from pathlib import Path | ||
|
||
import airflow | ||
from airflow.configuration import conf | ||
|
||
from cosmos.constants import DEFAULT_COSMOS_CACHE_DIR_NAME | ||
from cosmos.constants import DEFAULT_COSMOS_CACHE_DIR_NAME, DEFAULT_OPENLINEAGE_NAMESPACE | ||
|
||
# In MacOS users may want to set the envvar `TMPDIR` if they do not want the value of the temp directory to change | ||
DEFAULT_CACHE_DIR = Path(tempfile.gettempdir(), DEFAULT_COSMOS_CACHE_DIR_NAME) | ||
cache_dir = Path(conf.get("cosmos", "cache_dir", fallback=DEFAULT_CACHE_DIR) or DEFAULT_CACHE_DIR) | ||
enable_cache = conf.get("cosmos", "enable_cache", fallback=True) | ||
propagate_logs = conf.getboolean("cosmos", "propagate_logs", fallback=True) | ||
dbt_docs_dir = conf.get("cosmos", "dbt_docs_dir", fallback=None) | ||
dbt_docs_conn_id = conf.get("cosmos", "dbt_docs_conn_id", fallback=None) | ||
|
||
try: | ||
LINEAGE_NAMESPACE = conf.get("openlineage", "namespace") | ||
except airflow.exceptions.AirflowConfigException: | ||
LINEAGE_NAMESPACE = os.getenv("OPENLINEAGE_NAMESPACE", DEFAULT_OPENLINEAGE_NAMESPACE) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
Cosmos Config | ||
============= | ||
|
||
This page lists all available Airflow configurations that affect ``astronomer-cosmos`` Astronomer Cosmos behavior. They can be set in the ``airflow.cfg file`` or using environment variables. | ||
|
||
.. note:: | ||
For more information, see `Setting Configuration Options <https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html>`_. | ||
|
||
**Sections:** | ||
|
||
- [cosmos] | ||
- [openlineage] | ||
|
||
[cosmos] | ||
~~~~~~~~ | ||
|
||
.. _cache_dir: | ||
|
||
`cache_dir`_: | ||
The directory used for caching Cosmos data. | ||
|
||
- Default: ``{TMPDIR}/cosmos_cache`` (where ``{TMPDIR}`` is the system temporary directory) | ||
- Environment Variable: ``AIRFLOW__COSMOS__CACHE_DIR`` | ||
|
||
.. _enable_cache: | ||
|
||
`enable_cache`_: | ||
Enable or disable caching of Cosmos data. | ||
|
||
- Default: ``True`` | ||
- Environment Variable: ``AIRFLOW__COSMOS__ENABLE_CACHE`` | ||
|
||
.. _propagate_logs: | ||
|
||
`propagate_logs`_: | ||
Whether to propagate logs in the Cosmos module. | ||
|
||
- Default: ``True`` | ||
- Environment Variable: ``AIRFLOW__COSMOS__PROPAGATE_LOGS`` | ||
|
||
.. _dbt_docs_dir: | ||
|
||
`dbt_docs_dir`_: | ||
The directory path for dbt documentation. | ||
|
||
- Default: ``None`` | ||
- Environment Variable: ``AIRFLOW__COSMOS__DBT_DOCS_DIR`` | ||
|
||
.. _dbt_docs_conn_id: | ||
|
||
`dbt_docs_conn_id`_: | ||
The connection ID for dbt documentation. | ||
|
||
- Default: ``None`` | ||
- Environment Variable: ``AIRFLOW__COSMOS__DBT_DOCS_CONN_ID`` | ||
|
||
[openlineage] | ||
~~~~~~~~~~~~~ | ||
|
||
.. _namespace: | ||
|
||
`namespace`_: | ||
The OpenLineage namespace for tracking lineage. | ||
|
||
- Default: If not configured in Airflow configuration, it falls back to the environment variable ``OPENLINEAGE_NAMESPACE``, otherwise it uses ``DEFAULT_OPENLINEAGE_NAMESPACE``. | ||
- Environment Variable: ``AIRFLOW__OPENLINEAGE__NAMESPACE`` | ||
|
||
.. note:: | ||
For more information, see `Openlieage Configuration Options <https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html>`_. | ||
|
||
Environment Variables | ||
~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
.. _LINEAGE_NAMESPACE: | ||
|
||
`LINEAGE_NAMESPACE`_: | ||
The OpenLineage namespace for tracking lineage. | ||
|
||
- Default: If not configured in Airflow configuration, it falls back to the environment variable ``OPENLINEAGE_NAMESPACE``, otherwise it uses ``DEFAULT_OPENLINEAGE_NAMESPACE``. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters