Skip to content

Commit

Permalink
Improve logs to troubleshoot issue in 1.4.0a2 with astro-cli (#947)
Browse files Browse the repository at this point in the history
Improve the logs so we can understand which Dataset URIs Cosmos was
setting, while trying to execute a task in Airflow 2.9:
```
[2024-05-07, 14:20:09 UTC] {taskinstance.py:441} ▼ Post task execution logs [2024-05-07, 14:20:09 UTC] {taskinstance.py:2905} ERROR - Task failed with exception Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 400, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/base.py", line 266, in execute
    self.build_and_run_cmd(context=context, cmd_flags=self.add_cmd_flags())
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 470, in build_and_run_cmd
    result = self.run_command(cmd=dbt_cmd, env=env, context=context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 343, in run_command
    outlets = self.get_datasets("outputs")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 410, in get_datasets
    return [Dataset(uri) for uri in uris]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 410, in <listcomp>
    return [Dataset(uri) for uri in uris]
            ^^^^^^^^^^^^
  File "<attrs generated init airflow.datasets.Dataset>", line 3, in __init__
    _setattr('uri', __attr_converter_uri(uri))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/datasets/__init__.py", line 81, in _sanitize_uri
    parsed = normalizer(parsed)
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/postgres/datasets/postgres.py", line 34, in sanitize_uri
    raise ValueError("URI format postgres:// must contain database, schema, and table names")
```

This improvement allowed us to confirm how the Dataset URIs Cosmos was
attempting to generate, allowing us to log the following issue:
#945

(cherry picked from commit c7a4599)
  • Loading branch information
tatiana authored May 9, 2024
1 parent 9338e91 commit 7ad332a
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions cosmos/operators/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ def get_datasets(self, source: Literal["inputs", "outputs"]) -> list[Dataset]:
for output in getattr(completed, source):
dataset_uri = output.namespace + "/" + output.name
uris.append(dataset_uri)
logger.debug("URIs to be converted to Dataset: %s", uris)
return [Dataset(uri) for uri in uris]

def register_dataset(self, new_inlets: list[Dataset], new_outlets: list[Dataset]) -> None:
Expand Down

0 comments on commit 7ad332a

Please sign in to comment.