Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix DbtTestOperator openlineage-integration-common issue when test does not have test_metadata #545

Closed
2 tasks
tatiana opened this issue Sep 18, 2023 · 1 comment · Fixed by #558 or #565
Closed
2 tasks

Fix DbtTestOperator openlineage-integration-common issue when test does not have test_metadata #545

tatiana opened this issue Sep 18, 2023 · 1 comment · Fixed by #558 or #565
Labels
bug Something isn't working good first issue Good for newcomers
Milestone
1.1.2

Comments

@tatiana
Copy link
Collaborator

tatiana commented Sep 18, 2023

Issue reported in the Slack #airflow-dbt channel by Javier Hernández Novoa, on the DbtTestOperator:

https://apache-airflow.slack.com/archives/C059CC42E9W/p1695067128727239

[2023-09-18, 19:54:12 UTC] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 360, in execute
    self.build_and_run_cmd(context=context)
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 356, in build_and_run_cmd
    result = self.run_command(cmd=dbt_cmd, env=env, context=context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 230, in run_command
    self.calculate_openlineage_events_completes(env, Path(tmp_project_dir))
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 278, in calculate_openlineage_events_completes
    events = openlineage_processor.parse()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 211, in parse
    events += self.parse_test(context, nodes)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 308, in parse_test
    assertions = self.parse_assertions(context, nodes)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 367, in parse_assertions
    assertion=test_node["test_metadata"]["name"],
              ~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'test_metadata'

This issue is similar to OpenLineage/OpenLineage#2093.

Two actions, that can be split in two tickets as well:
@tatiana tatiana added the bug Something isn't working label Sep 18, 2023
@tatiana tatiana added this to the 1.1.2 milestone Sep 18, 2023
@tatiana tatiana added the good first issue Good for newcomers label Sep 18, 2023
@tatiana tatiana mentioned this issue Sep 27, 2023
Merged
tatiana added a commit that referenced this issue Sep 27, 2023
@tatiana @navedgaras @marco9663
Fix DbtTestLocalOperator when using on_warning_callback (#558)
4c8d6b0 
Since 1.1.0, the `on_warning_callback` functionality no longer works. It
worked on 1.0.5
    
Closes: #549
Closes: #545 (the fix was necessary to make the `on_warning_callback`
test pass)

Co-authored-by: Edgaras Navickas <[email protected]>
Co-authored-by: Marco Yuen <[email protected]>
tatiana added a commit that referenced this issue Sep 27, 2023
@tatiana @navedgaras @marco9663
Fix DbtTestLocalOperator when using on_warning_callback (#558)
a8a28fa 
Since 1.1.0, the `on_warning_callback` functionality no longer works. It
worked on 1.0.5
    
Closes: #549
Closes: #545 (the fix was necessary to make the `on_warning_callback`
test pass)

Co-authored-by: Edgaras Navickas <[email protected]>
Co-authored-by: Marco Yuen <[email protected]>
@tatiana
Copy link
Collaborator Author

tatiana commented Sep 27, 2023

The fix was released as part of astronomer-cosmos 1.1.2:

@javihernovoa javihernovoa mentioned this issue Sep 28, 2023
Closed
2 tasks
@tatiana tatiana mentioned this issue Sep 28, 2023
Merged
tatiana added a commit that referenced this issue Sep 28, 2023
@tatiana
Fix behaviour when openlineage raises KeyError (#565)
7be2a0a 
Make Cosmos more resilient to Openlineage errors.

The Openlineage library raises `KeyError` for some community dbt
projects (e.g.
https://apache-airflow.slack.com/archives/C059CC42E9W/p1695067128727239):
```
[2023-09-18, 19:54:12 UTC] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 360, in execute
    self.build_and_run_cmd(context=context)
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 356, in build_and_run_cmd
    result = self.run_command(cmd=dbt_cmd, env=env, context=context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 230, in run_command
    self.calculate_openlineage_events_completes(env, Path(tmp_project_dir))
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 278, in calculate_openlineage_events_completes
    events = openlineage_processor.parse()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 211, in parse
    events += self.parse_test(context, nodes)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 308, in parse_test
    assertions = self.parse_assertions(context, nodes)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 367, in parse_assertions
    assertion=test_node["test_metadata"]["name"],
              ~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'test_metadata'
```
Closes: #545
Co-authored-by: Javier Hernández Novoa <[email protected]>
tatiana added a commit that referenced this issue Sep 28, 2023
@tatiana @javihernovoa
Fix behaviour when openlineage raises KeyError (#565)
fb36be5 
Make Cosmos more resilient to Openlineage errors.

The Openlineage library raises `KeyError` for some community dbt
projects (e.g.
https://apache-airflow.slack.com/archives/C059CC42E9W/p1695067128727239):
```
[2023-09-18, 19:54:12 UTC] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 360, in execute
    self.build_and_run_cmd(context=context)
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 356, in build_and_run_cmd
    result = self.run_command(cmd=dbt_cmd, env=env, context=context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 230, in run_command
    self.calculate_openlineage_events_completes(env, Path(tmp_project_dir))
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 278, in calculate_openlineage_events_completes
    events = openlineage_processor.parse()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 211, in parse
    events += self.parse_test(context, nodes)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 308, in parse_test
    assertions = self.parse_assertions(context, nodes)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/processor.py", line 367, in parse_assertions
    assertion=test_node["test_metadata"]["name"],
              ~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'test_metadata'
```
Closes: #545

Co-authored-by: Javier Hernández Novoa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Milestone
1.1.2
1 participant
@tatiana