Skip to content

Commit

Permalink
Fix CI issue running integration tests (#640)
Browse files Browse the repository at this point in the history
At the moment, all PRs are failing due to issues with integration tests
not being able to run. An example of error message:
```
FAILED tests/test_example_dags.py::test_example_dag[extract_dag] - assert not {'/home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py': 'Traceback (most recent call last):\n  File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/selectors.py", line 416, in select\n    fd_event_list = self._selector.poll(timeout)\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.9-2.4/lib/python3.9/site-packages/airflow/utils/timeout.py", line 69, in handle_timeout\n    raise AirflowTaskTimeout(self.error_message)\nairflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /home/runner/work/astronomer-cosmos/astronomer-cosmos/dev/dags/example_cosmos_python_models.py after 30.0s.\nPlease take a look at these docs to improve your DAG import time:\n* https://airflow.apache.org/docs/apache-airflow/2.4.0/best-practices.html#top-level-python-code\n* https://airflow.apache.org/docs/apache-airflow/2.4.0/best-practices.html#reducing-dag-complexity, PID: 2558\n'}
```
This issue relates to #520 - something we'll be soon working on.

In the meantime, we're increasing the DAG Parsing time in the CI - so
our tests can run again.

We should also review how we run the integration tests - we probably do
not need to run all DAGs against the Python x Airflow matrix, and we
could parallelise some of those.

(cherry picked from commit 358f4b0)
  • Loading branch information
tatiana committed Nov 6, 2023
1 parent 557a5f7 commit c12e88a
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,9 @@ jobs:
env:
AIRFLOW_HOME: /home/runner/work/astronomer-cosmos/astronomer-cosmos/
AIRFLOW_CONN_AIRFLOW_DB: postgres://postgres:[email protected]:5432/postgres
PYTHONPATH: /home/runner/work/astronomer-cosmos/astronomer-cosmos/:$PYTHONPATH
AIRFLOW_CONN_DATABRICKS_DEFAULT: ${{ secrets.AIRFLOW_CONN_DATABRICKS_DEFAULT }}
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 90.0
PYTHONPATH: /home/runner/work/astronomer-cosmos/astronomer-cosmos/:$PYTHONPATH
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
DATABRICKS_WAREHOUSE_ID: ${{ secrets.DATABRICKS_WAREHOUSE_ID }}
Expand Down Expand Up @@ -192,6 +193,7 @@ jobs:
AIRFLOW_CONN_AIRFLOW_DB: postgres://postgres:[email protected]:5432/postgres
PYTHONPATH: /home/runner/work/astronomer-cosmos/astronomer-cosmos/:$PYTHONPATH
AIRFLOW_CONN_DATABRICKS_DEFAULT: ${{ secrets.AIRFLOW_CONN_DATABRICKS_DEFAULT }}
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 90.0
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
DATABRICKS_WAREHOUSE_ID: ${{ secrets.DATABRICKS_WAREHOUSE_ID }}
Expand Down Expand Up @@ -256,6 +258,7 @@ jobs:
env:
AIRFLOW_HOME: /home/runner/work/astronomer-cosmos/astronomer-cosmos/
AIRFLOW_CONN_AIRFLOW_DB: postgres://postgres:[email protected]:5432/postgres
AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 90.0
PYTHONPATH: /home/runner/work/astronomer-cosmos/astronomer-cosmos/:$PYTHONPATH
AIRFLOW_CONN_DATABRICKS_DEFAULT: ${{ secrets.AIRFLOW_CONN_DATABRICKS_DEFAULT }}
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
Expand Down

0 comments on commit c12e88a

Please sign in to comment.