Remove Database Isolation feature flag and run DB isolation tests #41067

potiuk · 2024-07-27T18:18:46Z

This PR removes AIP-44 feature flag and replaces "in-progress-disabled" test with dedicated "DatabaseIsolation" one.

The DatabaseIsolation test will run all "db-tests" with enabled DB isolation mode and running internal-api component - groups of tests marked with "skip-if-database-isolation" will be skipped.

tests/models/test_taskinstance.py (32 tests failing)
tests/operators/test_python.py (2 tests failing)
tests/sensors/test_external_task_sensor.py (2 tests failing)

Some of those tests might need to be excluded as they make no sense to be run (for example if those are tests that are testing scheduler or webserver code). It might also be that we need to improve our test harness for isolation mode in some cases.

How to run the tests? It's fairly easy to run the tests with breeeze (you can checkout this pr with gh pr checkout 41067 and work on top of it.

You can run tests using DB isolation already using database isolation. It works in the way that you need to enter breeze shell --database-isolation, split the terminals with tmux, run internal-api component in one of them and then you can run pytest tests in the other terminal. Again - sources are mounted to inside the container so you can easily iterate on tests (ctrl-c restart internal-api to pick up changes).

See: https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#running-tests-with-database-isolation for detailed description.

One tests that fully works for now is dag_processing/test_job_runner.py for example and a good way to see how it works - you will see that the test initializes DB direcly, but then the dag_processing code calls the internal-api server and you will see messages going back-forth in the "internal-api" output.

You should make sure that you use pytest --run-db-tests-only flag - and all non-db tests will be skipped, that should speed up your tests when you run whole module or directory.

How to submit fixes?

After you make a fix - submit a PR explaining which tests are fixed with Related: #41067 in the commit message, That's it. I will be continuously rebasing this PR on top of main and keeping the inventory of what has been fixed.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2024-07-27T18:27:01Z

This PR will have quite a number of failing tests still - some of them because they should just be disabled - on a module level in most cases (with adding pytest.mark.skuip-if-database-isolation to pytest.mark.db-test.

Some of them because there are still some missing pieces to fix in the internal API.

The "special" DatabaseIsolation test that replaces "InProgressDisabled" runs the special "Database Isolation" mode that allows all our test fixtures and generally "test code" to access the database, while any code that is being tested from "airflow" should get special TracingSession assigned that will fail if the "airflow" code will attempt to use the database directly rather than via internal API.

I will make an extra pass today - once it completes - and disable all the "big" modules that should be disabled from testing, and maybe fix some obvious cases. But anyone (especially @jscheffl @vincbeck and @dstandish ) are most welcome to sign-up to fix some of those in parallel (please comment here when you pick something - I will be merging my fixes individually as separate PRs and will keep on rebasing that PR until we will get it hopefully green before 2.10rc1.

aritra24 · 2024-07-27T18:32:52Z

I'll try taking a look at some failures tomorrow my time

jscheffl · 2024-07-27T20:05:35Z

I LOVE pull-requests which delete more lines than adding. But as being DRAFT, looking forward for review (if family time permits tomorrow)

Related: apache#41067

…pache#41117) When Database Isolation mode is on refreshing task should not check if task instance is already in the session - because the session is not there. Related: apache#41067

…pache#41127) The current code has been only checking when the session is created in tests - to see if the aiflow code is not using the session, However, Sometimes when session is created in the tests and passed to "airflow" code (say running a task) - the session can be passed directly and it does not matter where the session is created but where it is used. This PR implements checking of that - and handles better the edge cases. The message is also improved in this case - we show the exact case where the session was used in airflow, on top of the whole stacktrace so you should be able to see easily what the problem is. Related: apache#41067

…1129) Removes a lot of useless frames from the traceback: * pytest and pluggy * last two "create_session" tracebacks This way it is immediately visible where we should look for problems Related: apache#41067

Those tests should not be run for db isolation tests at all Related: apache#41067

* Remove incremental upper-bind limit The twisted/incremental#106 issue has been addressed in 24.7.2 so we are removing the limit - just in case we also exclude the buggy versions, even if they are yanked. * Skip DB isolation tests for www package Related: apache#41067

This PR adds possibility to handle tests that perform task.run call in DB isolation mode. This call is treated specially - i.e. it will be possible for the run() method to use the DB directly. to set up the task but then when _run_raw_task below will use the DB, there will be an error raised. That allows to fix the `test_core.py` and likely a number of other tests that rely on the run() method to run the tests. This required to improve the _set_ti_attrs method to handle dag_run inside the task instance pydantic to task instance mapping - because we are actually using TaskInstance for those tests not TaskInstancePydantic under the hood. Related: apache#41067

Related: apache#41067

…pache#41250) Some of the tests are skipped, if they are only relevant for webserver (auth tests) - the remaining tests are fixed. Related: apache#41067

Related: apache#41067

…#41252) During AWS provider tests fixing for database isolation, two problems have been found: * Trigger object did not have TriggerPydantic corresponding pydantic object which did not allow the Trigger object to get serialized for Trigger processing * Variable and connection accessors were not restored when Context got deserialized. Related: apache#41067

…he#41266) A lot of the "core" tests are not relevant for database isolation mode and they should be skipped. Related: apache#41067

All provider's tests should pass now in DB isolation mode * some tests are fixed * some modules are excluded as it makes no sense * some modules are excluded and note about adding information about incompatibilities left to be added in the documentation Related: apache#41067

…1296) Related: apache#41067

…#41309) Related: apache#41067

* XCom tests pass DB isolation mode The tests are disabled that are not supposed to work: * warnings are raised on internal API side * get_many is excluded Related: apache#41067 * Update tests/models/test_xcom.py

…che#41471) Related: apache#41067

potiuk · 2024-08-20T14:22:00Z

We still might remove the feature flag in 2.11. This will be easy - we do not need to keep that PR around for that.

…che#41471) Related: apache#41067 (cherry picked from commit 7718bd7)

…41610) * Enable pull requests to be run from v*test branches (#41474) (#41476) Since we switch from direct push of cherry-picking to open PRs against v*test branch, we should enable PRs to run for the target branch. (cherry picked from commit a9363e6) * Prevent provider lowest-dependency tests to run in non-main branch (#41478) (#41481) When running tests in v2-10-test branch, lowest depenency tests are run for providers - because when calculating separate tests, the "skip_provider_tests" has not been used to filter them out. This PR fixes it. (cherry picked from commit 75da507) * Make PROD image building works in non-main PRs (#41480) (#41484) The PROD image building fails currently in non-main because it attempts to build source provider packages rather than use them from PyPi when PR is run against "v-test" branch. This PR fixes it: * PROD images in non-main-targetted build will pull providers from PyPI rather than build them * they use PyPI constraints to install the providers * they use UV - which should speed up building of the images (cherry picked from commit 4d5f1c4) * Add WebEncoder for trigger page rendering to avoid render failure (#41350) (#41485) Co-authored-by: M. Olcay Tercanlı <[email protected]> * Incorrect try number subtraction producing invalid span id for OTEL airflow (issue #41501) (#41502) (#41535) * Fix for issue #39336 * removed unnecessary import (cherry picked from commit dd3c3a7) Co-authored-by: Howard Yoo <[email protected]> * Fix failing pydantic v1 tests (#41534) (#41541) We need to exclude some versions of Pydantic v1 because it conflicts with aws provider. (cherry picked from commit a033c5f) * Fix Non-DB test calculation for main builds (#41499) (#41543) Pytest has a weird behaviour that it will not collect tests from parent folder when subfolder of it is specified after the parent folder. This caused some non-db tests from providers folder have been skipped during main build. The issue in Pytest 8.2 (used to work before) is tracked at pytest-dev/pytest#12605 (cherry picked from commit d489826) * Add changelog for airflow python client 2.10.0 (#41583) (#41584) * Add changelog for airflow python client 2.10.0 * Update client version (cherry picked from commit 317a28e) * Make all test pass in Database Isolation mode (#41567) This adds dedicated "DatabaseIsolation" test to airflow v2-10-test branch.. The DatabaseIsolation test will run all "db-tests" with enabled DB isolation mode and running `internal-api` component - groups of tests marked with "skip-if-database-isolation" will be skipped. * Upgrade build and chart dependencies (#41570) (#41588) (cherry picked from commit c88192c) Co-authored-by: Jarek Potiuk <[email protected]> * Limit watchtower as depenendcy as 3.3.0 breaks moin. (#41612) (cherry picked from commit 1b602d5) * Enable running Pull Requests against v2-10-stable branch (#41624) (cherry picked from commit e306e7f) * Fix tests/models/test_variable.py for database isolation mode (#41414) * Fix tests/models/test_variable.py for database isolation mode * Review feedback (cherry picked from commit 736ebfe) * Make latest botocore tests green (#41626) The latest botocore tests are conflicting with a few requirements and until apache-beam upcoming version is released we need to do some manual exclusions. Those exclusions should make latest botocore test green again. (cherry picked from commit a13ccbb) * Simpler task retrieval for taskinstance test (#41389) The test has been updated for DB isolation but the retrieval of task was not intuitive and it could lead to flaky tests possibly (cherry picked from commit f25adf1) * Skip database isolation case for task mapping taskinstance tests (#41471) Related: #41067 (cherry picked from commit 7718bd7) * Skipping tests for db isolation because similar tests were skipped (#41450) (cherry picked from commit e94b508) --------- Co-authored-by: Jarek Potiuk <[email protected]> Co-authored-by: Brent Bovenzi <[email protected]> Co-authored-by: M. Olcay Tercanlı <[email protected]> Co-authored-by: Howard Yoo <[email protected]> Co-authored-by: Jens Scheffler <[email protected]> Co-authored-by: Bugra Ozturk <[email protected]>

…che#41471) Related: apache#41067 (cherry picked from commit 7718bd7)

) (#41954) * Simpler task retrieval for taskinstance test (#41389) The test has been updated for DB isolation but the retrieval of task was not intuitive and it could lead to flaky tests possibly (cherry picked from commit f25adf1) * Skip database isolation case for task mapping taskinstance tests (#41471) Related: #41067 (cherry picked from commit 7718bd7) --------- Co-authored-by: Jarek Potiuk <[email protected]> Co-authored-by: Tzu-ping Chung <[email protected]>

potiuk added the full tests needed We need to run full set of tests for this PR to merge label Jul 27, 2024

potiuk requested review from ashb, jedcunningham, kaxil, ryanahamilton, bbovenzi, pierrejeambrun and bolkedebruin as code owners July 27, 2024 18:18

boring-cyborg bot added area:CLI area:dev-tools area:serialization area:webserver Webserver related Issues labels Jul 27, 2024

potiuk marked this pull request as draft July 27, 2024 18:18

potiuk requested review from ferruzzi, dstandish, aritra24, jscheffl and vincbeck July 27, 2024 18:19

potiuk force-pushed the remove-database-isolation-feature-flag branch from d2fc06f to 78db57a Compare July 27, 2024 18:29

potiuk force-pushed the remove-database-isolation-feature-flag branch from 78db57a to f1ef44e Compare July 27, 2024 18:41

This was referenced Jul 27, 2024

AIP-44 Migrate to new serializer #31213

Open

Make sure all tests are passing in DB Isolation mode #41069

Closed

potiuk linked an issue Jul 27, 2024 that may be closed by this pull request

Make sure all tests are passing in DB Isolation mode #41069

Closed

potiuk force-pushed the remove-database-isolation-feature-flag branch from f1ef44e to 7454874 Compare July 27, 2024 18:59

potiuk mentioned this pull request Jul 27, 2024

Fix broken/renamed internal API method #41070

Merged

potiuk force-pushed the remove-database-isolation-feature-flag branch from 7454874 to f51a5de Compare July 27, 2024 19:57

potiuk force-pushed the remove-database-isolation-feature-flag branch from f51a5de to c641e40 Compare July 27, 2024 22:24

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Fix failing secret backends database isolation tests (apache#41088)

0e383d6

Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Disable db isolation tests for api_connexion tests (apache#41130)

ac06e20

Those tests should not be run for db isolation tests at all Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Remove API/CLI commends from db isolation mode tests (apache#41197)

f3e049e

Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Fix all smaller providers to work with DB isolation mode (apache#41251)

ce601f0

Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Fix all AWS provider tests for database isolation (apache#41253)

040189e

Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Fix almost 100 tests for taskinstance for DB isolation mode (apache#4…

7775048

…1296) Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Fix render template task_instance tests for db isolation mode (apache…

601536b

…#41309) Related: apache#41067

molcay pushed a commit to VladaZakharova/airflow that referenced this pull request Aug 19, 2024

Skip database isolation case for task mapping taskinstance tests (apa…

5b237b6

…che#41471) Related: apache#41067

romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Aug 20, 2024

Skip database isolation case for task mapping taskinstance tests (apa…

460a183

…che#41471) Related: apache#41067

potiuk closed this Aug 20, 2024

utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Aug 21, 2024

Skip database isolation case for task mapping taskinstance tests (apa…

15bb0aa

…che#41471) Related: apache#41067 (cherry picked from commit 7718bd7)

utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Sep 2, 2024

Skip database isolation case for task mapping taskinstance tests (apa…

0f35c53

…che#41471) Related: apache#41067 (cherry picked from commit 7718bd7)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Database Isolation feature flag and run DB isolation tests #41067

Remove Database Isolation feature flag and run DB isolation tests #41067

potiuk commented Jul 27, 2024 •

edited

Loading

potiuk commented Jul 27, 2024

aritra24 commented Jul 27, 2024

jscheffl commented Jul 27, 2024

potiuk commented Aug 20, 2024

Remove Database Isolation feature flag and run DB isolation tests #41067

Remove Database Isolation feature flag and run DB isolation tests #41067

Conversation

potiuk commented Jul 27, 2024 • edited Loading

potiuk commented Jul 27, 2024

aritra24 commented Jul 27, 2024

jscheffl commented Jul 27, 2024

potiuk commented Aug 20, 2024

potiuk commented Jul 27, 2024 •

edited

Loading