Fernet-key-rotation-optimisation

[bjankie1]

---

This is a follow up to reverted #40758

Current implementation of Fernet key rotation implicitly executes all() method on the processed tables leading to loading all rows to memory.
It's been observed that some users store additional data in variable table which is leading to memory issues during the operation.
This change introduces batch processing of fernet key rotation to avoid it. To be consistent across the tables (variable, connection, trigger) the batching operation was added for all of them.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[bjankie1]

Tests that are failing don't seem even remotely related to introduced change.

_ TestTimeDeltaSensorAsync.test_timedelta_sensor[data_interval_end1-delta1-True] _

self = <MagicMock name='defer' id='140275746064656'>

    def assert_called_once(self):
        """assert that the mock was called only once.
        """
        if not self.call_count == 1:
            msg = ("Expected '%s' to have been called once. Called %s times.%s"
                   % (self._mock_name or 'mock',
                      self.call_count,
                      self._calls_repr()))
>           raise AssertionError(msg)
E           AssertionError: Expected 'defer' to have been called once. Called 0 times.

/usr/local/lib/python3.8/unittest/mock.py:892: AssertionError

[potiuk]

Tests that are failing don't seem even remotely related to introduced change.

Yes. They are flaky tests that are failing in main too and need to be fixed there- previous attempt was not successful #40766

[potiuk]

Should be fixed by - #40813 - it was a really weird timing issue - when test was parsed before hour and executed after hour.

[bjankie1]

Thanks! I will observe this PR and re-run the tests once merged.

[potiuk]

Merged (and I rebased your PR as well)

[bjankie1]

It looks like more flaky tests failing.

[potiuk]

Yeah. Looks, like recent open-lineage change cc: @mobuchowski @kacpermuda - any hint?

[kacpermuda]

Just for reference, OL test was fixed in #40826.