Status of testing Providers that were prepared on July 28, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.27.0rc1

• Add RedriveExecution support to StepFunctionStartExecutionOperator (#40976): @gopidesupavan
• AIP-62: add hook lineage collection support for S3Hook (#40819): @mobuchowski
• Introduce Amazon Kinesis Analytics V2 (Managed Service for Apache Flink application) (#40765): @gopidesupavan
• Make EMR Container Trigger max attempts retries match the Operator (#41008): @o-nikolas
  Linked issues:
  • Linked Issue #40483: @akomisarek
• Fix RdsStopDbOperator operator in deferrable mode (#41059): @vincbeck
• Fix RedshiftCreateClusterOperator to always specify PubliclyAccessible (#40872): @vincbeck
• Update example_redshift and example_redshift_s3_transfers to use RedshiftDataHook instead of RedshiftSQLHook (#40970): @vincbeck
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• [AIP-62] Translate AIP-60 URI to OpenLineage (#40173): @kacpermuda
  Linked issues:
  • Linked Issue #40335: @JDarDagran
  • Linked Issue #38767: @mobuchowski
• Move AWS Managed Service for Apache Flink sensor states to Hook (#40896): @gopidesupavan
• Replace usages of task context logger with the log table (#40867): @dstandish

Provider fab: 1.2.2rc1

• Bug fix: sync perm command not able to use custom security manager (#41020): @dirrao
• Bump version checked by FAB provider on logout CSRF protection to 2.10.0 (#40784): @potiuk
  Linked issues:
  • Linked Issue #40470: @Al2a2d2m
  • Linked Issue #40479: @shahar1
• AIP-44 make database isolation mode work in Breeze (#40894): @potiuk

Provider google: 10.21.1rc1

• Dynamic import MySqlHook/PostgresHook in CloudSQLDatabaseHook (#41009): @jhongy1994
  Linked issues:
  • Linked Issue #40175: @bourliam
• Make google_analytics_admin system tests self sufficient (#40951): @moiseenkov
• Fix Custom Training Job operators to accept results without a model in def mode (#40685): @e-galan
  Linked issues:
  • Linked Issue #40476: @vignesh-sc
• Fix behavior for reattach_state parameter in BigQueryInsertJobOperator (#40664): @VladaZakharova
• Refactor dataproc system tests (#40720): @VladaZakharova
• openlineage: migrate OpenLineage provider to V2 facets. (#39530): @JDarDagran
• Resolve google cloud sql deprecation in tests (#40834): @dirrao
• Update default value for job_name in DataflowConfiguration (#40645): @e-galan
  Linked issues:
  • Linked Issue #40515: @turb

All users involved in the PRs:
@moiseenkov @dstandish @dirrao @potiuk @JDarDagran @gopidesupavan @mobuchowski @VladaZakharova @o-nikolas @e-galan @kacpermuda @vincbeck @jhongy1994

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

I checked all my changes. Also verfied that FAB 1.2.2 fixes the logout issue for 2.9.2. Looks good.

[moiseenkov]

Hi, #40951 works as expected

[JDarDagran]

#39530 adds apache-airflow-providers-common-compat>=1.1.0 dependency to amazon and google providers. You can't install new packages without apache-airflow-providers-common-compat==1.1.0rc0 release (which may be missed when following instructions from the docs to test RCs with flag --no-deps)

[potiuk]

#39530 adds apache-airflow-providers-common-compat>=1.1.0 dependency to amazon and google providers. You can't install new packages without apache-airflow-providers-common-compat==1.1.0rc0 release (which may be missed when following instructions from the docs to test RCs with flag --no-deps)

Ah right - so common-compat should be released as well here .. cc: @eladkal

[gopidesupavan]

Yes it is failing to install package.

Once it updated i will test these #40976 , #40765 and #40896.

[dirrao]

#41020: working as expected.

[eladkal]

I'll cut rc2 for amazon, google + rest of the providers after releasing fab.

[akomisarek]

I think I will wait for rc2 cut, as I am not able to test it easily in my environment (we start via Docker):

Without using --no-deps as reported above the Depencencies are not installing.

Adding --no-deps allows the dependencies to install, but I think I have got further version clashes:
airflow-1   |   File "/home/airflow/.local/lib/python3.10/site-packages/pydantic_core/__init__.py", line 29, in <module>
airflow-1   |     from .core_schema import CoreConfig, CoreSchema, CoreSchemaType, ErrorType
airflow-1   |   File "/home/airflow/.local/lib/python3.10/site-packages/pydantic_core/core_schema.py", line 15, in <module>
airflow-1   |     from typing_extensions import deprecated
airflow-1   | ImportError: cannot import name 'deprecated' from 'typing_extensions' (/home/airflow/.local/lib/python3.10/site-packages/typing_extensions.py)

We might have some dependencies causing those issues, but that's why I don't want to omit them!

Thanks, I will keep an eye on the thread and confirm/retest when the new RC is in place. Thanks for the support 🙏

[eladkal]

Thank you everyone. Providers are released.
Providers amazon and google are excluded - I will cut followup release shortly

I invite everyone to help improve providers for the next release, a list of open issues can be found here.