Status of testing of Apache Airflow 2.3.3rc3

[ephraimbuddy]

Body

We have a kind request for all the contributors to the latest Apache Airflow RC 2.3.3rc3.

Could you please help us to test the RC versions of Airflow?

Please let us know in the comment if the issue is addressed in the latest RC.

• Switch Markdown engine to markdown-it-py (#19702): @uranusjr @torbjornvatn
  Linked issues:
  • Switch Markdown engine to markdown-it-py (#16435)
• Fix StatD timing metric units (#21106): @viktorvia
  Linked issues:
  • Some timing metrics are in seconds but reported as milliseconds (#20804)
• Parse error for task added to multiple groups (#23071): @uranusjr @wolfier
  Linked issues:
  • Using TaskGroup without context manager (Graph view visual bug) (#14864)
• Check bag DAG schedule_interval match tiemtable (#23113): @uranusjr @sdebruyn
  Linked issues:
  • Show DAG parse error if 'timetable' and 'schedule_interval' are set together (#23087)
• Allow for LOGGING_LEVEL=DEBUG (#23360): @GApfel
• Fix closing connection dbapi.get_pandas_df (#23452): @bauerfranz @hubert-pietron
  Linked issues:
  • closing connection chunks in DbApiHook.get_pandas_df (#22947)
• Fix permission issue for dag that has dot in name (#23510): @ephraimbuddy
• ExternalTaskSensor respects soft_fail if the external task enters a failed_state (#23647): @argibbs @alexbegg
  Linked issues:
  • ExternalTaskSensor should skip if soft_fail=True and external task in one of the failed_states (#19754)
• Fix mapped task immutability after clear (#23667): @ephraimbuddy
• Maintain grid view selection on filtering upstream (#23779): @bbovenzi
• Add TaskInstance State 'REMOVED' to finished states and success states (#23797): @ephraimbuddy
• Do not fail requeued TIs (#23846): @malthe @tanelk
  Linked issues:
  • Race condition between Triggerer and Scheduler (#23824)
  • Fix race condition between triggerer and scheduler (#21316)
• Remove special serde logic for mapped op_kwargs (#23860): @uranusjr @erdos2n
  Linked issues:
  • Dynamic Task Mapping not working with op_kwargs in PythonOperator (#23833)
• Grid fix details button truncated and small UI tweaks (#23934): @pierrejeambrun
• Replace use_task_execution_date with use_task_logical_date (#23983): @eladkal
• Drop Python 3.6 compatibility objects/modules (#24048): @Taragolis
• Removing magic numbers from api_connecxion (#24050): @bernardocouto
• Rename Permissions to Permission Pairs. (#24065): @zachliu @gbonazzoli @jhtimmins
  Linked issues:
  • Random "duplicate key value violates unique constraint" errors when initializing the postgres database (#23512)
  • Airflow 2.3.1 - gunicorn keeps removing and adding Permission menu access on Permissions to role Admin (#23926)
• Handle occasional deadlocks in trigger with retries (#24071): @humit0 @potiuk
  Linked issues:
  • Triggerer process die with DB Deadlock (#23639)
• Reduce grid view API calls (#24083): @bbovenzi
• Unify return_code interface for task runner (#24093): @pingzh
• Do not support MSSQL less than v2017 in code (#24095): @ephraimbuddy
• Refactor DagRun.verify_integrity (#24114): @ephraimbuddy
• Don't crash scheduler if exec config has old k8s objects (#24117): @patryk126p @dstandish
  Linked issues:
  • Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe' (#23727)
• Fix useTasks crash on error (#24152): @bbovenzi
• Apply per-run log templates to log handlers (#24153): @uranusjr
• Prevent dag dependencies page from crashing (#24166): @bbovenzi
• Update 'rich' to latest version across the board. (#24186): @potiuk
• ScheduleInterval nullable true added in openapi spec (#24253): @omarsmak @Bowrna
  Linked issues:
  • ScheduleInterval schema in OpenAPI specs should have "nullable: true" otherwise generated OpenAPI client will throw an error in case of nullable "schedule_interval" (#22358)
• Use get_hostname instead of socket.getfqdn (#24260): @jedcunningham
• Optimize calendar view for cron scheduled DAGs (#24262): @jedcunningham @potiuk
  Linked issues:
  • Calendar view is taking too long to load in airflow 2.3.0 (#23602)
• Speed up grid_data endpoint by 10x (#24284): @ashb
• Fix bugs in URI constructor for MySQL connection (#24320): @MaksYermak
• Check for run_id for grid task groups (#24327): @jedcunningham @bbovenzi
  Linked issues:
  • Fix and speed up grid view (#23947)
  • Grid data: do not load all mapped instances (#23813)
• TI.log_url fix for map_index (#24335): @fritz-astronomer
  Linked issues:
  • TI.log_url is incorrect with mapped tasks (#24328)
• Add typescript (#24337): @bbovenzi
• DebugExecutor use ti.run() instead of ti._run_raw_task (#24357): @sarvothaman @o-nikolas
  Linked issues:
  • Backfill crashes with "KeyError: TaskInstanceKey" when task has retries (#13322)
• Mask secrets in stdout for CLI command (#24362): @uranusjr @alex-astronomer @marclamberti
  Linked issues:
  • Mask secrets in stdout for airflow tasks test ... CLI command (#17476) (#21281)
  • Sensitive variables don't get masked when rendered with airflow tasks test (#17476)
• Add tests for the grid_data endpoint (#24375): @ashb
• Fix links to sources for examples (#24386): @potiuk
• Upgrade FAB to 4.1.1 (#24399): @potiuk
• Fix flaky order of returned dag runs (#24405): @potiuk
• Improve grid rendering performance with a custom tooltip (#24417): @bbovenzi
• Upgrade to react 18 and chakra 2 (#24430): @bbovenzi
• Remove framer-motion from custom tooltip (#24449): @bbovenzi
• Add missing types to FSHook (#24470): @lmarvaud
• Return empty dict if Pod JSON encoding fails (#24478): @dstandish
• Upgrade to webpack 5 (#24485): @bbovenzi
• Add indexes for CASCADE deletes for task_instance (#24488): @wolfier @dstandish
  Linked issues:
  • airflow db clean task_instance takes a long time (#24484)
• Remove internet explorer support (#24495): @bbovenzi
• Add cache_ok flag to sqlalchemy TypeDecorators. (#24499): @jkinkead @nk74
  Linked issues:
  • SAWarning: TypeDecorator UtcDateTime(timezone=True) will not produce a cache key because the cache_ok attribute is not set to True (#22647)
• Fix toast messages (#24505): @bbovenzi
• Fix deprecated log_id_template value (#24506): @jedcunningham
• Migrate jsx files that affect run/task selection to tsx (#24509): @bbovenzi
• Seed log_template table (#24511): @jedcunningham
• Update flask-appbuilder authlib/oauth dependency (#24516): @potiuk
• Get rid of TimedJSONWebSignatureSerializer (#24519): @potiuk
  Linked issues:
  • Upgrade to FlaskAppBuilder 4.0.* (#22397)
• Do not calculate grid root instances (#24528): @bbovenzi
• Patch sql_alchemy_conn if old Postgres schemes used (#24569): @slyapustin @dstandish
  Linked issues:
  • Patch sql_alchemy_conn if old postgres scheme used (#22333)
• Fix migration 0080_2_0_2 - Replace null values before setting column not null (#24585): @ephraimbuddy @tm-drtina
  Linked issues:
  • Support generating SQL script for upgrades (#20962)
  • Migration changes column to NOT NULL without updating NULL data first (#24566)
• Move fallible ti.task.dag assignment back inside try/except block (#24533) (#24592): @ejk43
  Linked issues:
  • es_task_handler.py crashes accessing "View Logs in Elasticsearch" (#24533)
• Upgrade FAB to 4.1.2 (#24619): @potiuk
• We now need at least Flask-WTF 0.15 (#24621): @ashb @potiuk
  Linked issues:
  • Upgrade FAB to 4.1.1 (#24399)
• Don't render child rows for closed groups (#24637): @bbovenzi
• Fix timestamp defaults for sensorinstance (#24638): @dstandish
• Fix Grid vertical scrolling (#24684): @bbovenzi
• Debounce status highlighting in Grid view (#24710): @bbovenzi
• Address all yarn test warnings (#24722): @bbovenzi
• Fix grid date ticks (#24738): @bbovenzi
  Linked issues:
  • Fix Grid vertical scrolling (#24684)
• Remove upper-binding for SQLAlchemy (#24819): @potiuk
• Fix cycle bug with attaching label to task group (#24847): @ashb @rustikk
  Linked issues:
  • Applying labels to task groups shows a cycle in the graph view for the dag (#19785)
• improve grid date tick spacing (#24849): @bbovenzi
• Fix exception in mini task scheduler. (#24865): @ashb

Thanks to all who contributed to the release (probably not a complete list!):
@Bowrna @GApfel @uranusjr @alexbegg @sdebruyn @nk74 @argibbs @slyapustin @tanelk @rustikk @omarsmak @hubert-pietron @ashb @marclamberti @eladkal @wolfier @jhtimmins @torbjornvatn @bernardocouto @gbonazzoli @erdos2n @sarvothaman @bbovenzi @dstandish @potiuk @jedcunningham @o-nikolas @MaksYermak @humit0 @tm-drtina @Taragolis @ejk43 @ephraimbuddy @patryk126p @malthe @viktorvia @fritz-astronomer @jkinkead @alex-astronomer @pierrejeambrun @pingzh @lmarvaud @zachliu @bauerfranz

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[GApfel]

#23360 is addressed

[zoid-w]

#23833 is addressed by #23860 tested/verified on :

• 2.3.3rc2-python3.10
• 2.3.3rc3-python3.10

[phanikumv]

Tested that example_python_operator DAG runs successfully after bringing up airflow using 2.3.3rc3 image. Though this isn't related any of the fixes above, it ensured that the image is stable.

cc @kaxil

[raphaelauv]

[ejk43]

A quick note on issue #24533

I believe the relevant change (#24592) would be packaged as part of apache-airflow-providers-elasticsearch, not apache-airflow 2.3.3rc3 -- so I dont think this is actually solved with the 2.3.3rc3 release. May need a bump to apache-airflow-providers-elasticsearch?

[potiuk]

I believe the relevant change (#24592) would be packaged as part of apache-airflow-providers-elasticsearch, not apache-airflow 2.3.3rc3 -- so I dont think this is actually solved with the 2.3.3rc3 release. May need a bump to apache-airflow-providers-elasticsearch?

I think this was fix in both "core" and "es provider" - but the way it is implemented, does not require both sides to be fixed they are rather independent. So ES provider (when released) will fix the second part of the problem independently.

[potiuk]

I tested most of my stuff (marked is as ok). There is PR #24071 where It's hard to reproduce the problem as it was occasional deadlock so I asked the users who raised it in #23639 to help to test it.

[ephraimbuddy]

Airflow 2.3.3 has been released. Thank you all for testing this release