Broken macro support in DataProcJobBuilder task_id parameter

[rafalh]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

8.5.0

Apache Airflow version

2.3.4

Operating System

Linux

Deployment

Composer

Deployment details

No response

What happened

I am passing string containing macros to DataProcJobBuilder task_id parameter:

{{ (dag.dag_id + '-' + task.task_id.replace('_', '-'))[:90] }}

I use created job with DataprocSubmitJobOperator.
It worked fine in composer-2.0.27-airflow-2.2.5 (apache-airflow-providers-google==2022.9.6+composer). Now I tried upgrading to composer-2.1.2-airflow-2.3.4 (apache-airflow-providers-google==8.5.0) and it no longer works because after #23791 all "." are replaced by "_" before macro evaluation. It causes error:

'dag_dag_id' is undefined

What you think should happen instead

Optimally DataProcJobBuilder should allow using macros as before.
PR that broke this usage was supposed to fix task groups (#23439). Perhaps that fix (string replacement) could be moved to a different place e.g. DataprocJobBaseOperator default value. This way API of DataProcJobBuilder would still allow macros.

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

The replacement should be done AFTER running JINJA. Anyone wants to provide a fix? (@eladkal :) ? )

[github-actions]

This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.