[Airflow] Fixed run format returned by the lineage_parent_id macro

[blacklight]

Problem

The macro currently returns a string in the format <namespace>/<name>/<run_id>.

However, in this string:

• name should be <dag_id>.<task_id>, not a UUID.

• run_id should be a UUID, not <run_timestamp>.<try_number>.

This is to comply with the OpenLineage conventions used everywhere else.

Closes: #2488

Solution

As stated above, name and run_id should be populated in the expected format.

The change may break back-compatibility in cases where lineage_parent_id is used without keyword-arguments, since the signature has changed from:

def lineage_parent_id(run_id, task: "BaseOperator", task_instance: "TaskInstance"):
  ...

To:

def lineage_parent_id(
  task: "BaseOperator", task_instance: "TaskInstance", run_id: typing.Optional[str] = None
):
  ...

That makes the signature of lineage_parent_id consistent with that of lineage_run_id.

run_id is now an optional parameter - it defaults to the one calculated via lineage_run_id for the given instance, and it can optionally be overridden.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

One-line summary:

Fixed run format returned by the lineage_parent_id Airflow macro.

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[boring-cyborg]

Thanks for opening your first OpenLineage pull request! We appreciate your contribution. If you haven't already, please make sure you've reviewed our guide for new contributors (https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md).

[kacpermuda]

LGTM, if the CI goes through i think we can merge that @blacklight

[mobuchowski]

@blacklight still want to contribute that? I think this

OpenLineage/integration/airflow/tests/integration/tests/airflow/dags/dbt_dag.py

Line 11 in 31f8ce5

PLUGIN_MACRO = "{{ macros.OpenLineagePlugin.lineage_parent_id(run_id, task, task_instance) }}"

breaks tests.

[blacklight]

@mobuchowski yes I'm still planning to merge it - I just didn't have much bandwidth in these days to look into the failing tests.

I'll try and get the tests to pass today.

[mobuchowski]

@blacklight yes.

[mobuchowski]

The relevant test logs are in the artifacts tab on circleCI: https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9985/workflows/e3653e9e-2ffd-4f11-bf6d-673d80738427/jobs/199074/artifacts

[blacklight]

@mobuchowski I have opened a new PR here.

I had conflicting commits pushed both with my work and personal account, then upon git rebase HEAD~n --signoff git happened to eagerly also sign a bunch of merge commits - ops.

I'll close this one and wait for the tests to complete on the other side.