Add context to execution_date_fn in ExternalTaskSensor

[Acehaidrey]

The purpose of this pr is to get more flexibility with the ExternalTaskSensor to be able to pass the context in the execution_date_fn for the purpose of doing some logic that cannot just be handled with the execution date or the time delta.
One user case that regularly comes up is that the users want to get the external task/dag id's schedule run time and do some automated logic vs trying to manually go look up the time and do a timedelta.

This allows us to build smarter functions. This pr also keeps the legacy way to keep backwards compatibility if only one arg is passed, and throws an error if more than 2 args are passed.

A test case is written to pass two args to our function and make sure it passes/runs to success.

---

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[Acehaidrey]

@jhtimmins if you don't mind when you get a chance to look at some of this and share your thoughts as well

[jhtimmins]

@Acehaidrey Happy to take a look!

[Acehaidrey]

@jhtimmins thank you so much :) I know I've been taking a lot of your time these days so appreciate it!

[jhtimmins]

@Acehaidrey Of course :) Always happy to take a look.

I think the use case makes sense, but the implementation could be made more explicit.

Personally, I think it'd make more sense to use keyword arguments for both, and then determine behavior by the presence/absence of values.

def _handle_execution_date_fn(self, execution_date=None, context={}):
        # Assert that either execution_date or context is None
               # Otherwise, throw an exception.
        # if execution_date
               # return self.execution_date_fn(execution_date)
        # return self.execution_date_fn(context)

This would cover the following cases:

1. Current behavior w just execution_date value
2. Proposed behavior w context variables
3. Proposed behavior w empty context dictionary

Worth getting a few more pairs of eyes on this.

What do you think about this approach?

[jhtimmins]

For recommendation details, see the top-level conversation.
#8702 (comment)

[dimberman]

@Acehaidrey I agree with @jhtimmins. I think having one function is cleaner and will make it easier if we decide to deprecate one of the methods later.

[mik-laj]

Can you add some docs to guide?
https://airflow.readthedocs.io/en/latest/howto/operator/external_task_sensor.html

[Acehaidrey]

@jhtimmins @dimberman thank you both for your feedback.
I would really like that as well but I'm sorry I'm not seeing how to do this exactly.

Say a user has an execution_date_fn be

def myfunc(dt):
    print(dt)

So they pass this to the operator. Now in the operator execute code that we are making the changes, in the

def _handle_execution_date_fn(self, execution_date=None, context={})

I will be passing both from the execute method, date and context. But how do I know from the users point of view, if they were expecting the execution_date or the context.
Does that make sense? It could be obvious but I'm not seeing how to implement this so any help on this would be great.

[Acehaidrey]

@jhtimmins @dimberman can you please comment when you have a minute?

[Acehaidrey]

@jhtimmins sorry for pestering but would love to get these in and close out

[Acehaidrey]

@jhtimmins @dimberman @ashb mind chiming in on how to make the changes or reconsider these? would love to just advance this and close it out

[Acehaidrey]

@jhtimmins @ashb or anyone you think can. add to this convo mind doing that so we can get this closed out too?

[Acehaidrey]

@dimberman can you please review this again so we can get it completed. Thank you @ashb for looking over this and approving

[dimberman]

Hi @Acehaidrey apologies for the delay. I've merged the PR and will work to backport it on 1.10.11 :)

[Acehaidrey]

No problem thank you @dimberman !

[yuqian90]

This change stops people from doing loop variable capture. There are more friendly ways to allow compatibility, such as using inspect.signature to figure out the actual argument names.

    for day in range(1, 7):
        sensor = ExternalTaskSensor(
            task_id="sensor_{}".format(day),
            external_dag_id=external_dag_id,
            external_task_id=external_task_id,
            execution_date_fn=lambda execution_date, local_day=day: ...,
        )