[Feature]Open source AirflowFlyteOperator (@Lyft) for better integration with airflow

[kumare3]

Motivation: Why do you think this is important?
As many of the OSS users are on Airflow, it would really help the community to have AirflowFlyteOperator available in Open Source. This would make it possible to use Flyte when some existing workflows are leveraging Airflow.

Goal: What should the final outcome look like, ideally?
Users using Airflow should be able to leverage Flyte when possible and it should be possible to add a new step in the Airflow DAG, which runs the execution on Flyte

Describe alternatives you've considered
This is possible to do by install flytekit and writing an operator themselves or using the PythonOperator, but this has a higher learning curve and needs more work on the user.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

[Optional] Propose: Link/Inline
Operator interface:
(Project: str, Domain:str , LaunchPlanName:str, version: str=None, flyte_kwargs: Dict[str, Any]=None)

Steps

1. Retrieve Launch Plan using ("project", "domain", "name", [optional: "version"]). If version is not specified retrieve latest
2. Execute the launch plan with inputs=flyte_kwargs and generate a deterministic name for the run with the attempt number
3. Wait for the execution to complete
4. OnKill, pass the kill switch to Flyte

Is this a blocker for you to adopt Flyte
NA

[samhita-alla]

https://github.com/flyteorg/airflow-provider-flyte