Make --executor flag works for breeze start-airflow command

[potiuk]

Body

Currently breeze start-airflow command only uses LocalExecutor and starts scheduler, triggerer and webserver. However it would be nice if it accepts --executor flag that could accept other executors. This would be much nicer to test some scenarios - mostly with regards to Celery executor.

Such a flag could:

• enable --celery integraiton (With rabbitmq and redis)
• also start worker (optionallly more than one)
• also start flower

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ibrahimdurmus]

i can pick this up, if no one is looking at it. thanks

[victorjourne]

Hi @ibrahimdurmus, have you started working on it?

[ibrahimdurmus]

yes i am working on it thanks @victorjourne

[pierrejeambrun]

Looking forward to this one. I remember a couple of times where this would have been really handy 😀

[utkarsharma2]

@potiuk @pierrejeambrun Can you please assign this ticket to me?

[eladkal]

Did #30644 solve this issue?

[utkarsharma2]

Did #30644 solve this issue?

@eladkal This pr adds support for celeryExecutor and localExecutor. We still need support for the remaining executors. I was planning to work on the remaining tasks shortly.

[eladkal]

@utkarsharma2 are you still working on this issue?

[potiuk]

It's done actually :)

[potiuk]

the K8S executor is quite a bit more complex, and we actually have breeze k8s suite of commands that you can deploy Airflow with k8s - including, k8s, Celery and Local executor - We need a k8s instance to run k8s executor, It would be nice to support k8s executor from the docker-compose's breeze but it is quite complex. I'd rather get a better description on how to use things like https://www.getambassador.io/docs/telepresence/latest/quick-start for the k8s suite of commands and possibly add some integration there.

[potiuk]

I created #40005

[utkarsharma2]

@utkarsharma2 are you still working on this issue?

I wanted to work on it, but there was a learning curve for me. If no one picks up #40005 hopefully, I can resume working on it in the next week or so.