Don't schedule dummy tasks

[mik-laj]

#7871

---

Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[codecov-io]

Codecov Report

Merging #7880 into master will decrease coverage by 0.71%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7880      +/-   ##
==========================================
- Coverage   86.95%   86.23%   -0.72%     
==========================================
  Files         927      927              
  Lines       44974    44977       +3     
==========================================
- Hits        39107    38788     -319     
- Misses       5867     6189     +322     

Impacted Files	Coverage Δ
airflow/operators/dummy_operator.py	100.00% <ø> (ø)
airflow/jobs/scheduler_job.py	90.92% <100.00%> (+0.47%)	⬆️
...flow/providers/apache/cassandra/hooks/cassandra.py	21.25% <0.00%> (-72.50%)	⬇️
...w/providers/apache/hive/operators/mysql_to_hive.py	35.84% <0.00%> (-64.16%)	⬇️
airflow/kubernetes/volume_mount.py	44.44% <0.00%> (-55.56%)	⬇️
airflow/providers/postgres/operators/postgres.py	47.82% <0.00%> (-52.18%)	⬇️
airflow/providers/redis/operators/redis_publish.py	50.00% <0.00%> (-50.00%)	⬇️
airflow/kubernetes/volume.py	52.94% <0.00%> (-47.06%)	⬇️
airflow/providers/mongo/sensors/mongo.py	53.33% <0.00%> (-46.67%)	⬇️
airflow/kubernetes/pod_launcher.py	47.18% <0.00%> (-45.08%)	⬇️
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a6fd74e...d851a2f. Read the comment docs.

[eladkal]

I think it's a good idea that DummyOperator won't be scheduled but I don't agree with the statement "DummyOperator doesn't actually do work".

For example:
If the Dummy is marked always as success what will happen to cases where DummyOperator is used with trigger rules for some flows ?

Example:

final = DummyOperator(
                              dag=dag,
                              task_id="error_check",
                              trigger_rule=TriggerRule.ONE_FAILED
                            )
[task1, task2, task3] >> final

In this case marking final as FAILED result in failure of the DAG which invoke on_failure_callback.

Also it's possible in the above example that there is ExternalSensor from dag2 on final so it's important that final won't be marked as SUCCESS automatically.

Probably just need to update Updating.md alerting change of behavior as DummyOperator will not longer honor trigger_rule. Maybe even not allowing users to overwrite the default trigger_rule ?

[mik-laj]

But the trigger rules will still be respected. I added this code when the task is to be marked for execution by the scheduler and the trigger rules have already been checked. All callbacks will also be executed, because when the task has callbacks, it will still be passed to the scheduler.

This optimization applies only to cases when the task has no callbacks and when the decision to execute it has been made.

[eladkal]

@mik-laj This is what confused me.
Maybe better saying something like:
The task is evaluated by the scheduler but never processed by the executor
WDYT?

[potiuk]

Yeah. Might be worth adding!

[mik-laj]

I updated the description.

[potiuk]

Just a small update to description needed but otherwise good.

[potiuk]

Yeah. Might be worth adding!