Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist start/end date and duration for DummyOperator Task Instance #8663

Merged
merged 1 commit into from
May 2, 2020

Conversation

XD-DENG
Copy link
Member

@XD-DENG XD-DENG commented May 1, 2020

This PR is to address issue #8662

Currently, task instances of DummyOperators are always marked as SUCCESS then immediately being skipped. Other entries like start_date, end_date, and duration are not marked for it in the database (in task_instance table in DB, start_date, end_date, and duration are all NULL for DummyOperator tasks).

One of the issues resulted from this is that when we hover over task instances of DummyOperator in RBAC UI, the started and ended times are always showing the CURRENT time, instead of the the actual task started and ended time (like other operators/sensors).

For details, please refer to #8662


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label May 1, 2020
@XD-DENG XD-DENG force-pushed the feature/fix-date-for-dummyoperator branch from 3d70468 to fdc7694 Compare May 1, 2020 09:42
    Otherwise the behavior in UI is incorrect

    Addressing issue apache#8662
@XD-DENG XD-DENG force-pushed the feature/fix-date-for-dummyoperator branch from fdc7694 to 4a550ae Compare May 1, 2020 10:19
@ashb
Copy link
Member

ashb commented May 1, 2020

Sounds like there's also a bug in the UI code - I would have expected it to show nothing rather than current time

@XD-DENG
Copy link
Member Author

XD-DENG commented May 1, 2020

Hi @ashb , IMO as a user it's good to know when this DummyOperator task is processed by the scheduler, so showing the start/end time for DummyOperator tasks in UI is useful

@ashb
Copy link
Member

ashb commented May 1, 2020

Oh yes absolutely.

I just meant cos of the bug with the missing fields I would have expected the UI to show nothing.

@XD-DENG
Copy link
Member Author

XD-DENG commented May 1, 2020

Oh yes absolutely.

I just meant cos of the bug with the missing fields I would have expected the UI to show nothing.

Ha, get what you mean. That's exactly the behaviour in the non-RBAC UI.

@XD-DENG
Copy link
Member Author

XD-DENG commented May 1, 2020

@ashb you are right that the UI JS code is handling this correctly,

function generateTooltipDateTimes(startDate, endDate, dagTZ) {
and
tt += generateTooltipDateTimes(ti.start_date, ti.end_date, dagTZ); // dagTZ has been defined in dag.html
assumes that ti.start_date and ti.end_date are always available. While they are none, it falls back to "default" value which is current time.

However I assume this is no longer an issue/bug if this PR is merged?

@ashb
Copy link
Member

ashb commented May 1, 2020

Probably worth another pr to fix that.

(This was a bug I introduced in my UI tz work - moment.utc(unrefined) returns now, we didn't used to call moment at all.)

@XD-DENG
Copy link
Member Author

XD-DENG commented May 2, 2020

Hi @kaxil mind taking a look when you have time? Need to merge this before I proceed to merge #8667 (which has been approved already). Thanks :)

@XD-DENG XD-DENG merged commit d92e848 into apache:master May 2, 2020
@XD-DENG XD-DENG deleted the feature/fix-date-for-dummyoperator branch May 2, 2020 19:13
kaxil pushed a commit that referenced this pull request Jun 28, 2020
…8663)

Otherwise the behaviour in UI is incorrect
Addressing issue #8662

(cherry picked from commit d92e848)
potiuk pushed a commit that referenced this pull request Jun 29, 2020
…8663)

Otherwise the behaviour in UI is incorrect
Addressing issue #8662

(cherry picked from commit d92e848)
@kaxil kaxil added the type:improvement Changelog: Improvements label Jul 1, 2020
kaxil pushed a commit that referenced this pull request Jul 1, 2020
…8663)

Otherwise the behaviour in UI is incorrect
Addressing issue #8662

(cherry picked from commit d92e848)
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
…pache#8663)

Otherwise the behaviour in UI is incorrect
Addressing issue apache#8662

(cherry picked from commit d92e848)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants