Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-1396] Properly cleanup DAGScheduler on job cancellation. #305

Closed
wants to merge 2 commits into from
Closed

[SPARK-1396] Properly cleanup DAGScheduler on job cancellation. #305

wants to merge 2 commits into from

Commits on Apr 7, 2014

  1. Properly cleanup DAGScheduler on job cancellation. 

    Previously, when jobs were cancelled, not all of the state in the
DAGScheduler was cleaned up, leading to a slow memory leak in the
DAGScheduler.  As we expose easier ways ot cancel jobs, it's more
important to fix these issues.

This commit adds 3 tests.  “run shuffle with map stage failure” is
a new test to more thoroughly test this functionality, and passes on
both the old and new versions of the code.  “trivial job
cancellation” fails on the old code because all state wasn’t cleaned
up correctly when jobs were cancelled (we didn’t remove the job from
resultStageToJob).  “failure of stage used by two jobs” fails on the
old code because taskScheduler.cancelTasks wasn’t called for one of
the stages (see test comments).
    @kayousterhout
    kayousterhout committed Apr 7, 2014
    Configuration menu
    Copy the full SHA
    9217080 View commit details
    Browse the repository at this point in the history

  2. Mark review comments

    @kayousterhout
    kayousterhout committed Apr 7, 2014
    Configuration menu
    Copy the full SHA
    f33d844 View commit details
    Browse the repository at this point in the history