Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wait for all Sessions during pantsd shutdown (cherrypick of #11929) #11934

Merged
merged 1 commit into from
Apr 16, 2021

Conversation

stuhood
Copy link
Member

@stuhood stuhood commented Apr 16, 2021

As described in #11618, when pantsd intentionally exits due to low memory, a few types of work can be cut short:

  1. if the run ends in Ctrl+C, processes that were cancelled may not have had time to be dropped before pantsd exits.
  2. async StreamingWorkunitHandler threads might still be running.

This change adds orderly-shutdown mechanisms to the Scheduler/Core to join all ongoing Sessions (including the SWH), and improves tests to ensure that the SWH is waited for.

Additionally, in the last commit, added purging of the pantsd metadata as soon as we decide to restart, which should reduce (but probably not eliminate) the incidence of item 1. from #11618. Work for #11831 will likely further harden this path.

[ci skip-build-wheels]

As described in pantsbuild#11618, when `pantsd` intentionally exits due to low memory, a few types of work can be cut short:
1. if the run ends in Ctrl+C, processes that were cancelled may not have had time to be dropped before `pantsd exits.
2. async StreamingWorkunitHandler threads might still be running.

This change adds orderly-shutdown mechanisms to the `Scheduler`/`Core` to join all ongoing `Sessions` (including the SWH), and improves tests to ensure that the SWH is waited for.

Additionally, in the last commit, added purging of the `pantsd` metadata as soon as we decide to restart, which should reduce (but probably not eliminate) the incidence of item 1. from pantsbuild#11618. Work for pantsbuild#11831 will likely further harden this path.

[ci skip-build-wheels]
@stuhood stuhood merged commit e3b1b4d into pantsbuild:2.4.x Apr 16, 2021
@stuhood stuhood deleted the stuhood/11929-for-2.4.x branch April 16, 2021 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants