Add global counters, and deprecate per-workunit counters #14541
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…reamingWorkunitContext`. # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
stuhood
commented
Feb 20, 2022
Comment on lines
+809
to
+815
| // TODO: Temporarily attaching the global counters to the "root" workunit. Callers should | ||
| // switch to consuming `StreamingWorkunitContext.get_metrics`. | ||
| // Remove this deprecation after 2.14.0.dev0. | ||
| if workunit.parent_id.is_none() { | ||
| let mut metrics = workunit_store.get_metrics(); | ||
|
|
||
| metrics.insert("DEPRECATED_ConsumeGlobalCountersInstead", 0); |
There was a problem hiding this comment.
our backend don't expect any specific counters. but if counters name change we might need to update an internal metrics dashboard created by @Eric-Arellano
There was a problem hiding this comment.
looking at buildsense it seems that we no longer get metrics with this change.
so I think the proper way to make this change is to tweak our backend to work with the new counters (might need a new version of the tc pants plugin) and only then merge this.
otherwise, we will loose pants metrics which we do rely on for some stuff.
There was a problem hiding this comment.
our backend don't expect any specific counters. but if counters name change we might need to update an internal metrics dashboard created by @Eric-Arellano
actually not true. I forgot about some calculations we do based on those.
asherf
approved these changes
Feb 20, 2022
Eric-Arellano
approved these changes
Feb 20, 2022
alonsodomin
pushed a commit
to alonsodomin/pants
that referenced
this pull request
Feb 25, 2022
…#14541) In order to unblock optionally recording workunits without losing counter accuracy (in pantsbuild#13483), move to recording counters globally via the `StreamingWorkunitContext`. [ci skip-build-wheels]
stuhood
added a commit
that referenced
this pull request
Apr 11, 2022
…e children (#15088) #15080 was caused by two factors: 1. #14541 made counters global via a new API, and attached them (as deprecated) to "the root workunit" (with "has no parent id" as the heuristic that something was the root workunit). 2. #14856 moved to calculating the parent(s) of a node based on a running graph of workunits (to allow #14680 to eventually add multiple parents), which meant that when nodes completed out of order, we might not have any parents for them. Together: this meant that when workunits completed asynchronously, we might not have parents for them, and because of the deprecation, we would attach the counters to multiple workunits. A consumer which was aggregating the counters would end up with an inaccurate total. Fixes #15080.
stuhood
added a commit
to stuhood/pants
that referenced
this pull request
Apr 11, 2022
…e children (pantsbuild#15088) 1. pantsbuild#14541 made counters global via a new API, and attached them (as deprecated) to "the root workunit" (with "has no parent id" as the heuristic that something was the root workunit). 2. pantsbuild#14856 moved to calculating the parent(s) of a node based on a running graph of workunits (to allow pantsbuild#14680 to eventually add multiple parents), which meant that when nodes completed out of order, we might not have any parents for them. Together: this meant that when workunits completed asynchronously, we might not have parents for them, and because of the deprecation, we would attach the counters to multiple workunits. A consumer which was aggregating the counters would end up with an inaccurate total. Fixes pantsbuild#15080.
stuhood
added a commit
that referenced
this pull request
Apr 11, 2022
…e children (cherrypick of #15088) (#15103) #15080 was caused by two factors: 1. #14541 made counters global via a new API, and attached them (as deprecated) to "the root workunit" (with "has no parent id" as the heuristic that something was the root workunit). 2. #14856 moved to calculating the parent(s) of a node based on a running graph of workunits (to allow #14680 to eventually add multiple parents), which meant that when nodes completed out of order, we might not have any parents for them. Together: this meant that when workunits completed asynchronously, we might not have parents for them, and because of the deprecation, we would attach the counters to multiple workunits. A consumer which was aggregating the counters would end up with an inaccurate total. Fixes #15080.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to unblock optionally recording workunits without losing counter accuracy (in #13483), move to recording counters globally via the
StreamingWorkunitContext.[ci skip-build-wheels]