Make identity store loading and alias merging deterministic #28867
+362
−48
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
To optimize loading entities into the IdentityStore during unseal we current load the 256 buckets from storage in parallel. We then process them in whatever order they load. This determinism should be fine because each entity should be loaded by ID and so the loading order shouldn't matter.
But in real-life historical (and potentially current) bugs can cause there to be duplicated aliases in storage. We have for many years attempted to cleanup and merge such problematic duplicate aliases on load, however we always merge them in the order they are encountered during loading. Because this is non-deterministic, it means that different aliases can "win" this merge process after different unseals causing unpredictable behaviour. It will often also mean that Enterprise Performance Standbys may end up with a different view of the entities than the active node causing inconsistent results depending on which node responds to a request.
This PR retains the parallel loading optimization but fixes the order of processing loaded buckets to ensure that all nodes will resolve any duplicates identically.
We've reviewed this in the Enterprise PR extensively and performed performance testing that shows that even though there is a theoretical worst-case that might make this new approach slower (say if the first bucket takes a long time to load), in practice it's not measurably different (mainly because the current code doesn't realise ideal parallelism anyway due to contention in other layers of storage).
This should fix issues where duplicates (caused by other bugs) then cause inconsistent responses from different servers.
JIRA: VAULT-31384
Ent PR: https://github.com/hashicorp/vault-enterprise/pull/6776
RFC: https://docs.google.com/document/d/16Tbsngmzg9tuJu1G8s1uSJGDvp-YS5UUUVVboDvuQpc/edit?tab=t.0
TODO only if you're a HashiCorp employee
to N, N-1, and N-2, using the
backport/ent/x.x.x+ent
labels. If this PR is in the CE repo, you should only backport to N, using thebackport/x.x.x
label, not the enterprise labels.of a public function, even if that change is in a CE file, double check that
applying the patch for this PR to the ENT repo and running tests doesn't
break any tests. Sometimes ENT only tests rely on public functions in CE
files.
in the PR description, commit message, or branch name.
description. Also, make sure the changelog is in this PR, not in your ENT PR.