Keep secret metadata list between pollings #144
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is changing and why?
The secret-syncer keeps a list of secret metadata that is uses for fetching secrets from the cache, and for filtering. Previously this list was overwritten on every poling round. The problem with this is that if fetching this data is interrupted or incomplete, we lose our "previous knowledge" of what secrets are out there, and especially in case of filtered lists, this can lead to incomplete secret generation.
This change modifies the bookkeeping on this secret metadata list: on every polling round we just update a list that is otherwise kept in memory. This way if a polling round fails or fails short, we can still have knowledge of what secrets existed in SecretsManager a few polling rounds ago.
What's up with deleted secrets?
When the secret metadata list was updated on every polling round, secrets that got deleted were simply no longer appeared in the overwritten list, because the AWS SecretsManager API doesn't list deleted secret as a default. With "memory" introduced, we need to start keep track of secrets that are deleted, otherwise they will be hanging around in the list forever as fresh secrets.
To address this, we are now fetching the deleted secrets too from AWS SecretsManager, and actively removing them from the local secret metadata list, if we find them. This way the local metadata list only contains active secrets, and it will not grow longer than the list of active AWS secrets.
What happens if a deleted secret is referenced?
The same as before this change: the secrets ID is matched against the local metadata list; since it is not found (because deleted secrets are removed), k8s-secret-syncer makes a request straight to the AWS SecretsManager API to get the secret. On this request we receive and log an error message about the secret not being available; and the Kubernetes Secret is not updated.
The only difference is if the polling fails for a few cycles right after the secret is deleted from AWS. Before this the secret would be missing from the local metadata list, and as a reaction to that requested straight from the AWS SecretsManager API, where the request would fail. Now the already cached value would be used (with no update to the Kubernetes Secret) until the polling resumes and we get note of the secret removal.