[s3-repository] Support IAM roles for Kubernetes service accounts #81255
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arteam
force-pushed
the
amazon-web-identity-token
branch
20 times, most recently
from
4c753f2
to
199892a
Compare
arteam
changed the title
arteam
force-pushed
the
amazon-web-identity-token
branch
4 times, most recently
from
3923d93
to
db09282
Compare
arteam
added
:Distributed/Discovery-Plugins
tlrx
reviewed
Jan 17, 2022
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
Show resolved
Hide resolved
|
|
||
| public void shutdown() { | ||
| if (credentialsProvider != null) { | ||
| credentialsProvider.close(); |
There was a problem hiding this comment.
Maybe use IOUtils.close() here so that all objects are teared down but an exception is still retrhrown?
...les/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AwsS3ServiceImplTests.java
Show resolved
Hide resolved
|
@elasticmachine update branch |
tlrx
approved these changes
Jan 19, 2022
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
Outdated
Show resolved
Hide resolved
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
Outdated
Show resolved
Hide resolved
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java
Show resolved
Hide resolved
...les/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AwsS3ServiceImplTests.java
Show resolved
Hide resolved
...les/repository-s3/src/test/java/org/elasticsearch/repositories/s3/S3ClientSettingsTests.java
Show resolved
Hide resolved
|
@elasticmachine update branch |
|
@elasticmachine update branch |
|
Thank you very much, Tanguy! |
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Jan 19, 2022
…astic#81255) There have been many requests to support repository-s3 authentication via IAM roles in Kubernetes service accounts. The AWS SDK is supposed to support them out of the box with the aws-java-sdk-sts library. Unfortunately, we can't use WebIdentityTokenCredentialsProvider from the SDK. It reads the token from AWS_WEB_IDENTITY_TOKEN_FILE environment variable which is usually mounted to /var/run/secrets/eks.amazonaws.com/serviceaccount/token and the S3 repository doesn't have the read permission to read it. We don't want to hard-code a file permission for the repository, because the location of AWS_WEB_IDENTITY_TOKEN_FILE can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory. To overcome this limitation, this change adds a custom WebIdentityCredentials provider that reads the service account from a symlink to AWS_WEB_IDENTITY_TOKEN_FILE created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentification. Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the assume-role-with-web-identity call to the service and returns a response with test credentials. Fixes elastic#52625
💚 Backport successful
|
arteam
added a commit
that referenced
this pull request
Jan 19, 2022
…ts (#81255) (#82796) There have been many requests to support repository-s3 authentication via IAM roles in Kubernetes service accounts. The AWS SDK is supposed to support them out of the box with the aws-java-sdk-sts library. Unfortunately, we can't use WebIdentityTokenCredentialsProvider from the SDK. It reads the token from AWS_WEB_IDENTITY_TOKEN_FILE environment variable which is usually mounted to /var/run/secrets/eks.amazonaws.com/serviceaccount/token and the S3 repository doesn't have the read permission to read it. We don't want to hard-code a file permission for the repository, because the location of AWS_WEB_IDENTITY_TOKEN_FILE can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory. To overcome this limitation, this change adds a custom WebIdentityCredentials provider that reads the service account from a symlink to AWS_WEB_IDENTITY_TOKEN_FILE created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentification. Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the assume-role-with-web-identity call to the service and returns a response with test credentials. Fixes #52625
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There have been many requests to support
repository-s3authentication via IAM roles in Kubernetes service accounts.The AWS SDK is supposed to support them out of the box with the
aws-java-sdk-stslibrary. Unfortunately, we can't useWebIdentityTokenCredentialsProviderfrom the SDK. It reads the token fromAWS_WEB_IDENTITY_TOKEN_FILEenvironment variable which is usually mounted to/var/run/secrets/eks.amazonaws.com/serviceaccount/tokenand the S3 repository doesn't have the permissions to read it. We don't want to hard-code a file permission for the S3 repository, because the location ofAWS_WEB_IDENTITY_TOKEN_FILEcan change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory.To overcome this limitation, this change adds a custom
WebIdentityCredentialsprovider that reads the service account from a symlink toAWS_WEB_IDENTITY_TOKEN_FILEcreated in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentication like this:Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the
assume-role-with-web-identitycall to the service and returns a response with test credentials.Fixes #52625