Refactor AWS credentials to support credential expiry and greater modularity

[sjperkins]

In #115 (review), the following code base was pointed out: https://github.com/aws/aws-sdk-go/tree/main/aws/credentials.

I've refactored the AWS credentials to generally follow the referenced patterns and return a CachedCredentialProvider which in turn proxies other CredentialProviders. See the following Class diagram:

classDiagram
   direction UD
   class AwsCredentialProvider {
     <<Abstract>>
     +GetCredentials() ResultAwsCredentials
     +ExpiresAt() ResultTime
     +IsExpired() bool
   }
   class ExpiryCredentialProvider {

     +Time expiration_
     +Function clock_
     +SetExpiration(Time, Duration) 
   }

   AwsCredentialProvider <|-- ExpiryCredentialProvider
   AwsCredentialProvider <|-- FileCredentialProvider
   AwsCredentialProvider <|-- EnvironmentCredentialProvider
   AwsCredentialProvider <|-- ChainedCredentialProvider
   ExpiryCredentialProvider <|-- EC2CredentialProvider
   AwsCredentialProvider <|-- CachedCredentialProvider
   EnvironmentCredentialProvider  *--  ChainedCredentialProvider
   FileCredentialProvider  *--  ChainedCredentialProvider
   EC2CredentialProvider *-- ChainedCredentialProvider
   ChainedCredentialProvider *-- CachedCredentialProvider
   note for ExpiryCredentialProvider "Implements \nexpiry with a \nTime Variable and\nClock function"
   note for EnvironmentCredentialProvider "Once retrieved,\nnever expires"
   note for FileCredentialProvider "Once retrieved,\nnever expires"
   note for ChainedCredentialProvider  "Iteratively retrieves credentials\nfrom a list of providers\nIsExpiry + ExpiresAt proxies\nprovider which last retrieved creds"
   note for CachedCredentialProvider "Caches Credentials\nRetrieves new credentials on expiry\nGuards encapsulated Provider with a mutex"

Loading

Then

• CachedCredentialProvider: Credential retrieval protected by a mutex and credentials cached until they expire
  • ChainCredentialProvider: Retrieves the first valid credentials form a list of credentials
    • EnvironmentCredentialProvider (never expires)
    • FileCredentialProvider (never expires)
    • EC2MetadataCredentialProvider (time based expiry, inherits from ExpiryCredentialProvider)

In particular, the ExpiryCredentialProvider implements a clock based expiry mechanism that will primarily be used with the EC2Metadata server.

Other than that,

• each provider is split into it's own header, source and test case
• Credential provider code has been moved into the tensorstore/kvstore/s3/credentials folder.
• Manual IWYU has been applied.

#115 will depend on this, I can merge this code into that PR if you'd prefer to review the changes that way.

[laramiel]

Is this intended to replace #115 or is it more or less part of the changeset where #115 gets submitted first? NM, you suggest this happens first. Ok, I'll switch and look at it next.

My comment there was more for reference since I couldn't find the documentation to some of the AWS types; the go code seemed like the easier code to read to discern aws intent...

I'm not sure whether decomposing the responsibilities is very useful when there is really only a single user/implementation of most of these(but I may bias towards functions where others don't).

[sjperkins]

Maybe running the clangd language server will give you better suggestions than iwyu... iwyu certainly has some misses. In general you can drop an iwyu-included header if none of the types declared in the header are directly spelled in the source files. Unless the file is adl-used...

I installed:

• clangd 17
• https://github.com/hedronvision/bazel-compile-commands-extractor to create the compile_commands.json

and configured clangd as follows:

If:
  PathMatch: tensorstore/kvstore/s3/*
Diagnostics:
  UnusedIncludes: Strict
  MissingIncludes: Strict

but, while it did seem to be fairly good at spotting unused includes, it didn't seem to detect missing includes all that well. I've applied best effort here but its still been a very manual process and I don't feel confident about it. I may need to spend more time working on configuring it, maybe on the weekend.

[laramiel]

Can you update the change description to better reflect the current state? I'll import and see what else I need; maybe only the updated description.

[sjperkins]

Can you update the change description to better reflect the current state?

I've done so.

[laramiel]

Ok, it looks like that didn't squash properly. Well next time. :)

[sjperkins]

Ok, it looks like that didn't squash properly

Maybe because there wasn't a merge from master for this commit daed8c2.

Squashing the PR into one commit may be a viable future strategy, although doing so loses the PR history.