Token expired errors with webIdentityToken, when the token is valid

[automagic]

What happened?

Using a webIdentityToken obtained from Vault, pulumi-aws is unable to assume the roll, getting a '403 ExpiredToken' error from AWS. However, when the same token is used via CLI aws sts assume-role-with-web-identity the credentials are returned.

Example

I tried using the token in the environment as follows:

pulumiConfig:
    aws:assumeRoleWithWebIdentity:
      roleArn: arn:aws:iam::999999999:role/foo
      webIdentityToken: ${vault.secrets.identity.token}

But it is failing to retrieve the token:
Details: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 83805f40-faa6-4c05-8ff0-883cbaf478bb, api error ExpiredToken: The security token included in the request is expired

However, I can assume the role using the cli:

aws sts assume-role-with-web-identity --role-arn arn:aws:iam::999999999:role/foo --role-session-name test-session --web-identity-token $(pulumi config get --path aws:assumeRoleWithWebIdentity.webIdentityToken)

Output of pulumi about

CLI
Version 3.113.3
Go Version go1.22.2
Go Compiler gc

Host
OS darwin
Version 13.5.1
Arch arm64

Backend
Name pulumi.com
URL https://app.pulumi.com/cdoan
User cdoan
Organizations cdoan, nvidia
Token type personal

Additional context

The token is being obtained from HashiCorp Vault via ESC and passed to the provider in pulumiConfig

Also tried using webIdenityTokenFile with the same result.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[corymhall]

@automagic do you know what command they are running when this fails? There is this open issue where credentials are cached during refresh & destroy pulumi/esc#199

[mikhailshilkov]

@automagic Could you please respond when you have a minute?

[mjeffryes]

closing as probable duplicate of pulumi/esc#199