(cli): source_profile + credential_process not respected

[rittneje]

Describe the bug

We configured our profile like so:

aws configure --profile base set credential_process cred-process.sh
aws configure --profile default set source_profile base
aws configure --profile default set role_arn arn:aws:iam::<redacted>:role/<redacted>

The AWS CLI works perfectly fine with this setup. However, CDK throws a nonsense exception.

Error: Need to perform AWS calls for account <redacted>, but no credentials have been configured
at SdkProvider.forEnvironment (/usr/local/lib/node_modules/aws-cdk/lib/api/aws-auth/sdk-provider.ts:179:46)
at CloudFormationDeployments.prepareSdkFor (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:437:22)
at CloudFormationDeployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:333:70)
at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
at initCommandLine (/usr/local/lib/node_modules/aws-cdk/lib/cli.ts:341:12)

Expected Behavior

It should work without issue.

Current Behavior

See above.

Reproduction Steps

See above.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.28.0 (build ba233f0)

Framework Version

No response

Node.js Version

v16.15.1

OS

Alpine 3.16

Language

Python

Language Version

3.10.5

Other information

No response

[rix0rrr]

Please run the command again with -v (cdk deploy -v) and paste the output.

The contents of ~/.aws/credentials and ~/.aws/config would also help.

[rittneje]

cdk deploy -v

Click to expand

CDK toolkit version: 2.28.0 (build ba233f0)
Command line arguments: {
  _: [ 'deploy' ],
  app: 'REDACTED.out',
  a: 'REDACTED.out',
  v: 1,
  verbose: 1,
  home: '/.cdk',
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: true,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  logs: true,
  '$0': '/usr/local/bin/cdk',
  STACKS: [ '*' ],
  'S-t-a-c-k-s': [ '*' ]
}
cdk.json: {
  "app": "python3 REDACTED.py",
  "requireApproval": "never"
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'REDACTED.out',
  requireApproval: 'never',
  context: {},
  debug: false,
  assetMetadata: true,
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
Resolving default credentials
Unable to determine the default AWS account: ProcessCredentialsProviderFailure: Profile default did not include credential process
    at ProcessCredentials2.load (/usr/local/lib/node_modules/aws-sdk/lib/credentials/process_credentials.js:102:11)
    at ProcessCredentials2.coalesceRefresh (/usr/local/lib/node_modules/aws-sdk/lib/credentials.js:205:12)
    at ProcessCredentials2.refresh (/usr/local/lib/node_modules/aws-sdk/lib/credentials/process_credentials.js:163:10)
    at ProcessCredentials2.get2 [as get] (/usr/local/lib/node_modules/aws-sdk/lib/credentials.js:122:12)
    at resolveNext2 (/usr/local/lib/node_modules/aws-sdk/lib/credentials/credential_provider_chain.js:125:17)
    at /usr/local/lib/node_modules/aws-sdk/lib/credentials/credential_provider_chain.js:126:13
    at /usr/local/lib/node_modules/aws-sdk/lib/credentials.js:124:23
    at /usr/local/lib/node_modules/aws-sdk/lib/credentials.js:212:15
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 'ProcessCredentialsProviderFailure',
  time: 2022-07-07T12:27:28.900Z
}
context: {
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
--app points to a cloud assembly, so we bypass synth

✨  Synthesis time: 0.03s

REDACTED: deploying...

 ❌  REDACTED failed: Error: Need to perform AWS calls for account REDACTED, but no credentials have been configured
    at SdkProvider.forEnvironment (/usr/local/lib/node_modules/aws-cdk/lib/api/aws-auth/sdk-provider.ts:179:46)
    at CloudFormationDeployments.prepareSdkFor (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:437:22)
    at CloudFormationDeployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:333:70)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/lib/cli.ts:341:12)
Could not refresh notices: Error: Client network socket disconnected before secure TLS connection was established
Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/.cdk/cache/notices.json'
Could not refresh notices: Error: Client network socket disconnected before secure TLS connection was established
Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/.cdk/cache/notices.json'

Need to perform AWS calls for account REDACTED, but no credentials have been configured
Error: Need to perform AWS calls for account REDACTED, but no credentials have been configured
    at SdkProvider.forEnvironment (/usr/local/lib/node_modules/aws-cdk/lib/api/aws-auth/sdk-provider.ts:179:46)
    at CloudFormationDeployments.prepareSdkFor (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:437:22)
    at CloudFormationDeployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:333:70)
    at CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/lib/cli.ts:341:12)

$AWS_CONFIG_FILE

[profile base]
credential_process = credential_process.sh
[default]
source_profile = base
role_arn = arn:aws:iam::REDACTED:role/REDACTED
role_session_name = REDACTED

The $AWS_SHARED_CREDENTIALS_FILE file does not exist.

[rittneje]

Based on that, there are two mysterious things.

1. Somehow the SDK is mixing up the two profiles and looking for credential_process under default instead of base.
2. There is no /usr/local/lib/node_modules/aws-sdk folder so I don't know what those logs are referring to.

[rix0rrr]

Unfortunately this is a bug we'll have to fix in AWS SDK JS v2.

[rittneje]

@rix0rrr Any update on this? Was a bug filed against the JS SDK?

[rittneje]

ping @rix0rrr

[rittneje]

@rix0rrr

[rittneje]

@rix0rrr @TheRealAmazonKendra When will this bug be fixed?

[pkit]

@rix0rrr in my case it happens when aws-sdk-v2 tries to use sso re-auth in the middle of some aws process.
Essentially botching the cache in ~/.aws/sso/cache/
Deleting the cache fixes the problem.

[karnauskas]

I think I have the same issue, setup without SSO.

$ cdk --version
2.92.0 (build bf62e55)
$ yarn info aws-sdk version
2.1437.0

No ~/.aws/credentials file; ~/.aws/config:

[profile AAAA]
credential_process=/opt/homebrew/bin/aws-vault exec --duration=15m --prompt=terminal -j AAAA
mfa_serial=arn:aws:iam::11..111:mfa/REDACTED
region=eu-west-2

[profile BBBB]
region=eu-west-2
role_arn=arn:aws:iam::22..222:role/REDACTED
source_profile=AAAA

cdk --profile BBBB bootstrap aws://22..222/eu-west-2 --debug
...
 ⏳  Bootstrapping environment aws://22..222/eu-west-2...
 ❌  Environment aws://22..222/eu-west-2 failed bootstrapping: Error: Need to perform AWS calls for account 22..222, but no credentials have been configured
...

CDK is not picking up the right profile, not asking for mfa. All good with aws cli.

[ThomasEg]

Can confirm this issue still exists. I'm trying to avoid having credentials in clear-text and I'm using credential_process to achieve this together with 1password. It works with aws-cli, but not CDK.

Sample credentials-file with both hardcoded credentials and credential_process for testing:

[base]
region = eu-central-1
;aws_access_key_id = [redacted]
;aws_secret_access_key=[redacted]
credential_process = [redacted]

[subaccount]
region=eu-central-1
role_arn=arn:aws:iam::123456789123:role/OrganizationAccountAccessRole
source_profile=base

Note: Hardcoded credentials are not in use in above due to ;, but it makes it easy to switch between the two approaches during testing. Also note that the credential_process returns the exact same secret id and secret key as entered in the hardcoded section. In my use case i use 1password to store the credentials and the command in credential_process retrieves the credentials from the 1password vault. No config-file is used in this setup.

Testing of CLI with:

aws s3 ls --profile=subaccount

...yields the exact same result with either hardcoded credentials or with credential_process enabled. The result is a correct list of available buckets on the account. This is as expected.

Testing of CDK with:

npx cdk diff -v --profile=subaccount

...works as expected with hardcoded credentials. It returns a proper diff for the stack and is as expected. But with credential_process enabled it gives the following error:

Could not assume arn:aws:iam::123456789123:role/cdk-xxxxxxxxxx-lookup-role-123456789123-eu-central-1, proceeding anyway.
[10:38:33] Reading cached notices from C:\Users\name\.cdk\cache\notices.json

Need to perform AWS calls for account 123456789123, but no credentials have been configured
[10:38:33] Error: Need to perform AWS calls for account 123456789123, but no credentials have been configured
    at SdkProvider.forEnvironment (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:407:659633)
    at async Deployments.cachedSdkForEnvironment (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:449:12570)
    at async Deployments.prepareSdkFor (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:449:8085)
    at async Deployments.readCurrentTemplateWithNestedStacks (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:449:4307)
    at async CdkToolkit.diff (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:449:194385)
    at async exec4 (C:\Users\name\Documents\GitHub\someproject\node_modules\aws-cdk\lib\index.js:504:54331)

... it should return the exact same result in both scenarios...

A fix of the bug is preferred, but if this isn't happening then if anyone has a workaround I'll be happy to hear it(and likely others hitting the same issue in the future)...

System info used for the above tests:
OS: Windows 11, (v10.0.22631)
CDK_CLI: 2.152.0 (build faa7d79)
AWS_CLI: 2.17.17
NodeJS: 20.16.0 / NPM: 10.8.1

[rix0rrr]

Since this functionality is provided by the underlying SDK that we use, we are bound by it.

We have plans to migrate CDK to AWS SDK v3. After that, we will re-evalutate this issue.

[TheRealAmazonKendra]

Our credential chain is defined here (there's more below but none of those are set in these use cases so we can pretend they don't exist for the purposes of this explanation). So, we're basically checking the following sources in the following order:

• EnvironmentCredentials
• profileCredentials (this is a patched version of SharedIniFileCredentials)
• SsoCredentials
• ProcessCredentials

Environment Credentials are not set so that link in the credential chain fails and it moves on to the next source.

In the AWS SDK v2, and our patched version that is based off of it, only source_profile and credential_source is supported. In both the use cases listed in this issue, source_profile is set, but that source profile points to a profile that uses credential_process. Neither the SDK's SharedIniFileCredentials nor our PatchedSharedIniFileCredentials supports credential_process so that link in the credential chain fails.

SSO Credentials are not set so that link in the credential chain fails and it moves on to the next source.

ProcessCredentials does support credential_process but only when it's set either in the ~/.aws/credentials file or in the file defined by AWS_SHARED_CREDENTIALS_FILE. In the cases above, it was only present in the config file, and the profile that it is under is not the profile that the credential chain is searching for.

The credential chain provided by the AWS SDK v3 does support this setup so this issue should be resolved when we merge and release #31702.