Not able to get IAM credentials with IMDSv1

[friedrichg]

Describe the bug
Cortex is not able get IAM credentials with IMDSv1 auth

To Reproduce
Steps to reproduce the behavior:

1. Ensure http://169.254.169.254/latest/api/token is not accessible for pods. (For example: using kube2iam)
2. Start container with IAM role with Cortex e2cfc51

Click to expand!

$ /bin/cortex -blocks-storage.backend=s3 -blocks-storage.s3.bucket-name=bucket --target compactor --log.level=debug -blocks-storage.s3.endpoint=s3.dualstack.eu-west-1.amazonaws.com
level=info ts=2022-10-05T20:31:27.500222976Z caller=main.go:194 msg="Starting Cortex" version="(version=1.13.0, branch=master, revision=e2cfc51)"
level=info ts=2022-10-05T20:31:27.500504103Z caller=server.go:306 http=[::]:9009 grpc=[::]:9008 msg="server listening on addresses"
level=debug ts=2022-10-05T20:31:27.500922031Z caller=api.go:141 msg="api: registering route" methods=GET path=/config auth=false
level=debug ts=2022-10-05T20:31:27.501010322Z caller=api.go:141 msg="api: registering route" methods=GET path=/ auth=false
level=debug ts=2022-10-05T20:31:27.501028576Z caller=api.go:141 msg="api: registering route" methods=GET path=/debug/fgprof auth=false
level=debug ts=2022-10-05T20:31:27.501092732Z caller=api.go:141 msg="api: registering route" methods=GET path=/memberlist auth=false
level=debug ts=2022-10-05T20:31:27.501181221Z caller=api.go:141 msg="api: registering route" methods=GET,POST path=/compactor/ring auth=false
level=debug ts=2022-10-05T20:31:27.501209338Z caller=api.go:141 msg="api: registering route" methods=GET path=/services auth=false
level=debug ts=2022-10-05T20:31:27.501263744Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=memberlist-kv
level=info ts=2022-10-05T20:31:27.501381946Z caller=module_service.go:64 msg=initialising module=server
level=debug ts=2022-10-05T20:31:27.501487025Z caller=module_service.go:54 msg="module waiting for initialization" module=memberlist-kv waiting_for=server
level=info ts=2022-10-05T20:31:27.501624576Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=debug ts=2022-10-05T20:31:27.501672565Z caller=module_service.go:54 msg="module waiting for initialization" module=compactor waiting_for=server
level=info ts=2022-10-05T20:31:27.501683884Z caller=module_service.go:64 msg=initialising module=compactor
level=info ts=2022-10-05T20:31:27.501896465Z caller=blocks_cleaner.go:144 component=cleaner msg="started blocks cleanup and maintenance"
level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"
level=info ts=2022-10-05T20:31:27.540381948Z caller=compactor.go:621 component=compactor msg="discovering users from bucket"
level=info ts=2022-10-05T20:31:27.540545681Z caller=cortex.go:422 msg="Cortex started"

3. Get access denied from s3

level=error ts=2022-10-05T20:31:27.540353246Z caller=blocks_cleaner.go:155 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to discover users from bucket: Access Denied"

Expected behavior
it should continue without problem

Environment:

• Infrastructure: Kubernetes
• Deployment tool: jsonnet

Additional Context
It works in the previous commit f550642

Previous

cortex/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go

Lines 291 to 298 in f550642

	// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
	token, _ := fetchIMDSToken(client, endpoint)
	// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
	u, err := getIAMRoleURL(endpoint)
	if err != nil {
	return ec2RoleCredRespBody{}, err
	}

Current

cortex/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go

Lines 291 to 301 in e2cfc51

	// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
	token, err := fetchIMDSToken(client, endpoint)
	if err != nil {
	return ec2RoleCredRespBody{}, err
	}
	// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
	u, err := getIAMRoleURL(endpoint)
	if err != nil {
	return ec2RoleCredRespBody{}, err
	}

This bugfix minio/minio-go#1682 is most likely the culprit

[alvinlin123]

@friedrichg I am curious about the use of disabling the IMDv2's session API? The reason I am asking is I want to make a case to have the minio revert/fix the bug fix :)

[friedrichg]

kube2iam does not support the session API jtblin/kube2iam#282

There is no other reason.

I also think the bugfix in minio-go is probably correct for minio but not for AWS.

[alvinlin123]

@friedrichg this is resolved with the PR right?

[friedrichg]

yes, Thanks!