STS: AssumeRole, https response error StatusCode: 403 #2567
Assignees
Labels
bug
This issue is a bug.
needs-reproduction
This issue needs reproduction.
response-requested
Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Comments
stgleb
added
bug
go.mod |
|
@RanVaknin FYI, MFA is disabled. |
|
Hi @stgleb , Thanks for reaching out and for all the code.
By comparing the request and response logs we can identify any discrepancies between the two states (working and broken) and would help us identify the offending part of the request that might point to a recent change. Thanks, |
RanVaknin
added
response-requested
|
@RanVaknin So far this is logs for current version, so looks like token is issued, but then is invalid. |
|
|
Hi @stgleb , Thanks for the logs. From what I can see in both logs provided the Assume role itself succeeds, but I don't see the requests made with those temporary credentials to EC2, which are the one in question. Are you able to provide those? Thanks again, |
RanVaknin
added
response-requested
|
yeah, I've also noticed that. I'll grab then DescribeRegions logs as well. |
github-actions
bot
removed
the
response-requested
|
Request AssumeRole Response AssumeRole Describe cluster |
1 task
|
@stgleb -- I'm having trouble making sense of these request logs. The underlying cause of your originally reported error is a failed call to STS AssumeRole: Your request logs here show a failed AssumeRole, with the error you first reported, against the version you reported broken. But your log here shows a successful AssumeRole, against the version you reported broken. Note that the outer DescribeRegions context is largely immaterial. You're hitting an error on credentials retrieval, which would be triggered by calling any operation if credentials had not yet been retrieved or were previously cached and expired. The only difference I'm seeing between these two requests is the role ARN. So, followup questions--
|
lucix-aws
added
the
response-requested
|
yes, issue is still there with following error message: we create roles per each run, so it is consistently happens with any role. At appears after upgrading aws-sdk-go-v2 versions. |
|
We have to break this down further. The following code recreates an package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
"github.com/aws/aws-sdk-go-v2/service/sts"
)
const (
region = "..."
roleARN = "..."
akid = "..."
secret = "..."
session = "..."
)
var externalID = "..." // clusterID in your code
func main() {
cfg, err := config.LoadDefaultConfig(
context.Background(),
config.WithRegion(region),
config.WithCredentialsProvider(
credentials.NewStaticCredentialsProvider(akid, secret, session),
),
)
if err != nil {
panic(err)
}
sts := sts.NewFromConfig(cfg)
provider := stscreds.NewAssumeRoleProvider(sts, roleARN, func(o *stscreds.AssumeRoleOptions) {
o.ExternalID = &externalID
})
creds, err := provider.Retrieve(context.Background())
if err != nil {
panic(err)
}
fmt.Println(creds)
}Do the following:
and let us know the results |
lucix-aws
added
needs-reproduction
github-actions
bot
removed
the
response-requested
lucix-aws
added
the
response-requested
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
|
@lucix-aws I can confirm that you introduced somehow breaking changes in your SDK. It doesn't retry config.WithRetryer(func() aws.Retryer {
r := retry.NewStandard(func(opts *retry.StandardOptions) {
opts.MaxAttempts = 5
opts.MaxBackoff = 10 * time.Second
})
return NewCustomRetryer(r)
}),
// custom retrier code
if v.ErrorCode() == "InvalidClientTokenId" {
return aws.TrueTernary
} |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Acknowledgements
go get -u github.com/aws/aws-sdk-go-v2/...)Describe the bug
After updating aws dependencies I have following error on DescribeRegions:
Expected Behavior
Expected 200 Ok
Current Behavior
operation error EC2: DescribeRegions, get identity: get credentials: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 9b051175-e056-4572-bbdd-1fa1990e5b99, api error InvalidClientTokenId: The security token included in the request is invalid.
Reproduction Steps
Possible Solution
No response
Additional Information/Context
Code:
sts code:
When error happens:
AWS Go SDK V2 Module Versions Used
current dependencies
Before
Compiler and Version used
go version go1.22.1 darwin/arm64
Operating System and version
macOS
The text was updated successfully, but these errors were encountered: