-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data race in credentials. #3524
Comments
I'm interested to work on it. |
Thanks for reporting this issue @bitcodr. Could you provide a example of how the SDK is being configured to better help us investigate this issue? An example that reproduces this issue would also be very helpful. |
I think I've found the underlying issue here. aws-sdk-go/aws/credentials/credentials.go Lines 235 to 238 in fd796c5
aws-sdk-go/aws/credentials/credentials.go Lines 303 to 306 in fd796c5
There is no lock held during the call to
One possible implication of this is that the second thread might erroneously use the expiration time of the newer creds after fetching the older creds from the |
There is already an attempted fix in #3448. However, this fix has a problem - using a lock in this manner means that if there is an in-progress call to One way I think the issue can be mitigated is to first clear out the existing credentials after discovering they are expired, but before fetching new ones. That should prevent the second thread from seeing the old credentials with the new expiration. // The provider will record the expiration time internally without holding the write lock.
// However, we only fetch credentials while c.creds is the zero value (Value{}), and
// isExpired will only consult the provider if c.creds is NOT the zero value.
// Additionally, singleRetrieve sets c.creds while holding the write lock after the provider has returned.
// Thus holding the read lock here is sufficient for thread-safety.
c.m.RLock()
if curCreds := c.creds; !c.isExpired(curCreds) {
c.m.RUnlock()
return curCreds, nil
}
c.m.RUnlock()
...
func (c *Credentials) singleRetrieve(ctx Context) (creds interface{}, err error) {
// Another thread may have gotten new credentials between this thread releasing the read lock
// and acquiring the write lock. Double check the credentials are still expired to be safe.
c.m.Lock()
if curCreds := c.creds; !c.isExpired(curCreds) {
c.m.Unlock()
return curCreds, nil
}
c.creds = Value{}
c.m.Unlock()
...
if err == nil {
c.m.Lock()
c.creds = creds
c.m.Unlock()
}
} |
Updated PRhttps://github.com//pull/3448 to integrate your updates, and made few tweaks to the test. Once this is merged in it should ensure this race condition doesn't happen anymore. |
Fixed in #3448. Thanks for creating this bug request and PR contributing to its fix. We've merged in the change and it will be included in the SDK's next release. Let us know if you have any additional issues, or feedback. |
|
=== ### Service Client Updates * `service/cloudfront`: Updates service API and documentation * CloudFront adds support for managing the public keys for signed URLs and signed cookies directly in CloudFront (it no longer requires the AWS root account). * `service/ec2`: Updates service API and documentation * instance-storage-info nvmeSupport added to DescribeInstanceTypes API * `service/globalaccelerator`: Updates service API and documentation * `service/glue`: Updates service API and documentation * AWS Glue crawlers now support incremental crawls for the Amazon Simple Storage Service (Amazon S3) data source. * `service/kendra`: Updates service API and documentation * This release adds custom data sources: a new data source type that gives you full control of the documents added, modified or deleted during a data source sync while providing run history metrics. * `service/organizations`: Updates service documentation * AWS Organizations renamed the 'master account' to 'management account'. ### SDK Bugs * `aws/credentials`: Fixed a race condition checking if credentials are expired. ([#3448](#3448)) * Fixes [#3524](#3524) * `internal/ini`: Fixes ini file parsing for cases when Right Hand Value is missed in the last statement of the ini file ([#3596](#3596)) * related to [#2800](#2800)
Release v1.35.12 (2020-10-21) === ### Service Client Updates * `service/cloudfront`: Updates service API and documentation * CloudFront adds support for managing the public keys for signed URLs and signed cookies directly in CloudFront (it no longer requires the AWS root account). * `service/ec2`: Updates service API and documentation * instance-storage-info nvmeSupport added to DescribeInstanceTypes API * `service/globalaccelerator`: Updates service API and documentation * `service/glue`: Updates service API and documentation * AWS Glue crawlers now support incremental crawls for the Amazon Simple Storage Service (Amazon S3) data source. * `service/kendra`: Updates service API and documentation * This release adds custom data sources: a new data source type that gives you full control of the documents added, modified or deleted during a data source sync while providing run history metrics. * `service/organizations`: Updates service documentation * AWS Organizations renamed the 'master account' to 'management account'. ### SDK Bugs * `aws/credentials`: Fixed a race condition checking if credentials are expired. ([#3448](#3448)) * Fixes [#3524](#3524) * `internal/ini`: Fixes ini file parsing for cases when Right Hand Value is missed in the last statement of the ini file ([#3596](#3596)) * related to [#2800](#2800)
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug
I found some data race warnning of credentials in our test server.
Version of AWS SDK for Go?
v1.33.0
Version of Go (
go version
)?go1.15
To Reproduce (observed behavior)
There are multiple aws clients such like
sns.New(session.New())
in our code.Expected behavior
No data race.
It seems that the code in credentials is not thread safe?
aws-sdk-go/aws/credentials/credentials.go
Lines 175 to 189 in 4ca4815
The text was updated successfully, but these errors were encountered: