-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WaitForState() leaves Refresh() running in the background on timeout #530
Comments
I believe I'm seeing this issue as well in the context of the AWS provider (v3.75.2, which I understand uses terraform-plugin-sdk v2.10.1) creating an EC2 instance:
From this Terraform run we saw that two EC2 instances were created in our AWS console. 😮 I think that indicates that an EC2 instance creation request was still running when the WaitForState timeout occurred and then a second EC2 instance creation request was submitted, and then both of those requests ultimately succeeded. I'm hoping that #828 (which I understand copies #529 but fixes the unit tests) can be reviewed soon. |
SDK version
Relevant provider source code
Taken from: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_iam_role.go#L152
Debug Output
[DEBUG] [aws-sdk-go] DEBUG: Request iam/CreateRole Details:
[DEBUG] [aws-sdk-go] DEBUG: Send Request iam/CreateRole failed, attempt 0/25, error RequestError: send request failed
[DEBUG] [aws-sdk-go] DEBUG: Retrying Request iam/CreateRole, attempt 1
[DEBUG] [aws-sdk-go] DEBUG: Request iam/CreateRole Details:
[WARN] WaitForState timeout after 30s
[WARN] WaitForState starting 30s refresh grace period
[DEBUG] [aws-sdk-go] DEBUG: Send Request iam/CreateRole failed, attempt 1/25, error RequestError: send request failed
[DEBUG] [aws-sdk-go] DEBUG: Retrying Request iam/CreateRole, attempt 2
[DEBUG] [aws-sdk-go] DEBUG: Request iam/CreateRole Details:
[ERROR] WaitForState exceeded refresh grace period
[DEBUG] [aws-sdk-go] DEBUG: Request iam/CreateRole Details:
Expected Behavior
resource.Retry() blocks until callback finishes - even on timeout
Actual Behavior
Callback is still running when timeout happens. For example, AWS provider issues another, parallel, CreateRole(). In many cases this results double creation attempt, and eventually a failure in the plugin.
Error: Error creating IAM Role hello-world-ssm_role: EntityAlreadyExists: Role with name hello-world-ssm_role already exists.
status code: 409, request id: removed
on main.tf line 18, in resource "aws_iam_role" "ssm_role":
18: resource "aws_iam_role" "ssm_role"
References
We have been running a crude patch ( #529 ) in production for a few weeks with good results.
The text was updated successfully, but these errors were encountered: