TF tries to destroy Lambda function which was manually deleted

[ZsoltPath]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.2
+ provider registry.terraform.io/hashicorp/aws v3.14.1

Affected Resource(s)

• aws_lambda_function

Debug Output

Can't upload a 60MB log to Gist.

Expected Behavior

I expect TF to discover that the resource is already deleted outside of TF and don't even list it as a task for the destroy.

Actual Behavior

TF lists the Lambda function as a resource to delete then the API call fails.

aws lambda get-function --function-name pr-112-mentoradvice-edge-lambda --region us-east-1

An error occurred (ResourceNotFoundException) when calling the GetFunction operation: Function not found: arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda

  # module.advice.module.lambda.aws_lambda_function.lambda will be destroyed
  - resource "aws_lambda_function" "lambda" {
      - arn                            = "arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda" -> null
      - filename                       = ".terraform/modules/advice.lambda/modules/compute/aws-lambda/builds/edge.js_a22ab202d5e1f86503550481a910fa8bc5ff87fd.zip" -> null
      - function_name                  = "pr-112-mentoradvice-edge-lambda" -> null
      - handler                        = "edge.handler" -> null
      - id                             = "pr-112-mentoradvice-edge-lambda" -> null
      - invoke_arn                     = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda/invocations" -> null
      - last_modified                  = "2020-10-28T21:24:23.144+0000" -> null
      - layers                         = [] -> null
      - memory_size                    = 128 -> null
      - publish                        = true -> null
      - qualified_arn                  = "arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda:1" -> null
      - reserved_concurrent_executions = -1 -> null
      - role                           = "arn:aws:iam::830138816992:role/pr-112-mentoradvice-edge-lambda" -> null
      - runtime                        = "nodejs10.x" -> null
      - source_code_hash               = "mOgNvBfn6kYoIkNyFi3ySP2ozYQw0MUHwulVi6648XY=" -> null
      - source_code_size               = 1213 -> null
      - tags                           = {} -> null
      - timeout                        = 5 -> null
      - version                        = "1" -> null

      - tracing_config {
          - mode = "PassThrough" -> null
        }
    }

module.advice.module.lambda.aws_lambda_function.lambda: Destroying... [id=pr-112-mentoradvice-edge-lambda]

Error: error deleting Lambda Function (pr-112-mentoradvice-edge-lambda): ResourceNotFoundException: Function not found: arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda
{
  RespMetadata: {
    StatusCode: 404,
    RequestID: "95bb3e85-fc36-45c9-851d-3bc517960f2a"
  },
  Message_: "Function not found: arn:aws:lambda:us-east-1:830138816992:function:pr-112-mentoradvice-edge-lambda",
  Type: "User"
}

Steps to Reproduce

1. Create Lambda function via TF
2. Delete Lambda function manually in AWS Console
3. Destroy the TF state

Important Factoids

It is a Lambda@Edge in us-east-1 region

[ewbankkit]

Related:

• [PLACEHOLDER] Consistent Resource Not Found Error Handling #15945
• RDS - error deleting, instance not found #12900
• resource/aws_app_cookie_stickiness_policy: Do not return error from Delete when resource not found #15781
• resource/aws_lb_cookie_stickiness_policy: Do not return error from Delete when resource not found #15782

Yes, we are not handling that error:

terraform-provider-aws/aws/resource_aws_lambda_function.go

Lines 677 to 692 in ee1bcab

	func resourceAwsLambdaFunctionDelete(d *schema.ResourceData, meta interface{}) error {
	conn := meta.(*AWSClient).lambdaconn
	log.Printf("[INFO] Deleting Lambda Function: %s", d.Id())
	params := &lambda.DeleteFunctionInput{
	FunctionName: aws.String(d.Get("function_name").(string)),
	}
	_, err := conn.DeleteFunction(params)
	if err != nil {
	return fmt.Errorf("error deleting Lambda Function (%s): %w", d.Id(), err)
	}
	return nil
	}

[ZsoltPath]

@ewbankkit I'm not sure how it works in other function where this is handled properly.
But I'd think it shouldn't even bring it up to delete. Just found it in discovery phase that the resource is already gone, nothing to delete.

[ryan-neff]

Would this behavior almost be expected due to the fact that terraform is relying solely on the tfstate where this lambda still resides?
The next step to remedy this I believe would be running a manual terraform state rm [options] ADDRESS.. in order to return to parity between what's in state and what resources are in aws.

I suppose just removing the state on destroy if the resource doesn't exist may be useful, but I don't know how other resources are handling that.

[bflad]

Generally, Terraform should be refreshing the resource state in the planning phase before destroy, which is a common scenario we handle to automatically remove resources from state when receiving "resource not found" type errors. However, this error is possible for operators when (currently, potentially not in the future) working around that behavior with the not recommended -refresh=false flag, in certain create before destroy scenarios, and other race conditions with external systems (e.g. humans in manual approval situations 😉 ) between the plan and apply phases.

Terraform AWS Provider resources should ignore "resource not found" type errors in their Delete functions, e.g. in this case it should be:

_, err := conn.DeleteFunction(params) 

if tfawserr.ErrCodeEquals(err, lambda.ErrCodeResourceNotFoundException) {
	return nil 
}

if err != nil { 
	return fmt.Errorf("error deleting Lambda Function (%s): %w", d.Id(), err) 
} 

If anyone is interested in fixing this, the code lives in aws/resource_aws_lambda_function.go. We do not have a great way to add acceptance testing for this case because it requires functionality not available in the Terraform Plugin SDK, but running the existing acceptance tests to check for regressions is fine in this case. See also Running Acceptance Testing section in the Contributing Guide for details about that.

[ryan-neff]

@bflad I can take it.

[anGie44]

Hi @ZsoltPath, preventing the error behavior during a terraform destroy has been addressed with the merge of #16183, which will release with the upcoming release of v3.20.0 of the Terraform AWS Provider .

[ZsoltPath]

Yay! Thanks all!

…

On Thu, 3 Dec 2020, 17:36 angie pinilla, ***@***.***> wrote: Hi @ZsoltPath <https://github.com/ZsoltPath>, preventing the error behavior during a terraform destroy has been addressed with the merge of #16183 <#16183>, which will release with the upcoming release of v3.20.0 of the Terraform AWS Provider . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16158 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOLGJ2KVUVY3RRKLDM4SO6TSS7EA5ANCNFSM4TTDBQMQ> .

[ghost]

This has been released in version 3.20.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!