Terraform 1.6.x does not find *.tfstate file in s3 which was created using older versions of terraform

[jurna]

Terraform Version

Terraform v1.6.4
on linux_amd64

Terraform Configuration Files

Here is backend.tf file:

terraform {
  backend "s3" {
    bucket         = "terraform-bucket"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
    key            = "./terraform.tfstate"
    region         = "eu-central-1"
  }

Debug Output

Here is log from terraform 1.5.x (credentials removed) (result - 200 OK):

-----------------------------------------------------
2023-11-15T12:11:55.779+0200 [DEBUG] [aws-sdk-go] {}
2023-11-15T12:11:55.780+0200 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /terraform.tfstate HTTP/1.1
Host: terraform-bucket.s3.eu-central-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.5.7 aws-sdk-go/1.44.122 (go1.20.7; linux; amd64)
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231115T101155Z
Accept-Encoding: gzip


-----------------------------------------------------
2023-11-15T12:11:55.991+0200 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 384760
Accept-Ranges: bytes
Content-Type: application/json
Date: Wed, 15 Nov 2023 10:11:56 GMT
Etag: "f357341b9645d1643cb8ee79022cd215"
Last-Modified: Fri, 27 Oct 2023 08:46:51 GMT
Server: AmazonS3
X-Amz-Id-2: Y9LAIgVKl5bjua2wExd1quq5KXdnDq5wCd8eE3GbFJlqo3bNOkVvqjoZITgeZkp9JuMHn8wRw0E=
X-Amz-Request-Id: BQPC6ES63540GTHN
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: nhKueNmyZNat3tqAlmwAeGFTdcYYwWPN

Here is the log from terraform 1.6.x (404 error):

2023-11-15T12:15:07.874+0200 [INFO]  backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate
2023-11-15T12:15:07.875+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.request.header.x_amz_date=20231115T101507Z http.method=HEAD net.peer.name=terraform-bucket.s3.eu-central-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.3 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.3 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=AKIA************AKO7/20231115/eu-central-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.url=https://terraform-bucket.s3.eu-central-1.amazonaws.com/./terraform.tfstate http.request.header.amz_sdk_invocation_id=813d4010-7c72-4ccf-a30e-59331be39944 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.body=""
2023-11-15T12:15:07.921+0200 [DEBUG] backend-s3: HTTP Response Received: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.response.header.date="Wed, 15 Nov 2023 10:15:07 GMT" http.response.header.server=AmazonS3 http.response.body="" http.duration=46 http.status_code=404 http.response.header.x_amz_request_id=NX3QC1MC0N9N5TW6 http.response.header.x_amz_id_2="yUmgCqK3zpSB0bReJ5bxxCy0B88ZhARUw5Omn5VsbCUzGSUbADoqnOBurOZZ+GL0a6eDbY7gl4Q=" http.response.header.content_type=application/xml

Expected Behavior

Expect plan successfully executed

Actual Behavior

terraform plan returns error:

│ Error: error loading state: state data in S3 does not have the expected content.
│
│ The checksum calculated for the state stored in S3 does not match the checksum
│ stored in DynamoDB.
│
│ Bucket: ocb-ocb-terraform-state
│ Key: ./terraform.tfstate
│ Calculated checksum:
│ Stored checksum: bb17f4c60b305db0f0fd8210c8da8332
│
│ This may be caused by unusually long delays in S3 processing a previous state
│ update. Please wait for a minute or two and try again.
│
│ If this problem persists, and neither S3 nor DynamoDB are experiencing an
│ outage, you may need to manually verify the remote state and remove the Digest
│ value stored in the DynamoDB table
│
│
│

Steps to Reproduce

Using terraform 1.5.x apply plan:

• terraform init
• terraform apply

Upgrade to 1.6.x and then apply:

• terraform apply

1.6 version returns error:

Additional Context

No response

References

Also created issue in terragrunt, because backend.tf is actually created by terragrunt:
gruntwork-io/terragrunt#2792

[TheEdgeOfRage]

Have you found any workarounds for this? Like manually updating the state file to transition to 1.6?

[gdavison]

Hi @jurna and @TheEdgeOfRage. As a workaround, you can remove the leading ./ from the key.

Terraform v1.5.x and lower used the AWS SDK for Go v1, which stripped the leading ./ from S3 object keys, as well as making some other changes such as removing double slashes. Terraform v1.6 uses the AWS SDK for Go v2, which uses the key as-is

[Ficqueth]

So glad I found this! I was just having the same problem and removing the ./ for the key in the backend fixed it.
It seems like a big breaking change to me, I am really surprised that there is not more documentation on this.