aws_kinesis_stream: state always records shard_count = 1

[tphummel]

regardless of the actual shard_count, terraform plan attempts to destroy and recreate the stream if the resource has a setting > 1. The state file does not appear to be capturing the actual value.

I'm running Terraform 0.6.2 on OSX 10.10.4: 14.4.0 Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64 x86_64

Here are the steps to reproduce:

main.tf

variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "aws_region" {
  default = "us-west-2"
}

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region = "${var.aws_region}"
}

resource "aws_kinesis_stream" "test" {
  name = "tf-kinesis-bug"
  shard_count = 2
}

here is the output from running plan, apply, then plan again. the issue appears on the second plan:

➜  tf-kinesis-shard-bug  terraform plan -var "aws_access_key=${AWS_ACCESS_KEY_ID}" -var "aws_secret_key=${AWS_SECRET_ACCESS_KEY}" -out=plan .
Refreshing Terraform state prior to plan...


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan

+ aws_kinesis_stream.test
    arn:         "" => "<computed>"
    name:        "" => "tf-kinesis-bug"
    shard_count: "" => "2"


Plan: 1 to add, 0 to change, 0 to destroy.
➜  tf-kinesis-shard-bug  terraform apply plan
aws_kinesis_stream.test: Creating...
  arn:         "" => "<computed>"
  name:        "" => "tf-kinesis-bug"
  shard_count: "" => "2"
aws_kinesis_stream.test: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
➜  tf-kinesis-shard-bug  terraform plan -var "aws_access_key=${AWS_ACCESS_KEY_ID}" -var "aws_secret_key=${AWS_SECRET_ACCESS_KEY}" -out=plan .
Refreshing Terraform state prior to plan...

aws_kinesis_stream.test: Refreshing state... (ID: arn:aws:kinesis:us-west-2:XXX:stream/tf-kinesis-bug)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan

-/+ aws_kinesis_stream.test
    arn:         "arn:aws:kinesis:us-west-2:XXX:stream/tf-kinesis-bug" => "<computed>"
    name:        "tf-kinesis-bug" => "tf-kinesis-bug"
    shard_count: "1" => "2" (forces new resource)


Plan: 0 to add, 1 to change, 0 to destroy.

here is the content of my local state file after apply, showing the shard_count setting stuck at 1:

{
    "version": 1,
    "serial": 6,
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "aws_kinesis_stream.test": {
                    "type": "aws_kinesis_stream",
                    "primary": {
                        "id": "arn:aws:kinesis:us-west-2:XXX:stream/tf-kinesis-bug",
                        "attributes": {
                            "arn": "arn:aws:kinesis:us-west-2:XXX:stream/tf-kinesis-bug",
                            "id": "arn:aws:kinesis:us-west-2:XXX:stream/tf-kinesis-bug",
                            "name": "tf-kinesis-bug",
                            "shard_count": "1"
                        }
                    }
                }
            }
        }
    ]
}

[tphummel]

So I believe I have the solution to this. The DescribeStream api request returns an array of Shards on which you had been using len() to get the count. However, all calls to DescribeStream had been passing the param Limit: 1. This limit caps the # of shards returned in the response, which is why the count always ended up as 1 in the state file regardless of how many shards were actually created. With the Limit parameter omitted, the response will include all shards. FYI, the Limit parameter is optional with an allowable range of 0 to 10,000.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.