provider/aws: elasticache cluster exports nodes before they exist

[juniorplenty]

Originally reported by @saulshanabrook here: #1965
Verified it still exists in 0.5.2

Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* Resource 'aws_elasticache_cluster.default' does not have attribute 'cache_nodes.0.address' for variable 'aws_elasticache_cluster.default.cache_nodes.0.address'

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

cache_nodes.0 doesn't exist because elasticache isn't really done provisioning the cluster, and terraform doesn't wait for it to do so.

[juniorplenty]

Here are test configs, note the route53 record creation at the bottom that attempts to use a non-existant cluster node index:

### VARS #################################
variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "region" {
  default = "us-east-1"
}
variable "availability_zone" {
  default = "us-east-1e"
}

### VPC ##################################
provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region     = "${var.region}"
}
resource "aws_vpc" "default" {
  cidr_block = "10.250.0.0/16"
}
resource "aws_internet_gateway" "default" {
  vpc_id = "${aws_vpc.default.id}"
}
resource "aws_subnet" "public" {
  vpc_id = "${aws_vpc.default.id}"
  cidr_block = "10.250.2.0/24"
  availability_zone = "${var.availability_zone}"
  map_public_ip_on_launch = true
}
resource "aws_subnet" "private" {
  vpc_id = "${aws_vpc.default.id}"
  cidr_block = "10.250.3.0/24"
  availability_zone = "${var.availability_zone}"
  map_public_ip_on_launch = true
}
resource "aws_route_table_association" "private" {
  subnet_id = "${aws_subnet.private.id}"
  route_table_id = "${aws_route_table.public.id}"
}
resource "aws_route_table" "public" {
  vpc_id = "${aws_vpc.default.id}"
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.default.id}"
  }
}
resource "aws_route_table_association" "public" {
  subnet_id = "${aws_subnet.public.id}"
  route_table_id = "${aws_route_table.public.id}"
}
resource "aws_route53_zone" "main" {
  name = "test.infra"
  vpc_id = "${aws_vpc.default.id}"
}

### ELASTICACHE ##########################
resource "aws_elasticache_subnet_group" "test" {
  name = "test"
  description = "Test Redis Subnet Group"
  subnet_ids = ["${aws_subnet.private.id}"]
}
resource "aws_elasticache_cluster" "test" {
  cluster_id = "test"
  subnet_group_name = "${aws_elasticache_subnet_group.test.name}"
  num_cache_nodes = 1
  node_type = "cache.t2.micro"
  engine = "redis"
  engine_version = "2.8.19"
  parameter_group_name = "default.redis2.8"
}

### THIS EXPOSES THE CONDITION
resource "aws_route53_record" "test-cache" {
  zone_id = "${aws_route53_zone.main.id}"
  name = "test-cache"
  type = "CNAME"
  ttl = "300"
  records = ["${ aws_elasticache_cluster.test.cache_nodes.0.address }"]
}

[ahamidi]

👍 Blocked on this as well.

[catsby]

Hello friends –

I've tried on version 0.5.2, and current master(e794287), and cannot reproduce this issue.
I used the config shown above, and every attempt resulted in a clean plan-apply-create.
I wanted in the console as well, and the Route 53 record doesn't come along until after the cluster is up, which itself does not report "available" until the node reports "available".

If you're still hitting this with master, or have another case in 0.5.2 that reproduces this, I can take another look.

Otherwise, I'm going to close this issue tomorrow. Thank you for the report, I'm hoping I can hear back from someone with a new example, or a confirmed "closed" on this.

Thanks!

[catsby]

I did open #2128 as an extra safety-net of sorts, take a look!

[juniorplenty]

It doesn't happen every time - don't know what to say, just hit this again today. #2128 probably handles it but hard to reproduce and test unless there's some way to mock the AWS calls during testing.

[catsby]

@juniorplenty I've worked with AWS API enough to both be unable to reproduce this, and totally believe you that it happens 😄

I asked for other examples just incase there was some variable or otherwise external thing we weren't noticing that was the true cause, but it's probably just some AWS API weirdness that may have passed and won't resurface until it's least convenient. I imagine we'll merge 2128 and just roll with it

[catsby]

I just merged #2128 to help here, please check out master and let me know if this happens again.

Thanks for reporting!

[juniorplenty]

@catsby couple of updates:

• 0.5.3 still exhibits this behavior - I'm assuming your patch above made it in there but I haven't checked
• I've been able to observe terraform continuing with provisioning (and triggering this race condition) when my elasticache cluster is still very much in the "creating" state. I've seen clusters hang for 10+ minutes in "creating" state, with multiple terraform runs (including the original create) being happy to proceed with dependent resources (e.g. the Route 53 record above) anyway

[juniorplenty]

by the way here's the full error I'm getting:

Error applying plan:

2 error(s) occurred:

* 1 error(s) occurred:

* aws_security_group.elasticache_audience_api: diffs didn't match during apply. This is a bug with Terraform and should be reported.
* 1 error(s) occurred:

* Resource 'aws_elasticache_cluster.audience_api_cache' does not have attribute 'cache_nodes.0.address' for variable 'aws_elasticache_cluster.audience_api_cache.cache_nodes.0.address'

[juniorplenty]

I've also noticed the same issue (continuing despite "creating" state) with RDS resources just FYI

[catsby]

@juniorplenty are you still using the same configuration? Can you gist some output that shows this, using TF_LOG=1 when running it? Be sure to omit any secrets!

[juniorplenty]

@catsby Here's log output from a failed run using the plan generated by the exact configs above (with AWS keys added of course) https://gist.github.com/juniorplenty/b99a85bca4ecf1362a8d

[juniorplenty]

@catsby wouldn't just waiting for cache_nodes.0 to be defined be a better test for when the cluster is really available? [UPDATE: duh, just remembered that's what your patch does. Hoping something in that log points to what's going on, it looks like that patch made it into 0.5.3 so there's a serious ghost in the machine here...)

[juniorplenty]

(Also - should this issue really be "closed"? The above logs verify that it's still broken in 0.5.3...)

[jonhatalla]

+1

[ahamidi]

FWIW, I'm still seeing the same issue as well.

[jhardin293]

+1

[juniorplenty]

@catsby can we at least get this issue re-opened? I posted the logs you asked for, they show it still happening in 0.5.3 -

[phinze]

Reopening and taking a look!

[catsby]

Sorry for the delay @juniorplenty – I'm taking another look here

[catsby]

From master, I ran the included plan file and got the included results:

• https://gist.github.com/catsby/b1c7ab061be821e363dc

TL;DR I can't reproduce this on master. @juniorplenty are you capable of building from master and trying?

Does anyone else who's reported this issue have a minimal config that reproduces it?

Thanks!

[catsby]

Hey all – with no further information and @phinze and I both not being able to reproduce this, I'm going to re-close this. I assume it's fixed on master (~ 6fdbca8), but if anyone can provide config and log to show otherwise we'll gladly dig in again.

Thanks!

[catsby]

@juniorplenty can you checkout #2842 ?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.