aws_msk_cluster - output bootstrap_brokers returns empty

[cdenneen]

Based on example (https://www.terraform.io/docs/providers/aws/r/msk_cluster.html#) I've setup 2 outputs (one for bootstrap_brokers and the other for zookeeper_connect_string):

output "zookeeper_connect_string" {
    value = "${aws_msk_cluster.kafka.zookeeper_connect_string}"
}

output "bootstrap_brokers" {
    value = "${aws_msk_cluster.kafka.bootstrap_brokers}"
}

However only zookeeper_connect_string returns during cluster creation.

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

bootstrap_brokers =
zookeeper_connect_string = 10.1.13.241:2181,10.1.11.98:2181,10.1.12.107:2181

[bflad]

Hi @cdenneen 👋 Thanks for reporting this and sorry you ran into trouble here.

A few questions for you:

• Is the bootstrap_brokers_tls attribute value filled in during creation?
• Does the bootstrap_brokers output value eventually fill in after a certain period of time?

If you are setting up client broker encryption in transit with just the TLS setting like the below:

resource "aws_msk_cluster" "example" {
  # ... other configuration ...
  encryption_info {
    # ... other configuration ...
    encryption_in_transit {
      client_broker = "TLS"
    }
  }
}

It is expected that the bootstrap_brokers value is empty, while the bootstrap_brokers_tls value is filled. We should certainly update our documentation for this case. 👍 Please let us know.

[cdenneen]

Ok probably should change example to output the bootstrap_servers_tls instead. Thanks for clarifying.

[bflad]

I have submitted the following documentation pull request, if you would like to look: #8957

[adv4000]

I have same issue
The following output return empty string:

output "msk_kafka_connections" {
  value = "${aws_msk_cluster.datalake.bootstrap_brokers}"
}

I'am not using any TLS
Tried to output also ${aws_msk_cluster.datalake.bootstrap_brokers_tls} got error:
Resource 'aws_msk_cluster.datalake' does not have attribute 'bootstrap_brokers_tls'
This is my cluster:

resource "aws_msk_cluster" "datalake" {
  cluster_name           = "${var.msk_cluster_name}"
  kafka_version          = "${var.kafka_version}"
  number_of_broker_nodes = "${var.number_of_broker_nodes}"
  enhanced_monitoring    = "${var.monitoring_level}"

  broker_node_group_info {
    instance_type   = "${var.instance_type}"
    ebs_volume_size = "${var.ebs_volume_size}"

    client_subnets = [
      "${data.terraform_remote_state.network.private_subnets.0}",
      "${data.terraform_remote_state.network.private_subnets.1}",
      "${data.terraform_remote_state.network.private_subnets.2}",
    ]

    security_groups = ["${aws_security_group.kafka_sg.id}"]
  }

  tags = {
    Name = "${var.msk_cluster_name}-cluster"
  }
}

Definitely BUG

[bflad]

Hi @adv4000 -- sorry for the frustrating situation.

Over in #8953 (comment) we noticed that while the AWS MSK documentation states the default value of client-broker encryption in transit is TLS_PLAINTEXT, which would populate both bootstrap_brokers and bootstrap_brokers_tls, that this is not true in the us-east-1 AWS region where it defaults to TLS and would only populate bootstrap_brokers_tls. This may be true in other AWS regions as well. That outcome of any MSK team feedback there will determine how we handle the behavior and document the example on the resource documentation page.

Regarding the error does not have attribute 'bootstrap_brokers_tls', can you please verify that you are running Terraform AWS Provider version 2.14.0, which contains that new attribute? Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[adv4000]

Hi @bflad
Yes I had previous version of AWS Provider, just updated it to 2.14 and now terraform apply
Show this:
encryption_info.0.encryption_in_transit.0.client_broker: "TLS" => "TLS_PLAINTEXT" (forces new resource)
Probably in previous version it was set to TLS and in 2.14 it set to TLS_PLAINTEXT as default.

I have recreated MKS Cluster from beginning and now output show this:

output "msk_kafka_connections" {
  value = "${aws_msk_cluster.datalake.bootstrap_brokers}"
}

Still Empty

output "msk_kafka_connections_tls" {
  value = "${aws_msk_cluster.datalake.bootstrap_brokers_tls}"
}

Show connections string with port 9094.

I'am using eu-west-1 Ireland

[adv4000]

I would recommend to make Reference only one Kafka Bootstrap, and not two References if they cannot exist together. one is always empty.

By now I will use this to output whatever is not empty:

output "msk_kafka_connections" {
  value = "${aws_msk_cluster.mvp.bootstrap_brokers != "" ? aws_msk_cluster.mvp.bootstrap_brokers : aws_msk_cluster.mvp.bootstrap_brokers_tls}"
}

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!