fix(#195): UserData behaviour change between 2 and 3 results in unexp…

…ected behaviour

Nodes that use custom `userdata` but don't use a custom-ami are creating a launch-template with the userdata in place
but AWS is then injecting their bootstrapper at the end of the userscript.
This means that `after_cluster_joining_userdata` will execute before cluster registration.

* Split the bootstrap out of the userdata templates into separate files, add ${bootstrap_script} into files in its place
* `launch_template.tf` Add precondition check; If `after_cluster_joining_userdata` is set but `ami_image_id` isn't and the OS is AL2/WINDOWS, show error
* `userdata.tf` Add `bootstrap_script` to local.userdata_vars; load in the userdata_bootstrap* file for the OS if the OS is AL2/Windows, otherwise, use empty string.
* `variables.tf` Add further detail to `after_cluster_joining_userdata`

.gitignore

@@ -17,3 +17,5 @@
 /.terraform.lock.hcl
 /.vscode
 /examples/complete/.terraform.lock.hcl
+
+/.infracost

README.md

@@ -81,6 +81,8 @@ This module supports all 4 OSes, but support for detailed configuration of the n
 are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
 supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.
 
+The fourth option is only available when setting an AMI ID for the node group; otherwise AWS will default to the released AMI for the OS and EKS will append your userdata with the bootstrap script.
+
 Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.
 
 
@@ -362,14 +364,14 @@ https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_additional_tag_map"></a> [additional\_tag\_map](#input\_additional\_tag\_map) | Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`.<br>This is for some rare cases where resources want additional configuration of tags<br>and therefore take a list of maps with tag key, value, and additional configuration. | `map(string)` | `{}` | no |
-| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional (bash for linux, powershell for Windows) commands to execute on each worker node after joining the EKS cluster<br>(after executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"<br>This can only be used with AL2 and Windows AMI types with a custom AMI set as EKS will inject a bootstrapper into all other user-scripts | `list(string)` | `[]` | no |
 | <a name="input_ami_image_id"></a> [ami\_image\_id](#input\_ami\_image\_id) | AMI to use, overriding other AMI specifications, but must match `ami_type`. Ignored if `launch_template_id` is supplied. | `list(string)` | `[]` | no |
 | <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | The EKS AMI "release version" to use. Defaults to the latest recommended version.<br>For Amazon Linux, it is the "Release version" from [Amazon AMI Releases](https://github.com/awslabs/amazon-eks-ami/releases)<br>For Bottlerocket, it is the release tag from [Bottlerocket Releases](https://github.com/bottlerocket-os/bottlerocket/releases) without the "v" prefix.<br>For Windows, it is "AMI version" from [AWS docs](https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-versions-windows.html).<br>Note that unlike AMI names, release versions never include the "v" prefix.<br>Examples:<br>  AL2: 1.29.3-20240531<br>  Bottlerocket: 1.2.0 or 1.2.0-ccf1b754<br>  Windows: 1.29-2024.04.09 | `list(string)` | `[]` | no |
 | <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64, AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD`. | `string` | `"AL2_x86_64"` | no |
 | <a name="input_associate_cluster_security_group"></a> [associate\_cluster\_security\_group](#input\_associate\_cluster\_security\_group) | When true, associate the default cluster security group to the nodes. If disabled the EKS managed security group will not<br>be associated to the nodes and you will need to provide another security group that allows the nodes to communicate with<br>the EKS control plane. Be aware that if no `associated_security_group_ids` or `ssh_access_security_group_ids` are provided,<br>then the nodes will have no inbound or outbound rules. | `bool` | `true` | no |
 | <a name="input_associated_security_group_ids"></a> [associated\_security\_group\_ids](#input\_associated\_security\_group\_ids) | A list of IDs of Security Groups to associate the node group with, in addition to the EKS' created security group.<br>These security groups will not be modified. | `list(string)` | `[]` | no |
 | <a name="input_attributes"></a> [attributes](#input\_attributes) | ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`,<br>in the order they appear in the list. New attributes are appended to the<br>end of the list. The elements of the list are joined by the `delimiter`<br>and treated as a single ID element. | `list(string)` | `[]` | no |
-| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
 | <a name="input_block_device_map"></a> [block\_device\_map](#input\_block\_device\_map) | Map of block device name specification, see [launch\_template.block-devices](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#block-devices). | <pre>map(object({<br>    no_device    = optional(bool, null)<br>    virtual_name = optional(string, null)<br>    ebs = optional(object({<br>      delete_on_termination = optional(bool, true)<br>      encrypted             = optional(bool, true)<br>      iops                  = optional(number, null)<br>      kms_key_id            = optional(string, null)<br>      snapshot_id           = optional(string, null)<br>      throughput            = optional(number, null)<br>      volume_size           = optional(number, 20)<br>      volume_type           = optional(string, "gp3")<br>    }))<br>  }))</pre> | <pre>{<br>  "/dev/xvda": {<br>    "ebs": {}<br>  }<br>}</pre> | no |
 | <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | DEPRECATED: Use `block_device_map` instead.<br>List of block device mappings for the launch template.<br>Each list element is an object with a `device_name` key and<br>any keys supported by the `ebs` block of `launch_template`. | `list(any)` | `null` | no |
 | <a name="input_bootstrap_additional_options"></a> [bootstrap\_additional\_options](#input\_bootstrap\_additional\_options) | Additional options to bootstrap.sh. DO NOT include `--kubelet-additional-args`, use `kubelet_additional_options` var instead. Not used with AL2023 AMI types. | `list(string)` | `[]` | no |

README.yaml

@@ -101,6 +101,8 @@ introduction: |-
   are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
   supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.
 
+  The fourth option is only available when setting an AMI ID for the node group; otherwise AWS will default to the released AMI for the OS and EKS will append your userdata with the bootstrap script.
+
   Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.
 
 

docs/terraform.md

@@ -49,14 +49,14 @@
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_additional_tag_map"></a> [additional\_tag\_map](#input\_additional\_tag\_map) | Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`.<br>This is for some rare cases where resources want additional configuration of tags<br>and therefore take a list of maps with tag key, value, and additional configuration. | `map(string)` | `{}` | no |
-| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional (bash for linux, powershell for Windows) commands to execute on each worker node after joining the EKS cluster<br>(after executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"<br>This can only be used with AL2 and Windows AMI types with a custom AMI set as EKS will inject a bootstrapper into all other user-scripts | `list(string)` | `[]` | no |
 | <a name="input_ami_image_id"></a> [ami\_image\_id](#input\_ami\_image\_id) | AMI to use, overriding other AMI specifications, but must match `ami_type`. Ignored if `launch_template_id` is supplied. | `list(string)` | `[]` | no |
 | <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | The EKS AMI "release version" to use. Defaults to the latest recommended version.<br>For Amazon Linux, it is the "Release version" from [Amazon AMI Releases](https://github.com/awslabs/amazon-eks-ami/releases)<br>For Bottlerocket, it is the release tag from [Bottlerocket Releases](https://github.com/bottlerocket-os/bottlerocket/releases) without the "v" prefix.<br>For Windows, it is "AMI version" from [AWS docs](https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-versions-windows.html).<br>Note that unlike AMI names, release versions never include the "v" prefix.<br>Examples:<br>  AL2: 1.29.3-20240531<br>  Bottlerocket: 1.2.0 or 1.2.0-ccf1b754<br>  Windows: 1.29-2024.04.09 | `list(string)` | `[]` | no |
 | <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64, AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD`. | `string` | `"AL2_x86_64"` | no |
 | <a name="input_associate_cluster_security_group"></a> [associate\_cluster\_security\_group](#input\_associate\_cluster\_security\_group) | When true, associate the default cluster security group to the nodes. If disabled the EKS managed security group will not<br>be associated to the nodes and you will need to provide another security group that allows the nodes to communicate with<br>the EKS control plane. Be aware that if no `associated_security_group_ids` or `ssh_access_security_group_ids` are provided,<br>then the nodes will have no inbound or outbound rules. | `bool` | `true` | no |
 | <a name="input_associated_security_group_ids"></a> [associated\_security\_group\_ids](#input\_associated\_security\_group\_ids) | A list of IDs of Security Groups to associate the node group with, in addition to the EKS' created security group.<br>These security groups will not be modified. | `list(string)` | `[]` | no |
 | <a name="input_attributes"></a> [attributes](#input\_attributes) | ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`,<br>in the order they appear in the list. New attributes are appended to the<br>end of the list. The elements of the list are joined by the `delimiter`<br>and treated as a single ID element. | `list(string)` | `[]` | no |
-| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
 | <a name="input_block_device_map"></a> [block\_device\_map](#input\_block\_device\_map) | Map of block device name specification, see [launch\_template.block-devices](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#block-devices). | <pre>map(object({<br>    no_device    = optional(bool, null)<br>    virtual_name = optional(string, null)<br>    ebs = optional(object({<br>      delete_on_termination = optional(bool, true)<br>      encrypted             = optional(bool, true)<br>      iops                  = optional(number, null)<br>      kms_key_id            = optional(string, null)<br>      snapshot_id           = optional(string, null)<br>      throughput            = optional(number, null)<br>      volume_size           = optional(number, 20)<br>      volume_type           = optional(string, "gp3")<br>    }))<br>  }))</pre> | <pre>{<br>  "/dev/xvda": {<br>    "ebs": {}<br>  }<br>}</pre> | no |
 | <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | DEPRECATED: Use `block_device_map` instead.<br>List of block device mappings for the launch template.<br>Each list element is an object with a `device_name` key and<br>any keys supported by the `ebs` block of `launch_template`. | `list(any)` | `null` | no |
 | <a name="input_bootstrap_additional_options"></a> [bootstrap\_additional\_options](#input\_bootstrap\_additional\_options) | Additional options to bootstrap.sh. DO NOT include `--kubelet-additional-args`, use `kubelet_additional_options` var instead. Not used with AL2023 AMI types. | `list(string)` | `[]` | no |
@@ -119,7 +119,7 @@
 
 | Name | Description |
 |------|-------------|
-| <a name="output_WARNING_cluster_autoscaler_enabled"></a> [WARNING\_cluster\_autoscaler\_enabled](#output\_WARNING\_cluster\_autoscaler\_enabled) | n/a |
+| <a name="output_WARNING_cluster_autoscaler_enabled"></a> [WARNING\_cluster\_autoscaler\_enabled](#output\_WARNING\_cluster\_autoscaler\_enabled) | WARNING |
 | <a name="output_eks_node_group_ami_id"></a> [eks\_node\_group\_ami\_id](#output\_eks\_node\_group\_ami\_id) | The ID of the AMI used for the worker nodes, if specified |
 | <a name="output_eks_node_group_arn"></a> [eks\_node\_group\_arn](#output\_eks\_node\_group\_arn) | Amazon Resource Name (ARN) of the EKS Node Group |
 | <a name="output_eks_node_group_cbd_pet_name"></a> [eks\_node\_group\_cbd\_pet\_name](#output\_eks\_node\_group\_cbd\_pet\_name) | The pet name of this node group, if this module generated one |

launch-template.tf

@@ -197,6 +197,12 @@ resource "aws_launch_template" "default" {
       condition     = contains(["AL2", "WINDOWS"], local.ami_os) || length(local.userdata_vars.after_cluster_joining_userdata) == 0 || (local.ami_os == "AL2" || local.ami_os == "WINDOWS")
       error_message = format("The input `after_cluster_joining_userdata` is not supported for %v.", title(lower(local.ami_os)))
     }
+
+    # The EKS bootstrap will be injected into all userdata if it is not a custom-ami
+    precondition {
+      condition     = length(local.userdata_vars.after_cluster_joining_userdata) == 0 || length(var.ami_image_id) != 0 && length(local.userdata_vars.after_cluster_joining_userdata) > 0 && (local.ami_os == "AL2" || local.ami_os == "WINDOWS")
+      error_message = format("The input `after_cluster_joining_userdata` is not supported for %v, an ami_image_id must be set for this functionality", title(lower(local.ami_os)))
+    }
   }
 }
 

userdata.tf

@@ -47,7 +47,11 @@ locals {
     WINDOWS      = "${path.module}/userdata_nt.tpl"
   }
 
-
+  userdata_bootstrapper_template_file = {
+    AL2          = "${path.module}/userdata_bootstrap.tpl"
+    BOTTLEROCKET = "${path.module}/userdata_bootstrap.tpl"
+    WINDOWS      = "${path.module}/userdata_bootstrap_nt.tpl"
+  }
 
   # When suppressing EKS bootstrap, add --register-with-taints to kubelet_extra_args,
   #   e.g. --register-with-taints=test=:PreferNoSchedule
@@ -68,12 +72,14 @@ locals {
   kubelet_extra_args_yaml = replace(local.kubelet_extra_args, "--", "\n      - >-\n        --")
 
   userdata_vars = {
+    bootstrap_script                = (length(var.ami_image_id) > 0 && length(var.after_cluster_joining_userdata) > 0 && (local.ami_os == "AL2" || local.ami_os == "WINDOWS")) ? file(local.userdata_template_file[local.ami_os]) : ""
     before_cluster_joining_userdata = length(var.before_cluster_joining_userdata) == 0 ? "" : join("\n", var.before_cluster_joining_userdata)
     kubelet_extra_args              = local.kubelet_extra_args
     kubelet_extra_args_yaml         = local.kubelet_extra_args_yaml
     bootstrap_extra_args            = length(var.bootstrap_additional_options) == 0 ? "" : join(" ", var.bootstrap_additional_options)
     after_cluster_joining_userdata  = length(var.after_cluster_joining_userdata) == 0 ? "" : join("\n", var.after_cluster_joining_userdata)
 
+
     cluster_endpoint           = local.get_cluster_data ? data.aws_eks_cluster.this[0].endpoint : null
     certificate_authority_data = local.get_cluster_data ? data.aws_eks_cluster.this[0].certificate_authority[0].data : null
     cluster_name               = local.get_cluster_data ? data.aws_eks_cluster.this[0].name : null

userdata.tpl

@@ -15,7 +15,7 @@ export KUBELET_EXTRA_ARGS="${kubelet_extra_args}"
 %{ endif }
 %{ if length(kubelet_extra_args) > 0 || length (bootstrap_extra_args) > 0 || length (after_cluster_joining_userdata) > 0 }
 
-/etc/eks/bootstrap.sh --apiserver-endpoint '${cluster_endpoint}' --b64-cluster-ca '${certificate_authority_data}' ${bootstrap_extra_args} '${cluster_name}'
+${bootstrap_script}
 
 ${after_cluster_joining_userdata}
 %{ endif }

userdata_bootstrap.tpl

@@ -0,0 +1 @@
+/etc/eks/bootstrap.sh --apiserver-endpoint '${cluster_endpoint}' --b64-cluster-ca '${certificate_authority_data}' ${bootstrap_extra_args} '${cluster_name}'

userdata_bootstrap_nt.tpl

@@ -0,0 +1,5 @@
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+
+& $EKSBootstrapScriptFile -EKSClusterName "${cluster_name}" -APIServerEndpoint "${cluster_endpoint}" -Base64ClusterCA "${certificate_authority_data}" ${bootstrap_extra_args} -KubeletExtraArgs "${kubelet_extra_args}" 3>&1 4>&1 5>&1 6>&1

userdata_nt.tpl

@@ -22,11 +22,7 @@ if ($disks_to_adjust -ne $null) {
   }
 }
 
-[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
-[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
-[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
-
-& $EKSBootstrapScriptFile -EKSClusterName "${cluster_name}" -APIServerEndpoint "${cluster_endpoint}" -Base64ClusterCA "${certificate_authority_data}" ${bootstrap_extra_args} -KubeletExtraArgs "${kubelet_extra_args}" 3>&1 4>&1 5>&1 6>&1
+${bootstrap_script}
 
 try{
 ${after_cluster_joining_userdata}