Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DRAFT] docs: update do-not-disrupt description #6977

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[DRAFT] docs: update do-not-disrupt description #6977

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 28 additions & 18 deletions website/content/en/docs/concepts/disruption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ Karpenter will add the `Drifted` status condition on NodeClaims if the NodeClaim
Automated forceful methods will begin draining nodes as soon as the condition is met. Note that these methods blow past NodePool Disruption Budgets, and do not wait for a pre-spin replacement node to be healthy for the pods to reschedule, unlike the graceful methods mentioned above. Use Pod Disruption Budgets and `do-not-disrupt` on your nodes to rate-limit the speed at which your applications are disrupted.

### Expiration
Karpenter will disrupt nodes as soon as they're expired after they've lived for the duration of the NodePool's `spec.template.spec.expireAfter`. You can use expiration to periodically recycle nodes due to security concern.
Karpenter will disrupt nodes as soon as they're expired after they've lived for the duration of the NodePool's `spec.template.spec.expireAfter`. You can use expiration to periodically recycle nodes due to security concern.

### Interruption

Expand All @@ -200,17 +200,25 @@ To enable interruption handling, configure the `--interruption-queue` CLI argume

## Controls

### TerminationGracePeriod
### TerminationGracePeriod

You can set a NodePool's `terminationGracePeriod` through the `spec.template.spec.terminationGracePeriod` field. This field defines the duration of time that a node can be draining before it's forcibly deleted. A node begins draining when it's deleted. Pods will be deleted preemptively based on its TerminationGracePeriodSeconds before this terminationGracePeriod ends to give as much time to cleanup as possible. Note that if your pod's terminationGracePeriodSeconds is larger than this terminationGracePeriod, Karpenter may forcibly delete the pod before it has its full terminationGracePeriod to cleanup.
You can set a NodePool's `terminationGracePeriod` through the `spec.template.spec.terminationGracePeriod` field.
This field defines the duration of time that a node can be draining before it's forcibly deleted.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This field defines the duration of time that a node can be draining before it's forcibly deleted.
This field defines the duration of time that a node can be draining before it's forcibly deleted.

A node begins draining when it's deleted.
Pods will be deleted preemptively based on its TerminationGracePeriodSeconds before this terminationGracePeriod ends to give as much time to cleanup as possible.
Note that if your pod's terminationGracePeriodSeconds is larger than this terminationGracePeriod, Karpenter may forcibly delete the pod before it has its full terminationGracePeriod to cleanup.

This is especially useful in combination with `nodepool.spec.template.spec.expireAfter` to define an absolute maximum on the lifetime of a node, where a node is deleted at `expireAfter` and finishes draining within the `terminationGracePeriod` thereafter. Pods blocking eviction like PDBs and do-not-disrupt will block full draining until the `terminationGracePeriod` is reached.
This is especially useful in combination with `nodepool.spec.template.spec.expireAfter` to define an absolute maximum on the lifetime of a node, where a node is deleted at `expireAfter` and finishes draining within the `terminationGracePeriod` thereafter.
Pods blocking eviction like PDBs and do-not-disrupt will block full draining until the `terminationGracePeriod` is reached.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Pods blocking eviction like PDBs and do-not-disrupt will block full draining until the `terminationGracePeriod` is reached.
Pods blocking eviction like PDBs and `do-not-disrupt` will block full draining until the `terminationGracePeriod` is reached.


For instance, a NodeClaim with `terminationGracePeriod` set to `1h` and an `expireAfter` set to `23h` will begin draining after it's lived for `23h`. Let's say a `do-not-disrupt` pod has `TerminationGracePeriodSeconds` set to `300` seconds. If the node hasn't been fully drained after `55m`, Karpenter will delete the pod to allow it's full `terminationGracePeriodSeconds` to cleanup. If no pods are blocking draining, Karpenter will cleanup the node as soon as the node is fully drained, rather than waiting for the NodeClaim's `terminationGracePeriod` to finish.
For instance, a NodeClaim with `terminationGracePeriod` set to `1h` and an `expireAfter` set to `23h` will begin draining after it's lived for `23h`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For instance, a NodeClaim with `terminationGracePeriod` set to `1h` and an `expireAfter` set to `23h` will begin draining after it's lived for `23h`.
For instance, a NodeClaim with `terminationGracePeriod` set to `1h` and an `expireAfter` set to `23h` will begin draining `23h` after its creation. The NodeClaim will then be allowed to drain for up to `1h` before its forcefully terminated from the cluster.

Let's say a `do-not-disrupt` pod has `TerminationGracePeriodSeconds` set to `300` seconds.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Let's say a `do-not-disrupt` pod has `TerminationGracePeriodSeconds` set to `300` seconds.
Let's say a `do-not-disrupt` pod has `TerminationGracePeriodSeconds` set to `300` seconds (`5m`).

If the node hasn't been fully drained after `55m`, Karpenter will delete the pod to allow it's full `terminationGracePeriodSeconds` to cleanup.
If no pods are blocking draining, Karpenter will cleanup the node as soon as the node is fully drained, rather than waiting for the NodeClaim's `terminationGracePeriod` to finish.

### NodePool Disruption Budgets

You can rate limit Karpenter's disruption through the NodePool's `spec.disruption.budgets`. If undefined, Karpenter will default to one budget with `nodes: 10%`. Budgets will consider nodes that are actively being deleted for any reason, and will only block Karpenter from disrupting nodes voluntarily through drift, emptiness, and consolidation. Note that NodePool Disruption Budgets do not prevent Karpenter from cleaning up expired or drifted nodes.
You can rate limit Karpenter's disruption through the NodePool's `spec.disruption.budgets`. If undefined, Karpenter will default to one budget with `nodes: 10%`. Budgets will consider nodes that are actively being deleted for any reason, and will only block Karpenter from disrupting nodes voluntarily through drift, emptiness, and consolidation. Note that NodePool Disruption Budgets do not prevent Karpenter from cleaning up expired or drifted nodes.

#### Reasons
Karpenter allows specifying if a budget applies to any of `Drifted`, `Underutilized`, or `Empty`. When a budget has no reasons, it's assumed that it applies to all reasons. When calculating allowed disruptions for a given reason, Karpenter will take the minimum of the budgets that have listed the reason or have left reasons undefined.
Expand All @@ -223,7 +231,7 @@ If the budget is configured with a percentage value, such as `20%`, Karpenter wi
For example, the following NodePool with three budgets defines the following requirements:
- The first budget will only allow 20% of nodes owned by that NodePool to be disrupted if it's empty or drifted. For instance, if there were 19 nodes owned by the NodePool, 4 empty or drifted nodes could be disrupted, rounding up from `19 * .2 = 3.8`.
- The second budget acts as a ceiling to the previous budget, only allowing 5 disruptions when there are more than 25 nodes.
- The last budget only blocks disruptions during the first 10 minutes of the day, where 0 disruptions are allowed, only applying to underutilized nodes.
- The last budget only blocks disruptions during the first 10 minutes of the day, where 0 disruptions are allowed, only applying to underutilized nodes.

```yaml
apiVersion: karpenter.sh/v1
Expand All @@ -232,20 +240,20 @@ metadata:
name: default
spec:
template:
spec:
spec:
expireAfter: 720h # 30 * 24h = 720h
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
budgets:
- nodes: "20%"
reasons:
reasons:
- "Empty"
- "Drifted"
- nodes: "5"
- nodes: "0"
schedule: "@daily"
duration: 10m
reasons:
reasons:
- "Underutilized"
```

Expand Down Expand Up @@ -274,8 +282,14 @@ Duration and Schedule must be defined together. When omitted, the budget is alwa

### Pod-Level Controls

You can block Karpenter from voluntarily choosing to disrupt certain pods by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the pod. This is useful for pods that you want to run from start to finish without disruption. By opting pods out of this disruption, you are telling Karpenter that it should not voluntarily remove a node containing this pod.
You can block Karpenter from voluntarily choosing to disrupt certain pods by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the pod.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can block Karpenter from voluntarily choosing to disrupt certain pods by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the pod.
You can block Karpenter from voluntarily disrupting pods by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the pod.

You can treat this annotation as a single-node, permanently blocking PDB.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can treat this annotation as a single-node, permanently blocking PDB.
You can treat this annotation as a single-pod, permanently blocking PDB.

This has the following consequences:
- Nodes with `do-not-disrupt` pods will be excluded from **voluntary** disruption, i.e. [Consolidation]({{<ref "#consolidation" >}}) and [Drift]({{<ref "#drift" >}}).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't voluntary disruption include drift? I'm not sure that they are considered different things here

- Like pods with a blocking PDB, pods with the `do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{ref "#terminationcontroller"}}).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Like pods with a blocking PDB, pods with the `do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{ref "#terminationcontroller"}}).
- Like pods with a fully blocking PDB, pods with the `do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{ref "#terminationcontroller"}}).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Like pods with a blocking PDB, pods with the `do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{ref "#terminationcontroller"}}).
- Like pods with a blocking PDB, pods with the `do-not-disrupt` annotation will **not** be gracefully evicted by the [Termination Controller]({{ref "#terminationcontroller"}}). These pods will either run to completion or be forcefully terminated when the node is near its terminationGracePeriod

Consider linking to terminationGracePeriod if you update the docs wording in this way too

Karpenter will not be able to complete termination of the node until one of the following conditions is met: the `do-not-disrupt` pod is removed, it enters a [terminating](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) or [terminal](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) state, or the owning NodeClaim's [TerminationGracePeriod]({{<ref "#terminationgraceperiod" >}}) has been met.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to create a sub-bulleted list here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Karpenter will not be able to complete termination of the node until one of the following conditions is met: the `do-not-disrupt` pod is removed, it enters a [terminating](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) or [terminal](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) state, or the owning NodeClaim's [TerminationGracePeriod]({{<ref "#terminationgraceperiod" >}}) has been met.
Karpenter will not be able to complete termination of the node until one of the following conditions is met: the `do-not-disrupt` pods are removed, the pods enter [terminating](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) or [terminal](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) states, or the owning NodeClaim's [TerminationGracePeriod]({{<ref "#terminationgraceperiod" >}}) has been met.


This is useful for pods that you want to run from start to finish without disruption.
Examples of pods that you might want to opt-out of disruption include an interactive game that you don't want to interrupt or a long batch job (such as you might have with machine learning) that would need to start over if it were interrupted.

```yaml
Expand All @@ -292,18 +306,14 @@ spec:
This annotation will be ignored for [terminating pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) and [terminal pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase) (Failed/Succeeded).
{{% /alert %}}

Examples of voluntary node removal that will be prevented by this annotation include:
- [Consolidation]({{<ref "#consolidation" >}})
- [Drift]({{<ref "#drift" >}})
- Expiration

{{% alert title="Note" color="primary" %}}
Voluntary node removal does not include [Interruption]({{<ref "#interruption" >}}) or manual deletion initiated through `kubectl delete node`. Both of these are considered involuntary events, since node removal cannot be delayed.
The `do-not-disrupt` annotation does **not** exclude nodes from involuntary disruption methods, i.e. [Expiration]({{<ref "#expiration" >}}), [Interruption]({{<ref "#interruption" >}}), and manual deletion (e.g. `kubectl delete node ...`).
{{% /alert %}}

### Node-Level Controls

You can block Karpenter from voluntarily choosing to disrupt certain nodes by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the node. This will prevent disruption actions on the node.
You can block Karpenter from voluntarily choosing to disrupt certain nodes by setting the `karpenter.sh/do-not-disrupt: "true"` annotation on the node.
This will prevent voluntary disruption actions against the node.

```yaml
apiVersion: v1
Expand Down
4 changes: 4 additions & 0 deletions website/content/en/docs/upgrading/upgrade-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,13 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced
* API Rename: NodePool’s ConsolidationPolicy `WhenUnderutilized` is now renamed to `WhenEmptyOrUnderutilized`
* Behavior Changes:
* Expiration is now forceful and begins draining as soon as it’s expired. Karpenter does not wait for replacement capacity to be available before draining, but will start provisioning a replacement as soon as the node is expired and begins draining.
* Pods with the `karpenter.sh/do-not-disrupt` annotation now block node termination. Termination of a node with these pods will be blocked until those pods are removed, enter a terminating or terminal state, or the NodeClaims's TerminationGracePeriod has expired.
* Karpenter's generated NodeConfig now takes precedence when generating UserData with the AL2023 `amiFamily`. If you're setting any values managed by Karpenter in your AL2023 UserData, configure these through Karpenter natively (e.g. kubelet configuration fields).
* Karpenter now adds a `karpenter.sh/unregistered:NoExecute` taint to nodes in injected UserData when using alias in AMISelectorTerms or non-Custom AMIFamily. When using `amiFamily: Custom`, users will need to add this taint into their UserData, where Karpenter will automatically remove it when provisioning nodes.
* Discovered standard AL2023 AMIs will no longer be considered compatible with GPU / accelerator workloads. If you're using an AL2023 EC2NodeClass (without AMISelectorTerms) for these workloads, you will need to select your AMI via AMISelectorTerms (non-alias).
* Karpenter now waits for underlying instances to be completely terminated before removing the associated nodes. This means it may take longer for nodes to be deleted and for nodeclaims to get cleaned up.
* NodePools now have [status conditions]({{< relref "../concepts/nodepools/#statusconditions" >}}) that indicate if they are ready. If not, then they will not be considered during scheduling.
* NodeClasses now have [status conditions]({{< relref "../concepts/nodeclasses/#statusconditions" >}}) that indicate if they are ready. If they are not ready, NodePools that reference them through their `nodeClassRef` will not be considered during scheduling.
* API Moves:
* ExpireAfter has moved from the `NodePool.Spec.Disruption` block to `NodePool.Spec.Template.Spec`, and is now a drift-able field.
* `Kubelet` was moved to the EC2NodeClass from the NodePool.
Expand Down
23 changes: 12 additions & 11 deletions website/content/en/docs/upgrading/v1-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,20 +75,20 @@ The upgrade guide will first require upgrading to your latest patch version prio
--set webhook.port=8443
```
{{% alert title="Note" color="warning" %}}
If you receive a `label validation error` or `annotation validation error` consult the [troubleshooting guide]({{<ref "../troubleshooting/#helm-error-when-installing-the-karpenter-crd-chart" >}}) for steps to resolve.
If you receive a `label validation error` or `annotation validation error` consult the [troubleshooting guide]({{<ref "../troubleshooting/#helm-error-when-installing-the-karpenter-crd-chart" >}}) for steps to resolve.
{{% /alert %}}

{{% alert title="Note" color="warning" %}}

As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:
As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:

```bash
```bash
export SERVICE_NAME=<karpenter webhook service name>
export SERVICE_NAMESPACE=<karpenter webhook service namespace>
export SERVICE_PORT=<karpenter webhook service port>
# NodePools
kubectl patch customresourcedefinitions nodepools.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# NodeClaims
# NodeClaims
kubectl patch customresourcedefinitions nodeclaims.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# EC2NodeClass
kubectl patch customresourcedefinitions ec2nodeclasses.karpenter.k8s.aws -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
Expand Down Expand Up @@ -142,20 +142,20 @@ kubectl patch customresourcedefinitions ec2nodeclasses.karpenter.k8s.aws -p "{\"
```

{{% alert title="Note" color="warning" %}}
If you receive a `label validation error` or `annotation validation error` consult the [troubleshooting guide]({{<ref "../troubleshooting/#helm-error-when-installing-the-karpenter-crd-chart" >}}) for steps to resolve.
If you receive a `label validation error` or `annotation validation error` consult the [troubleshooting guide]({{<ref "../troubleshooting/#helm-error-when-installing-the-karpenter-crd-chart" >}}) for steps to resolve.
{{% /alert %}}

{{% alert title="Note" color="warning" %}}

As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:
As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:

```bash
```bash
export SERVICE_NAME=<karpenter webhook service name>
export SERVICE_NAMESPACE=<karpenter webhook service namespace>
export SERVICE_PORT=<karpenter webhook service port>
# NodePools
kubectl patch customresourcedefinitions nodepools.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# NodeClaims
# NodeClaims
kubectl patch customresourcedefinitions nodeclaims.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# EC2NodeClass
kubectl patch customresourcedefinitions ec2nodeclasses.karpenter.k8s.aws -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
Expand Down Expand Up @@ -368,15 +368,15 @@ helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-cr

{{% alert title="Note" color="warning" %}}

As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:
As an alternative approach to updating the Karpenter CRDs conversion webhook configuration, you can patch the CRDs as follows:

```bash
```bash
export SERVICE_NAME=<karpenter webhook service name>
export SERVICE_NAMESPACE=<karpenter webhook service namespace>
export SERVICE_PORT=<karpenter webhook service port>
# NodePools
kubectl patch customresourcedefinitions nodepools.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# NodeClaims
# NodeClaims
kubectl patch customresourcedefinitions nodeclaims.karpenter.sh -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
# EC2NodeClass
kubectl patch customresourcedefinitions ec2nodeclasses.karpenter.k8s.aws -p "{\"spec\":{\"conversion\":{\"webhook\":{\"clientConfig\":{\"service\": {\"name\": \"${SERVICE_NAME}\", \"namespace\": \"${SERVICE_NAMESPACE}\", \"port\":${SERVICE_PORT}}}}}}}"
Expand Down Expand Up @@ -411,6 +411,7 @@ Karpenter should now be pulling and operating against the v1beta1 APIVersion as
* API Rename: NodePool’s ConsolidationPolicy `WhenUnderutilized` is now renamed to `WhenEmptyOrUnderutilized`
* Behavior Changes:
* Expiration is now forceful and begins draining as soon as it’s expired. Karpenter does not wait for replacement capacity to be available before draining, but will start provisioning a replacement as soon as the node is expired and begins draining.
* Pods with the `karpenter.sh/do-not-disrupt` annotation now block node termination. Termination of a node with these pods will be blocked until those pods are removed, enter a terminating or terminal state, or the NodeClaims's TerminationGracePeriod has expired.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we link out to the section of the docs that describes TerminationGracePeriod here?

* Karpenter's generated NodeConfig now takes precedence when generating UserData with the AL2023 `amiFamily`. If you're setting any values managed by Karpenter in your AL2023 UserData, configure these through Karpenter natively (e.g. kubelet configuration fields).
* Karpenter now adds a `karpenter.sh/unregistered:NoExecute` taint to nodes in injected UserData when using alias in AMISelectorTerms or non-Custom AMIFamily. When using `amiFamily: Custom`, users will need to add this taint into their UserData, where Karpenter will automatically remove it when provisioning nodes.
* Discovered standard AL2023 AMIs will no longer be considered compatible with GPU / accelerator workloads. If you're using an AL2023 EC2NodeClass (without AMISelectorTerms) for these workloads, you will need to select your AMI via AMISelectorTerms (non-alias).
Expand Down
Loading