Add support for GKE image streaming feature

[pentago]

Description

Add support for image streaming configuration as per feature announcement blog post.

New or Affected Resource(s)

• google_container_node_pool (maybe even google_container_cluster)

Potential Terraform Configuration

  node_config {
    image_streaming = enable
  }

References

Feature anouncement: https://cloud.google.com/blog/products/containers-kubernetes/introducing-container-image-streaming-in-gke

b/299312474

[rahul-mereddy]

when can this feature be added ?

[pentago]

when can this feature be added ?

It's released in one of previous releases.

[GregoireW]

There is strange thing around this...

https://cloud.google.com/blog/products/containers-kubernetes/introducing-container-image-streaming-in-gke reference an option at cluster level (and only at this level). And there is one displayed on the console.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#gcfs_config reference an option at nodepool level. Not sure how to display it, but I guess it exists...

If you set the gcfs option on your node pool and execute terraform, the "image streaming" option on the cluster is still disabled ... but kube event display some ImageStreaming activities.

So yes a little bit disturbing.

[pentago]

Logcal to me is that images are downloaded to nodes which actually run workloads compared to control plane which doesn't.

[cedws]

@pentago Based on what @GregoireW has said and my own findings it doesn't look like this is supported. At least, there is no image_streaming argument for google_container_node_pool, even in the beta provider. Would you mind clarifying?

[pentago]

Its called differently,something like *gcfs. Go through provider release changelog and you should find it.

[pentago]

hashicorp/terraform-provider-google-beta#3828

Added in 4.1.0

[GregoireW]

@cedws to be clear it works if you set the gcfs option on the node.

@pentago I just read the documentation, I guess there was an update few weeks back (or I did not read it correctly), now it is clearly stated the pool inherit the cluster option if it is set. https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming#enable_on_clusters

So yes, the terraform option is ok, (well, the name is not streaming so perhaps confusing ) A small tip on the cluster resource stating terraform will not set the streaming option globally but only on node pool may also be welcome.

[mrvarmazyar]

Hello everyone,
I put this configuration on my cluster level, and every time I apply the code, the cluster should be destroyed and recreated again, which doesn't make any sense.

  node_config {
    gcfs_config {
      enabled = true
    }
  }

I tried to deploy a cluster from scratch to test this feature, but I realized that this would not enable this feature on my default node pool.
To give you more context, I'm using node auto-provisioning on the GKE cluster, and it is supposed to be supported on the new provisioned nodes.

Any suggestion or tips to have this feature will be appreciated.

[GregoireW]

@mrvarmazyar

The google provider state (from here https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster#node_pool )

Warning: node pools defined inside a cluster can't be changed (or added/removed) after cluster creation without deleting and recreating the entire cluster.

So ... simply do not set node pool inside the cluster definition but use the recommended way and use additional(s) google_container_node_pool resource. Some update will still need to drop your pool and recreate a new one, but your cluster will not be destroyed.

By auto-provisioning I guess you means https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_node_pool#nested_autoscaling it should work without any issue (It works as expected on one of my cluster with this kind of setup) if all the requirements are ok : https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming#requirements

[mrvarmazyar]

@GregoireW First of all thanks for your quick response. I suppose it would be more helpful if you see my GKE config. I tried to follow the best practices:

resource "google_container_cluster" "mycluster" {
  remove_default_node_pool = true
  initial_node_count       = 1
  node_config {
    gcfs_config {
      enabled = true
    }
  }
  cluster_autoscaling {
    enabled             = true
    autoscaling_profile = "OPTIMIZE_UTILIZATION"
    .
    .
    .
    }
    .
    .
}

resource "google_container_node_pool" "mybaseline" {
  initial_node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 5
  }
  node_config {
    image_type      = "COS_CONTAINERD"
    .
    .
    .
    }
    .
    .
    .
}

This configuration is supposed to enable the image streaming on the cluster level and the Node auto provisioning follow that configuration, but the problem is this configuration doesn't work as expected. every time I get this result:

          + gcfs_config { # forces replacement
              + enabled = true # forces replacement
            }


            # (2 unchanged blocks hidden)
        }

        # (3 unchanged blocks hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

There are two questions and it would be great if you can answer them:

• The node auto-provisioning deploys new nodes with the image type of COS, not COS_CONTAINERD, how I can configure that in this case?
• What's the proper way to enable the gcfs on cluster level because in that way the new nodes are going to be provisioned with this feature.

Thanks a lot

[GregoireW]

@mrvarmazyar

The node_config part in google_container_cluster create a default node pool and set the option you give on it. It is not a default configuration for others pool. You can remote it.

Terraform set the gcfs configuration at node level, not cluster level, so you have to :

resource "google_container_node_pool" "mybaseline" {
  initial_node_count = 1

  autoscaling {
    min_node_count = 1
    max_node_count = 5
  }

  gcfs_config {
    enabled = true
  }

  node_config {
    image_type      = "COS_CONTAINERD"
    .
    .
    .
    }
    .
    .
    .
}

The pool should reference an image type Container-Optimized OS with Containerd (cos_containerd) The nodes will reference OS image as Container-Optimized OS from Google

Additional note: the feature display at cluster level Image streaming will still be Disabled but the pool will use image streaming.

[mcfedr]

This option is a bit weirdly implemented, as via the google apis, its clearly possible to update an existing node pool, but terraform seems this as a change that forces replacement

[rahul-mereddy]

@GregoireW I'm using https://github.com/terraform-google-modules/terraform-google-kubernetes-engine to build our GKE cluster and nodepool and can't add below strucutre.

is it something in the future that is going to support for enabiling image streaming and gcfs on the cluster level or nodepool level ?

gcfs_config {
enabled = true
}

node_config {
image_type = "COS_CONTAINERD"

[GregoireW]

@rahul-mereddy I'm not affiliated to google nor hashicorp. I'm just a user of this module. but this feature already exist on the nodepool level.
Check your provider version as from a previous comment, it has been added in 4.1.0 ( in the beta provider I guess, not sure which version on the standard one )

[otherguy]

It would still be nice to be able to set it on the cluster level so the corresponding UI option shows Enabled.

[ynwang007]

Current solution only support enabling image streaming on node pool level for a manually managed node pool (not via GKE Node Auto Provisioning. Unfortunately, for NAP, it does't support enabling image streaming at the node pool level yet, so for NAP user, current solution won't work.

GKE allows enabling image streaming on the cluster level by setting node_pool_defaults field when creating/updating the cluster. We should introduce node_pool_defaults field to google_container_cluster.

[NissesSenap]

Is this still an issue?
I tried to look at the documentation around this, but it wasn't great.
Anyone has an example on how to enable image streaming?

Or is still some feature that is missing?

[muffl0n]

I just configured

  node_pool_defaults {
    node_config_defaults {
      gcfs_config {
        enabled = true
      }
    }
  }

in resource google_container_cluster. After applying GKE rotated all nodes in all pools. These new nodes support image streaming.

Don't forget to enable API containerfilesystem.googleapis.com (see https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming#requirements)!

[otherguy]

@muffl0n that's indeed how you enable it! But unfortunately my comment (#10509 (comment)) still applies, as the cluster-wide flag for image streaming does not get set with this.

[dyu-bot]

Hey is anyone having issues with this? I tried the exact same config that @muffl0n used. The google_container_cluster got updated with that node_pool_defaults and I see "Image Streaming: Enabled" now on the console under cluster features section. However, the running nodes were not replaced.

It appears though replacement should be automatically triggered?

[dyu-bot]

Another issue probably not related to this terraform provider but GKE itself - I noticed even when image streaming is enabled at cluster level using node_pool_defaults, the nodepools still have image streaming disabled.

[Seelam-Ramesh-Reddy]

Yes I have enabled it on node level but When i view under features It still says disabled.