Gateway API: Revert Gateway and HTTPRoute objects from v1 to v1beta1

[abursavich]

Description

This reverts Kubernetes API interactions for Gateway and HTTPRoute objects from APIVersion v1 to v1beta1.

Fixes #4366

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

Hi @abursavich. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mloiseleur]

Thanks for this PR @abursavich
I'm wondering: why should we revert ?
Gateway API is still a WIP and, eventually, GKE will upgrade to it.

Maybe I missed something: so far, I understand that if we merge this PR, users using v1 for Gateway and HTTPRoute will be impacted and won't be able to use this version anymore.

[abursavich]

@mloiseleur, it's all rather confusing. This is the best way to support the most users, including v1 users.

There are two different release channels (Experimental and Standard). I tried to cover them a bit in the added documentation. The Experimental channel includes alpha, beta, and stable CRD versions. The Standard channel only includes beta and stable CRD versions. They matter for other details, but the channels aren't relevant to the main vbeta1 vs v1 issue here. That's mostly down to the fact that there are different versions released in each channel over time. Older releases in both channels are missing the v1 API.

The problem with GKE is that their Gateway API CRDs are pinned to the GKE cluster version. In order to get updates to GWAPI CRDs, users need to upgrade their GKE clusters. Some external-dns users are unable or unwilling to do that. So they're stuck with GWAPI CRDs that only contain v1beta1, from before v1 was released.

All versions of GWAPI CRD releases that contain the v1 API also contain the v1beta1 API. If you have v1 and v1beta1 installed the apiserver will transparently convert v1beta1 resources to v1 and back again, where the Kinds exist in both versions. Where the Kind exists in the v1beta1 API, it's safe to use v1beta1 instead of v1. If you only have v1beta1 installed you must use it instead of v1. So v1beta1 covers everyone for most cases.

GRPCRoute is an outlier. It's the reason for the "where the Kinds exist" caveats above. It has existed for longer in v1alpha2 (Experimental channel only) and now also exists in v1 (both channels). If we use v1alpha2 for GRPCRoute, we only support users on the Experimental channel. If we use v1 for GRPCRoute, then we don't support users of the older Experimental releases but we do support users of newer Experimental and Standard releases. We use v1 for GRPCRoute.

[mloiseleur]

Thanks, it's starting to become less blurry. I have a few comments.

The problem with GKE is that their Gateway API CRDs are pinned to the GKE cluster version. In order to get updates to GWAPI CRDs, users need to upgrade their GKE clusters. Some external-dns users are unable or unwilling to do that. So they're stuck with GWAPI CRDs that only contain v1beta1, from before v1 was released.

external-dns is maintained by very few people on their free time. When user does not / cannot upgrade its cluster, he can also use a previous version of external-dns. That's also what a user need to do when a specific regression has been introduced in the latest version for its infrastructure.

All versions of GWAPI CRD releases that contain the v1 API also contain the v1beta1 API.

I'm not sure of that. I'm aware of at least recent Traefik Proxy v3.1 coming with complete support of stable Gateway API v1. There is no v1beta1 in the recommended CRDs.

Instead of reverting now, and re-reverting in a few months, wdyt of (just) improving documentation about current situation and current known issues with Gateway API ?

[abursavich]

Regarding Traefik Proxy: Traefik & Kubernetes with Gateway API

This provider supports version v1.1.0 of the Gateway API specification.

It fully supports all HTTP core and some extended features, as well as the TCPRoute and TLSRoute resources from the Experimental channel.

The v1.1.0 referenced is a GWAPI release version and not an API version. The Experimental and Standard channels both contain the v1beta1 API at v1.1.0. You can see the CRD manifests here: https://github.com/kubernetes-sigs/gateway-api/tree/v1.1.0/config/crd

Here's more GWAPI docs on versioning: https://gateway-api.sigs.k8s.io/concepts/versioning/

Of interest:

Release Channels
[...]

Supported Versions
This project aims to provide support for a wide range of Kubernetes versions with consistent upgrade experiences across versions. To accomplish that, we commit to:
[...]
2. Ensure that all Standard Channel changes between v1beta1 and v1 are fully compatible and convertible.

The beta API will continue to exist for a long time, but it's being phased out functionally. Resources that already exist in v1beta1 will continue to be synced with their v1 counterparts, but new resources will jump from alpha to v1 skipping beta like GRPCRoute did.

external-dns is maintained by very few people on their free time.

Agreed! I implemented 99% of the Gateway API features in external-dns in my free time. Some others have added to it.

When user does not / cannot upgrade its cluster, he can also use a previous version of external-dns. That's also what a user need to do when a specific regression has been introduced in the latest version for its infrastructure. [...] Instead of reverting now, and re-reverting in a few months, wdyt of (just) improving documentation about current situation and current known issues with Gateway API ?

Personally, I would rather use v1 and be done with it. However, since I'm the de facto maintainer of GWAPI in external-dns, I try to check in occasionally and fix reasonable issues that people raise. I think using v1beta1 is reasonable since GKE is such a large provider and it's causing significant user friction. Dropping v1beta1 support and moving fully to v1 in external-dns will probably happen at some point, but it'll be at least a year from now.

[mloiseleur]

Wow. You are more than convincing 🙇.
If you want to join current maintainers, it's possible.
/ok-to-test
/lgtm

[abursavich]

The robot told me to:

/assign @johngmyers

[Raffo]

Thank you for the explanation @abursavich . I agree with your assessment and thankful for your thorough explanation.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Raffo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Raffo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alexstojda]

Thank you for this @abursavich 🙏🏼