HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

BadLiveware · 2024-09-17T09:20:02Z

Description:
I have 3 different gateways up and running, only 2 of which are used in this issue.
Only one gateway is shown in status, while both are configured. This results in things like external-dns only setting up DNS records for one gateway address.

Repro steps:
Overview:

─❯ kubectl get envoyproxy -A
NAMESPACE       NAME       AGE
envoy-gateway   cluster    2d22h
envoy-gateway   external   2d22h
envoy-gateway   internal   2d22h

─❯ kubectl get gatewayclass -A
NAME       CONTROLLER                                      ACCEPTED   AGE
cluster    gateway.envoyproxy.io/gatewayclass-controller   True       2d22h
external   gateway.envoyproxy.io/gatewayclass-controller   True       2d22h
internal   gateway.envoyproxy.io/gatewayclass-controller   True       13d

─❯ kubectl get gateway -A
NAMESPACE       NAME       CLASS      ADDRESS          PROGRAMMED   AGE
envoy-gateway   cluster    cluster    10.182.165.187   True         2d22h
envoy-gateway   external   external   10.182.83.109    True         2d22h
envoy-gateway   internal   internal   10.180.0.60      True         13d

─❯ kubectl get httproute -A
NAMESPACE     NAME          HOSTNAMES                                                         AGE
dev-ops       webapi        ["webapi.dev-ops.c.example.org","webapi.dev-ops.i.example.org"]   2d22h # This is attached to internal and cluster
echo-server   echo-server   ["echo-server-gw.dev.example.org"]                                14d # This is attached to external

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal
  namespace: envoy-gateway
spec:
  gatewayClassName: internal
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.i.example.org'
    name: http
    port: 80
    protocol: HTTP
status:
  addresses:
  - type: IPAddress
    value: 10.180.0.60
  conditions:
  - lastTransitionTime: "2024-09-14T11:07:57Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 8
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-14T11:09:29Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 8
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 8
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener has been successfully translated
      observedGeneration: 8
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener references have been resolved
      observedGeneration: 8
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cluster
  namespace: envoy-gateway
spec:
  gatewayClassName: cluster
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: '*.c.example.org'
    name: http
    port: 80
    protocol: HTTP
status:
  addresses:
  - type: IPAddress
    value: 10.182.165.187
  conditions:
  - lastTransitionTime: "2024-09-14T11:07:56Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 3
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-09-14T11:08:37Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 3
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Sending translated listener configuration to the data plane
      observedGeneration: 3
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener has been successfully translated
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-14T11:07:56Z"
      message: Listener references have been resolved
      observedGeneration: 3
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
    - group: gateway.networking.k8s.io
      kind: GRPCRoute

Same HTTPRoute switches its status between gateways

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: webapi
  namespace: dev-ops
spec:
  hostnames:
  - webapi.dev-ops.c.example.org
  - webapi.dev-ops.i.example.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: cluster
    namespace: envoy-gateway
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: internal
    namespace: envoy-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-auth
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /auth
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-browse
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /browse
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-dev
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /dev
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-discover
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /discover
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-images
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /images
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-marketing
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /marketing
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-member
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /member
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-myexample
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /myexample
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-selling
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /selling
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-shopping
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /shopping
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-09-17T08:00:53Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-17T08:00:53Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: internal
      namespace: envoy-gateway

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: webapi
  namespace: dev-ops
spec:
  hostnames:
  - webapi.dev-ops.c.example.org
  - webapi.dev-ops.i.example.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: cluster
    namespace: envoy-gateway
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: internal
    namespace: envoy-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-auth
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /auth
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-browse
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /browse
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-dev
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /dev
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-discover
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /discover
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-images
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /images
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-marketing
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /marketing
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-member
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /member
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-myexample
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /myexample
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-selling
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /selling
  - backendRefs:
    - group: ""
      kind: Service
      name: webapi-shopping
      port: 80
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: ""
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: /shopping
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-09-17T08:31:21Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-09-17T08:31:21Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: cluster
      namespace: envoy-gateway

(Part) Config dump to show both gateways are actually configured:
internal:

        {
          "@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
          "dynamicRouteConfigs": [
            {
              "lastUpdated": "2024-09-14T11:09:04.604Z",
              "routeConfig": {
                "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
                "ignorePortInHostMatching": true,
                "name": "envoy-gateway/internal/http",
                "virtualHosts": [
                  {
                    "domains": [
                      "webapi.dev-ops.i.example.org"
                    ],
                    "metadata": {
                      "filterMetadata": {
                        "envoy-gateway": {
                          "resources": [
                            {
                              "kind": "Gateway",
                              "name": "internal",
                              "namespace": "envoy-gateway",
                              "sectionName": "http"
                            }
                          ]
                        }
                      }
                    },
                    "name": "envoy-gateway/internal/http/webapi_dev-ops_i_example_org",
                    "routes": [
                      {
                        "match": {
                          "pathSeparatedPrefix": "/marketing"
                        },
                        "metadata": {
                          "filterMetadata": {
                            "envoy-gateway": {
                              "resources": [
                                {
                                  "kind": "HTTPRoute",
                                  "name": "webapi",
                                  "namespace": "dev-ops"
                                }
                              ]
                            }
                          }
                        },
                        "name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_i_example_org",
                        "route": {
                          "cluster": "httproute/dev-ops/webapi/rule/5",
                          "regexRewrite": {
                            "pattern": {
                              "regex": "^/marketing\\/*"
                            },
                            "substitution": "/"
                          },
                          "upgradeConfigs": [
                            {
                              "upgradeType": "websocket"
                            }
                          ]
                        }

cluster

        {
          "@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
          "dynamicRouteConfigs": [
            {
              "lastUpdated": "2024-09-14T11:08:30.023Z",
              "routeConfig": {
                "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
                "ignorePortInHostMatching": true,
                "name": "envoy-gateway/cluster/http",
                "virtualHosts": [
                  {
                    "domains": [
                      "webapi.dev-ops.c.example.org"
                    ],
                    "metadata": {
                      "filterMetadata": {
                        "envoy-gateway": {
                          "resources": [
                            {
                              "kind": "Gateway",
                              "name": "cluster",
                              "namespace": "envoy-gateway",
                              "sectionName": "http"
                            }
                          ]
                        }
                      }
                    },
                    "name": "envoy-gateway/cluster/http/webapi_dev-ops_c_example_org",
                    "routes": [
                      {
                        "match": {
                          "pathSeparatedPrefix": "/marketing"
                        },
                        "metadata": {
                          "filterMetadata": {
                            "envoy-gateway": {
                              "resources": [
                                {
                                  "kind": "HTTPRoute",
                                  "name": "webapi",
                                  "namespace": "dev-ops"
                                }
                              ]
                            }
                          }
                        },
                        "name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_c_example.org",
                        "route": {
                          "cluster": "httproute/dev-ops/webapi/rule/5",
                          "regexRewrite": {
                            "pattern": {
                              "regex": "^/marketing\\/*"
                            },
                            "substitution": "/"
                          },
                          "upgradeConfigs": [
                            {
                              "upgradeType": "websocket"
                            }
                          ]
                        }

Environment:
GKE 1.29
Envoy Gateway v1.1.0
Upgraded to envoy gateway v1.1.1 after writing this issue, and it remains an issue

Logs:
downloaded-logs-20240917-102549.csv

BadLiveware · 2024-09-17T09:36:04Z

Tried removing the extraneous resources, and it didnt help.

egctl status

❯ egctl x status all -v -A
NAME                    TYPE       STATUS    REASON     MESSAGE              OBSERVED GENERATION   LAST TRANSITION TIME
gatewayclass/cluster    Accepted   True      Accepted   Valid GatewayClass   1                     2024-09-14 12:00:45 +0200 CEST
gatewayclass/internal   Accepted   True      Accepted   Valid GatewayClass   2                     2024-09-14 12:00:45 +0200 CEST

NAMESPACE       NAME               TYPE         STATUS    REASON       MESSAGE                                                                    OBSERVED GENERATION   LAST TRANSITION TIME
envoy-gateway   gateway/cluster    Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-09-17 11:24:37 +0200 CEST
                                   Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-09-17 11:24:27 +0200 CEST
envoy-gateway   gateway/internal   Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-09-17 11:28:25 +0200 CEST
                                   Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-09-17 11:25:23 +0200 CEST

NAMESPACE   NAME               TYPE           STATUS    REASON         MESSAGE                                            OBSERVED GENERATION   LAST TRANSITION TIME
dev-ops     httproute/webapi   ResolvedRefs   True      ResolvedRefs   Resolved all the Object references for the Route   1                     2024-09-17 11:34:22 +0200 CEST
                               Accepted       True      Accepted       Route is accepted                                  1                     2024-09-17 11:34:22 +0200 CEST

Error: no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"
no matches for kind "BackendTLSPolicy" in version "gateway.networking.k8s.io/v1alpha2"

arkodg · 2024-09-17T21:49:08Z

another bug in the area of parentRef status #4127

arkodg · 2024-10-18T00:58:55Z

@zhaohuabing is this fixed ?

zhaohuabing · 2024-10-18T01:07:14Z

@arkodg Looks like this was caused by the same reason which has been fixed in #4336.

@BadLiveware Can you help confirm if this issue still exists in the latest EG?

BadLiveware · 2024-10-18T04:10:51Z

@BadLiveware Can you help confirm if this issue still exists in the latest EG?

@zhaohuabing Still an issue on v1.1.2 or do you mean v0.0.0-latest?

zhaohuabing · 2024-10-19T07:39:32Z

@BadLiveware Can you help confirm if this issue still exists in the latest EG?

@zhaohuabing Still an issue on v1.1.2 or do you mean v0.0.0-latest?

v0.0.0-latest, thanks.

zhaohuabing · 2024-10-22T08:50:34Z

@BadLiveware Did you get chance to test it on v0.0.0-latest?

BadLiveware · 2024-10-24T17:44:32Z

@zhaohuabing Yes, it still behaves the same, configures both gateways but only sets one in status

zhaohuabing · 2024-10-30T09:04:38Z

I reproduced this issue in my dev machine. The two Gateways belong to different GatewayClasses, so they were translated separately. As a result, the HTTPRoute parent status were overriden by the later translation.

This issue also exists for other Route types.

arkodg · 2024-10-30T17:13:22Z

thanks for triaging this one @zhaohuabing, to fix this issue, we'll need to not override the entire status.Parents field

gateway/internal/provider/kubernetes/status.go

Line 77 in b0c6f8c

hCopy.Status.Parents = val.Parents

and perform a Patch rather than a Update

gateway/internal/provider/kubernetes/status_updater.go

Line 110 in b0c6f8c

return u.client.Status().Update(context.Background(), newObj)

relates to kubernetes-sigs/gateway-api#1629

BadLiveware added the triage label Sep 17, 2024

arkodg added kind/bug Something isn't working help wanted Extra attention is needed and removed triage labels Sep 17, 2024

arkodg added this to the v1.2.0-rc1 milestone Sep 17, 2024

zhaohuabing mentioned this issue Sep 26, 2024

Some status updates are lost #4336

Closed

arkodg modified the milestones: v1.2.0-rc1, v1.2.0 Oct 18, 2024

zhaohuabing self-assigned this Oct 30, 2024

zhaohuabing mentioned this issue Oct 30, 2024

Fix: HTTPRoute status only shows one parent when targeting multiple Gateways from different Gatewayclasses #4575

Closed

zhaohuabing mentioned this issue Oct 31, 2024

fix: HTTPRoute status only shows one parent when targeting multiple Gateways from different GatewayClasses #4587

Merged

zhaohuabing closed this as completed in #4587 Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

BadLiveware commented Sep 17, 2024 •

edited

Loading

BadLiveware commented Sep 17, 2024

arkodg commented Sep 17, 2024

arkodg commented Oct 18, 2024

zhaohuabing commented Oct 18, 2024 •

edited

Loading

BadLiveware commented Oct 18, 2024 •

edited

Loading

zhaohuabing commented Oct 19, 2024

zhaohuabing commented Oct 22, 2024

BadLiveware commented Oct 24, 2024

zhaohuabing commented Oct 30, 2024 •

edited

Loading

arkodg commented Oct 30, 2024

HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264

Comments

BadLiveware commented Sep 17, 2024 • edited Loading

BadLiveware commented Sep 17, 2024

arkodg commented Sep 17, 2024

arkodg commented Oct 18, 2024

zhaohuabing commented Oct 18, 2024 • edited Loading

BadLiveware commented Oct 18, 2024 • edited Loading

zhaohuabing commented Oct 19, 2024

zhaohuabing commented Oct 22, 2024

BadLiveware commented Oct 24, 2024

zhaohuabing commented Oct 30, 2024 • edited Loading

arkodg commented Oct 30, 2024

BadLiveware commented Sep 17, 2024 •

edited

Loading

zhaohuabing commented Oct 18, 2024 •

edited

Loading

BadLiveware commented Oct 18, 2024 •

edited

Loading

zhaohuabing commented Oct 30, 2024 •

edited

Loading