Layer 7 Traffic Management CiliumEnvoyConfig Invalid

[haoyann]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I ues Layer 7 Traffic Management Demo and add Envoy load-balancing and URL re-writing , but it does not work.

kubectl apply -f envoy-test.yaml

kubectl exec -it -n cilium-test $CLIENT2 -- curl -v echo-same-node:8080/foo,
It looks like the path not re-writing the result is still 403.

*   Trying 10.103.160.238:8080...
* Connected to echo-same-node (10.103.160.238) port 8080 (#0)
> GET /foo HTTP/1.1
> Host: echo-same-node:8080
> User-Agent: curl/7.78.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 15
< content-type: text/plain
< date: Wed, 15 Jun 2022 08:14:43 GMT
< server: envoy
<
Access denied
* Connection #0 to host echo-same-node left intact

This is agent log, for this reason? [gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment

level=info msg="[  response trailer map: 152 bytes: grpc-message,grpc-status" subsys=envoy-main threadID=687
level=info msg="[runtime: layers:" subsys=envoy-main threadID=687
level=info msg="  - name: static_layer_0" subsys=envoy-main threadID=687
level=info msg="    static_layer:" subsys=envoy-main threadID=687
level=info msg="      overload:" subsys=envoy-main threadID=687
level=info msg="        global_downstream_max_connections: 50000" subsys=envoy-main threadID=687
level=info msg="[admin address: /var/run/cilium/envoy-admin.sock" subsys=envoy-admin threadID=687
level=info msg="[loading tracing configuration" subsys=envoy-config threadID=687
level=info msg="[loading 0 static secret(s)" subsys=envoy-config threadID=687
level=info msg="[loading 6 cluster(s)" subsys=envoy-config threadID=687
level=info msg="[loading 0 listener(s)" subsys=envoy-config threadID=687
level=info msg="[loading stats configuration" subsys=envoy-config threadID=687
level=info msg="[RTDS has finished initialization" subsys=envoy-runtime threadID=687
level=info msg="[cm init: initializing cds" subsys=envoy-upstream threadID=687
level=info msg="[starting main dispatch loop" subsys=envoy-main threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=1
level=info msg="[cds: add 2 cluster(s), remove 6 cluster(s)" subsys=envoy-upstream threadID=687
level=info msg="[cds: added/updated 2 cluster(s), skipped 0 unmodified cluster(s)" subsys=envoy-upstream threadID=687
level=info msg="[cm init: initializing secondary clusters" subsys=envoy-upstream threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=2
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=3
level=warning msg="[gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" subsys=envoy-config threadID=687
level=warning msg="[gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" subsys=envoy-config threadID=687
level=info msg="[cm init: all clusters initialized" subsys=envoy-upstream threadID=687
level=info msg="[all clusters initialized. initializing init manager" subsys=envoy-main threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=4
level=info msg="[lds: add/update listener 'envoy-prometheus-metrics-listener'" subsys=envoy-upstream threadID=687
level=info msg="Envoy: Accepted access log connection" subsys=envoy-manager
level=info msg="[lds: add/update listener 'envoy-lb-listener'" subsys=envoy-upstream threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=5
level=info msg="Adding new proxy port rules for envoy-lb-listener:18465" proxy port name=envoy-lb-listener subsys=proxy
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=6
level=info msg="[all dependencies initialized. starting workers" subsys=envoy-config threadID=687

Cilium Version

v1.12.0-rc1

Kernel Version

Linux master 5.18.2-1.el7.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Jun 4 09:07:48 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.23.6

Sysdump

cilium-sysdump-20220615-162112.zip

Relevant log output

level=info msg="[  response trailer map: 152 bytes: grpc-message,grpc-status" subsys=envoy-main threadID=687
level=info msg="[runtime: layers:" subsys=envoy-main threadID=687
level=info msg="  - name: static_layer_0" subsys=envoy-main threadID=687
level=info msg="    static_layer:" subsys=envoy-main threadID=687
level=info msg="      overload:" subsys=envoy-main threadID=687
level=info msg="        global_downstream_max_connections: 50000" subsys=envoy-main threadID=687
level=info msg="[admin address: /var/run/cilium/envoy-admin.sock" subsys=envoy-admin threadID=687
level=info msg="[loading tracing configuration" subsys=envoy-config threadID=687
level=info msg="[loading 0 static secret(s)" subsys=envoy-config threadID=687
level=info msg="[loading 6 cluster(s)" subsys=envoy-config threadID=687
level=info msg="[loading 0 listener(s)" subsys=envoy-config threadID=687
level=info msg="[loading stats configuration" subsys=envoy-config threadID=687
level=info msg="[RTDS has finished initialization" subsys=envoy-runtime threadID=687
level=info msg="[cm init: initializing cds" subsys=envoy-upstream threadID=687
level=info msg="[starting main dispatch loop" subsys=envoy-main threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=1
level=info msg="[cds: add 2 cluster(s), remove 6 cluster(s)" subsys=envoy-upstream threadID=687
level=info msg="[cds: added/updated 2 cluster(s), skipped 0 unmodified cluster(s)" subsys=envoy-upstream threadID=687
level=info msg="[cm init: initializing secondary clusters" subsys=envoy-upstream threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=2
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=3
level=warning msg="[gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" subsys=envoy-config threadID=687
level=warning msg="[gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" subsys=envoy-config threadID=687
level=info msg="[cm init: all clusters initialized" subsys=envoy-upstream threadID=687
level=info msg="[all clusters initialized. initializing init manager" subsys=envoy-main threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=4
level=info msg="[lds: add/update listener 'envoy-prometheus-metrics-listener'" subsys=envoy-upstream threadID=687
level=info msg="Envoy: Accepted access log connection" subsys=envoy-manager
level=info msg="[lds: add/update listener 'envoy-lb-listener'" subsys=envoy-upstream threadID=687
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=5
level=info msg="Adding new proxy port rules for envoy-lb-listener:18465" proxy port name=envoy-lb-listener subsys=proxy
level=info msg="starting xDS stream processing" subsys=xds xdsStreamID=6
level=info msg="[all dependencies initialized. starting workers" subsys=envoy-config threadID=687

Anything else?

install command

helm install cilium cilium/cilium --version v1.12.0-rc1 \
   --namespace kube-system \
   --set prometheus.enabled=true \
   --set operator.prometheus.enabled=true \
   --set hubble.enabled=true \
   --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
   --set enableIngressController=true \
   --set kubeProxyReplacement=probe \
   --set annotateK8sNode=true 

cilium status

    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium-operator    Running: 2
                  hubble-relay       Running: 1
                  cilium             Running: 3
Cluster Pods:     7/7 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.0-rc1: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.0-rc1: 2
                  hubble-relay       quay.io/cilium/hubble-relay:v1.12.0-rc1: 1

Code of Conduct

• I agree to follow this project's Code of Conduct