snippets are not always working with 2 Ingress controllers #10802
Comments
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
needs-triage
|
I have 2 Ingress Controllers installed. One has Reproduction in kind: we need to values files: values-1.yaml: controller:
allowSnippetAnnotations: false
ingressClassResource:
name: nginx-1
ingressClass: nginx-1values-2.yaml: controller:
allowSnippetAnnotations: true
ingressClassResource:
name: nginx-2
ingressClass: nginx-2and 2 Ingress objects: ingress-1.yaml: apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-1
spec:
ingressClassName: nginx-1
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80ingress-2.yaml: apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-2
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
server_tokens off;
spec:
ingressClassName: nginx-2
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80Try to install everything: helm upgrade -i ingress-nginx-1 ingress-nginx/ingress-nginx --version 4.9.0 -f values-1.yaml
helm upgrade -i ingress-nginx-2 ingress-nginx/ingress-nginx --version 4.9.0 -f values-2.yaml
kubectl apply -f ingress-1.yaml
kubectl apply -f ingress-2.yamlThe last command produces the error: kubectl apply -f ingress-2.yaml
Error from server (BadRequest): error when creating "ingress-2.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx.ingress.kubernetes.io/server-snippet annotation cannot be used. Snippet directives are disabled by the Ingress administratorI think this is a veto from the wrong admission controller (from ingress-1 Ingress controller) |
klauserber
changed the title
|
/remove-kind bug Need data because test on minikube works |
k8s-ci-robot
added
needs-kind
|
ok, thank you @longwuyuan it is solved. Key was indeed to follow the howto to deploy multiple ingress controllers: https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#how-can-i-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster It was the missing Here is a full working example (testet in kind): values-1.yaml controller:
allowSnippetAnnotations: false
ingressClassResource:
enabled: true
name: nginx-1
controllerValue: mydomain.org/ingress-1
ingressClass: nginx-1
ingressClassByName: truevalues-2.yaml controller:
allowSnippetAnnotations: true
ingressClassResource:
enabled: true
name: nginx-2
controllerValue: mydomain.org/ingress-2
ingressClass: nginx-2
ingressClassByName: trueingress-1.yaml apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-1
spec:
ingressClassName: nginx-1
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80ingress-2.yaml apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-2
annotations:
nginx.ingress.kubernetes.io/server-snippet: |-
server_tokens off;
spec:
ingressClassName: nginx-2
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80create two namespaces kubectl create ns ingress-1
kubectl create ns ingress-2install ingress-nginx in both namespaces helm upgrade -i -n ingress-1 ingress-nginx-1 ingress-nginx/ingress-nginx --version 4.9.0 -f values-1.yaml
helm upgrade -i -n ingress-2 ingress-nginx-2 ingress-nginx/ingress-nginx --version 4.9.0 -f values-2.yamlcreate two ingress objects kubectl -n ingress-1 apply -f ingress-1.yaml
kubectl -n ingress-2 apply -f ingress-2.yaml |
|
awesome. Thanks 🙏 /close |
|
@longwuyuan: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened:
Deployment of a simple Ingress with a server snippet leads to the error:
See my next comment to reproduce it in kind -> #10802 (comment)
Solution -> #10802 (comment)
Ingress:
It is a fresh helm install, here are the values:
I have tried to set
controller.config.allow-snippet-annotations: "true"with the same result.What you expected to happen:
-snippetannotations are working as expected. This is a stripped down example. In reality I want to set the Annotationnginx.ingress.kubernetes.io/modsecurity-snippetNGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Kubernetes version (use
kubectl version):Environment:
uname -a): Linux gke-prod-edge-76c8d95c-1hm2 5.15.133+ Basic structure #1 SMP Fri Dec 1 13:04:48 UTC 2023 x86_64 Intel(R) Xeon(R) CPU @ 2.20GHz GenuineIntel GNU/LinuxPlease mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.helm ls -A | grep -i ingresshelm -n <ingresscontrollernamespace> get values <helmreleasename>kubectl describe ingressclasseskubectl -n <ingresscontrollernamespace> get all -A -o wide-> to much Informations for nowkubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>-> no Service hostPort is usedCurrent state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o widekubectl -n <appnamespace> describe ing <ingressname>If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag -> not applicable
Others:
kubectl describe ...of any custom configmap(s) created and in useHow to reproduce this issue:
I cannot reproduce is with a minimal kind based example. Still try it.
Anything else we need to know:
The text was updated successfully, but these errors were encountered: