New NGINX Ingress causes 503 if service name is too long

[WolfspiritM]

What happened?

I'm currently having the problem that short service names work for ingresses but long names don't work. I get a 503 with latest nginx ingress controller (version 1.4.0). The old version works fine. The reason is that they moved from using endpoints to endpointslices. Now for long service names I get:

W1028 10:07:44.588930       7 endpointslices.go:81] Error obtaining Endpoints for Service "vcluster-dev-bbbbbb/dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210cf06f": no object matching key "vcluster-dev-bbbbbb/dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210cf06f" in local store
W1028 10:07:44.588956       7 controller.go:1112] Service "vcluster-dev-bbbbbb/dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210cf06f" does not have any active Endpoint.

However the endpoint slice is called dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210kz7cm I guess the problem is that endpoint slices are supposed to be called like the service with a 5 letter suffix and that doesn't work. Not sure who generates that endpointslice name like that and if it's a problem with syncing by vcluster. Moving back to a previous version of ingress-nginx makes it work again as they're using endpoints instead of slices there.

I think I found the problem. That's their code here:
https://github.com/kubernetes/ingress-nginx/blob/3579ed04870c77979ec5bf18f4cd00c8763615a1/internal/ingress/controller/store/endpointslice.go#L36
It seems like they match by "prefix" first and then check if the label matches.

If the endpoint slice is too long then it seems like it gets truncated with a value in the end and the prefix check doesn't work anymore.

For example this is working:
Service: vault-x-vault-x-mgmt
Endpoint: vault-x-vault-x-mgmt
Endpointslice: vault-x-vault-x-mgmt-cpp2r

For example this is NOT working:
Service: dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210cf06f
Endpoint: dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210cf06f
Endpointslice: dev-bbbbbb-product-manual-x-dev-bbbbbb-product-manua-30210kz7cm

Here the "prefix" check doesn't work.

I'm not sure if this is a general issue with kubernetes (nginx ingress) but even if it is then the long names caused by vcluster might make it happen way more often.

What did you expect to happen?

Ingress works with short and long names

How can we reproduce it (as minimally and precisely as possible)?

Use nginx ingress 1.4.0 on the host cluster. Create a service with a long name and an ingress. Accessing that ingress will return a 503.

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7", GitCommit:"e6f35974b08862a23e7f4aad8e5d7f7f2de26c15", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:14Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7+k3s1", GitCommit:"7af5b16788afe9ce1718d7b75b35eafac7454705", GitTreeState:"clean", BuildDate:"2022-10-25T19:31:34Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}

Host cluster Kubernetes distribution

v1.24.7+k3s1

vlcuster version

$ vcluster --version
0.12.2

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k8s

OS and Arch

OS
linux (amd64)
OS Image
openSUSE MicroOS

[FabianKramm]

@WolfspiritM thanks for creating this issue! While this is unfortunate, I believe the problem lies on the ingress-nginx side as this exact same problem also occurs for long service names without vcluster. This just seem to happen here more often as vcluster is concatinating names which results in longer names in general, but this isn't a bug per se. Instead the ingress-nginx folks need to work on their endpointslice detection as this seems problematic for all long service names.

[WolfspiritM]

@FabianKramm Thanks for clarifying. I opened an issue in the nginx-ingress repository 😄