connect() to unix:/tmp/nginx/prometheus-nginx.socket failed (2: No such file or directory)

[mmasdoria]

What happened:

ingress nginx controller send several metric message to premetheus socket and stop working suddenly with following error :

connect to unix:/tmp/nginx/prometheus-nginx.socket, fd:16 #5185
connect() to unix:/tmp/nginx/prometheus-nginx.socket failed (2: No such file or directory), context: ngx.timer
lua tcp socket connect: -5
lua tcp socket error retval handler

When Ithe issue occur prometheus-nginx.socket disappears from /tmp/nginx and nginx.pid too !

What you expected to happen:
ngress nginx controller should be work and send metrics into socket without any issue

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

---

NGINX Ingress controller
Release: v1.2.1
Build: 08848d6
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10

Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7-gke.900", GitCommit:"e35c4457f66187eff006dda6d2c0fe12144ef2ec", GitTreeState:"clean", BuildDate:"2022-10-26T09:25:34Z", GoVersion:"go1.18.7b7", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE

• OS (e.g. from /etc/os-release): Alpine Linux v3.14

• Kernel (e.g. uname -a): Linux gitlab-nginx-ingress-controller-7ccd9cc975-c8zgx 5.10.147+ Basic structure  #1 SMP Sat Oct 22 14:54:40 UTC 2022 x86_64 Linux

• Install tools: Helm

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
    Google GKE cluster v1.24.7-gke.900

• Basic cluster related info:

  • kubectl version
    Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.5", GitCommit:"e979822c185a14537054f15808a118d7fcce1d6e", GitTreeState:"clean", BuildDate:"2022-09-14T16:42:36Z", GoVersion:"go1.18.6", Compiler:"gc", Platform:"linux/amd64"}

  • kubectl get nodes -o wide
    NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
    gke-aa-tools-default-pool-9428837a-ek58 Ready 22h v1.24.7-gke.900 10.65.0.28 Container-Optimized OS from Google 5.10.147+ containerd://1.6.6
    gke-aa-tools-default-pool-9428837a-zzoo Ready 22h v1.24.7-gke.900 10.65.0.26 Container-Optimized OS from Google 5.10.147+ containerd://1.6.6

• How was the ingress-nginx-controller installed:
  Install within Gitlab helm chart
  NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
  gitlab gitlab 106 2022-12-02 19:45:42.001655181 +0100 CET deployed gitlab-6.6.0 15.6.0

• Current State of the controller:

  • kubectl describe ingressclasses
    Name: gitlab-nginx
    Labels: app=nginx-ingress
    app.kubernetes.io/component=controller
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/version=1.2.1
    chart=nginx-ingress-4.1.3
    component=controller
    helm.sh/chart=nginx-ingress-4.1.3
    heritage=Helm
    release=gitlab
    Annotations: meta.helm.sh/release-name: gitlab
    meta.helm.sh/release-namespace: gitlab
    Controller: k8s.io/ingress-nginx
    Events:

  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
    Name: gitlab-nginx-ingress-controller-7ccd9cc975-c8zgx
    Namespace: gitlab
    Priority: 0
    Node: gke-aa-tools-default-pool-9428837a-zzoo/10.65.0.26
    Start Time: Fri, 02 Dec 2022 19:59:50 +0100
    Labels: app=nginx-ingress
    component=controller
    pod-template-hash=7ccd9cc975
    release=gitlab
    Annotations:
    Status: Running
    IP: 10.0.0.126
    IPs:
    IP: 10.0.0.126
    Controlled By: ReplicaSet/gitlab-nginx-ingress-controller-7ccd9cc975
    Containers:
    controller:
    Container ID: containerd://d4fa1891c269deb00ff50943b74af3a5c5df9230eeb7dfe6526361f7fc9b8659
    Image: registry.gitlab.com/gitlab-org/cloud-native/mirror/images/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
    Image ID: registry.gitlab.com/gitlab-org/cloud-native/mirror/images/ingress-nginx/controller@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
    Ports: 80/TCP, 443/TCP, 10254/TCP, 22/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
    /nginx-ingress-controller
    --publish-service=$(POD_NAMESPACE)/gitlab-nginx-ingress-controller
    --election-id=ingress-controller-leader
    --controller-class=k8s.io/ingress-nginx
    --configmap=$(POD_NAMESPACE)/gitlab-nginx-ingress-controller
    --tcp-services-configmap=gitlab/gitlab-nginx-ingress-tcp
    --ingress-class=gitlab-nginx
    --v=5
    State: Running
    Started: Fri, 02 Dec 2022 20:37:48 +0100
    Last State: Terminated
    Reason: Error
    Exit Code: 1
    Started: Fri, 02 Dec 2022 20:05:03 +0100
    Finished: Fri, 02 Dec 2022 20:37:47 +0100
    Ready: False
    Restart Count: 2
    Requests:
    cpu: 100m
    memory: 100Mi
    Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
    POD_NAME: gitlab-nginx-ingress-controller-7ccd9cc975-c8zgx (v1:metadata.name)
    POD_NAMESPACE: gitlab (v1:metadata.namespace)
    LD_PRELOAD: /usr/local/lib/libmimalloc.so
    Mounts:
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54jtc (ro)
    Conditions:
    Type Status
    Initialized True
    Ready False
    ContainersReady False
    PodScheduled True
    Volumes:
    kube-api-access-54jtc:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional:
    DownwardAPI: true
    QoS Class: Burstable
    Node-Selectors:
    Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
    Type Reason Age From Message

  ---

  Normal RELOAD 29m nginx-ingress-controller NGINX reload triggered due to a change in configuration
  Warning Unhealthy 2m18s (x423 over 66m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500

  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
    Name: gitlab-nginx-ingress-controller
    Namespace: gitlab
    Labels: app=nginx-ingress
    app.kubernetes.io/component=controller
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/version=1.2.1
    chart=nginx-ingress-4.1.3
    component=controller
    helm.sh/chart=nginx-ingress-4.1.3
    heritage=Helm
    release=gitlab
    Annotations: meta.helm.sh/release-name: gitlab
    meta.helm.sh/release-namespace: gitlab
    Selector: app=nginx-ingress,component=controller,release=gitlab
    Type: LoadBalancer
    IP Family Policy: SingleStack
    IP Families: IPv4
    IP: X.X.X.X
    IPs: X.X.X.X
    IP: Y.Y.Y.Y
    LoadBalancer Ingress: Y.Y.Y.Y
    Port: http 80/TCP
    TargetPort: http/TCP
    NodePort: http 30020/TCP
    Endpoints:
    Port: https 443/TCP
    TargetPort: https/TCP
    NodePort: https 30837/TCP
    Endpoints:
    Port: gitlab-shell 22/TCP
    TargetPort: gitlab-shell/TCP
    NodePort: gitlab-shell 30701/TCP
    Endpoints:
    Session Affinity: None
    External Traffic Policy: Local
    HealthCheck NodePort: 31185
    Events:

• Current state of ingress object, if applicable:
  pod/gitlab-nginx-ingress-controller-7ccd9cc975-c8zgx 0/1 Running 2 (23m ago) 61m 10.0.0.126 gke-aa-tools-default-pool-9428837a-zzoo
  service/gitlab-nginx-ingress-controller LoadBalancer X.X.X.X Y.Y.Y.Y 80:30020/TCP,443:30837/TCP,22:30701/TCP 2y303d app=nginx-ingress,component=controller,release=gitlab
  service/gitlab-nginx-ingress-controller-metrics ClusterIP X.X.X.X 10254/TCP 74m app=nginx-ingress,component=controller,release=gitlab
  deployment.apps/gitlab-nginx-ingress-controller 0/1 1 0 3y90d controller registry.gitlab.com/gitlab-org/cloud-native/mirror/images/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 app=nginx-ingress,component=controller,release=gitlab

How to reproduce this issue:

Install Gitlab Helm Chart v6.6.0
helm upgrade -n gitlab gitlab gitlab/gitlab --version v6.6.0

Anything else we need to know:

[k8s-ci-robot]

@mmasdoria: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug
@mmasdoria There is no clear and detailed information here as to what is the problem in the ingress-nginx-controller that needs to be fixed.

There can be multiple reasons for a socket becoming unavailable. So please post information that the problem of a socket going away was created by the ingress-nginx-controller code/executable.

You may also get better help from the gitlab support team or forums because your installation of the ingress-nginx-controller is bundled with gitlab.

In any case a step-by-step procedure to copy/paste in a minikube or kind cluster may be required for someone to understand whaat is the problem to be solved. And since the installation method of the ingress-nginx-controller is not based on this project's recommended documentation, it will be harder for others to couple gitlab bundle installs and then try to reproduce the problem.

[mmasdoria]

I find the problem.
I use nginx controller for other namesapce than gitlab with scope enabled = false.

Mattermost ingress annnotation is : nginx.ingress.kubernetes.io/configuration-snippet: proxy_cache mattermost_cache; proxy_cache_revalidate on; proxy_cache_min_uses 2; proxy_cache_use_stale timeout; proxy_cache_lock on;

And I have added 3 years ago in ingress controller config map this line : http-snippet proxy_cache_path /tmp/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

Since ingress controller 1.2.1 nginx temp path chnage from /tmp to /tmp/nginx.
ANd that my probleme as snippet use this new path for cache mattermost data ...

So I have change the snippet to :http-snippet proxy_cache_path /tmp/nginx/mattermost levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

And now all is working.

So be careful to not use /tmp/nginx as other stuff than nginx needs.
Regards.

[longwuyuan]

/remove-kind bug
/close

[k8s-ci-robot]

@longwuyuan: Those labels are not set on the issue: kind/bug

In response to this:

/remove-kind bug
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.