Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

timeout connecting to upstream for one app and other is working- 504 Bad Gateway timeout #10847

Open
ajmalkhalil opened this issue Jan 14, 2024 · 7 comments
Open

timeout connecting to upstream for one app and other is working- 504 Bad Gateway timeout #10847

ajmalkhalil opened this issue Jan 14, 2024 · 7 comments
Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@ajmalkhalil
Copy link

ajmalkhalil commented Jan 14, 2024 •
edited
Loading

timeout connecting to upstream for one keycloak app, keycloak service is woring fine but ingress controller is not
On browser getting 504 Bad Gateway timeout:

Keycloak should work and should be accessible on the browser, ingress rules files are correct and ingress controller two pods are running:

NGINX Ingress controller version
kubectl exec -it pod/ingress-nginx-controller-95799797c-r89z7 -n ingress-nginx -- /nginx-ingress-controller --version

NGINX Ingress controller
Release: v1.9.5
Build: f503c4b
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8", GitCommit:"66fee42707cd7f5a89f1987f7cb81b02dd19161c", GitTreeState:"clean", BuildDate:"2023-11-15T16:59:43Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Bare Metal Red Hat servers:

  • OS NAME="Red Hat Enterprise Linux"
    VERSION="8.9 (Ootpa)"
    ID="rhel"
    ID_LIKE="fedora"
    VERSION_ID="8.9"
    PLATFORM_ID="platform:el8"
    PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
    HOME_URL="https://www.redhat.com/"
    DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"

  • Kernel (uname -a
    Linux Work-node1 4.18.0-513.9.1.el8_9.x86_64 Basic structure  #1 SMP Thu Nov 16 10:29:04 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
    ):

  • Install tools:

    • I used kubeadm to install cluster.

  • Basic cluster related info:

    • kubectl version... Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8", GitCommit:"66fee42707cd7f5a89f1987f7cb81b02dd19161c", GitTreeState:"clean", BuildDate:"2023-11-15T16:59:43Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}
    • kubectl get nodes -o wide kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-node Ready control-plane 33d v1.27.8 10.0.7.35 <none> Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.9.1.el8_9.x86_64 docker://24.0.7 work-node1 Ready <none> 33d v1.27.8 10.0.7.36 <none> Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.9.1.el8_9.x86_64 docker://24.0.7 worker-node2 Ready <none> 30d v1.27.8 10.0.7.37 <none> Red Hat Enterprise Linux 8.9 (Ootpa) 4.18.0-513.9.1.el8_9.x86_64 docker://24.0.7

  • **I Installed ingress nginx controller github page using helm
    helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace
    **:

    • If helm was used then please show output of helm ls -A | grep -i ingress helm ls -A | grep -i ingress ingress-nginx ingress-nginx 2 2024-01-14 23:40:30.749478736 +0300 +03 deployed ingress-nginx-4.9.0 1.9.5
    • If helm was used then please show output of `helm -n get values
      helm -n ingress-nginx get values ingress-nginx
      USER-SUPPLIED VALUES:
      controller:
      config:
      log-format-escape-json: "true"
      log-format-upstream: '{"timestamp":"$time_iso8601","requestID":"$req_id","proxyUpstreamName":"$proxy_upstream_name","proxyAlternativeUpstreamName":"$proxy_alternative_upstream_name","upstreamStatus":"$upstream_status","upstreamAddr":"$upstream_addr","timeLocal":"$time_local","XForwaredFor":"$http_x_forwarded_for","request":{"requestMethod":"$request_method","requestUrl":"$host$request_uri","status":"$status","requestSize":"$request_length","responseSize":"$upstream_response_length","userAgent":"$http_user_agent","remoteIp":"$remote_addr","remoteUser":"$remote_user","referer":"$http_referer","latency":"$upstream_response_time
      s","requestTime":"$request_time","protocol":"$server_protocol","requestPlain":"$request"}}'
      log-level: debug
      use-forwarded-headers: "true"
      replicaCount: 2
      defaultBackend:
      enabled: true
      service:
      annotations:
      prometheus.io/port: "10254"
      prometheus.io/scrape: "true"

    `

    • helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace -f ingress-controller-values.yaml
    • Only one ingress controller with two pods are installed

  • Current State of the controller:

    • `kubectl describe ingressclasses
      kubectl describe ingressclasses
      Name: nginx
      Labels: app.kubernetes.io/component=controller
      app.kubernetes.io/instance=ingress-nginx
      app.kubernetes.io/managed-by=Helm
      app.kubernetes.io/name=ingress-nginx
      app.kubernetes.io/part-of=ingress-nginx
      app.kubernetes.io/version=1.9.5
      helm.sh/chart=ingress-nginx-4.9.0
      Annotations: meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress-nginx
      Controller: k8s.io/ingress-nginx
      Events:

    `

    • `kubectl -n get all -A -o wide
      kubectl -n ingress-nginx get all -A -o wide
      NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
      cert-manager pod/cert-manager-75997f4b44-lh9hm 1/1 Running 1 (14d ago) 15d 10.244.2.138 worker-node2
      cert-manager pod/cert-manager-cainjector-769785cd7b-hngfk 1/1 Running 9 (14d ago) 15d 10.244.2.139 worker-node2
      cert-manager pod/cert-manager-webhook-6bc9944d78-tfzl2 1/1 Running 0 15d 10.244.2.141 worker-node2
      default pod/keycloak-keycloakx-0 1/1 Running 0 5h45m 10.244.1.54 work-node1
      default pod/keycloak-keycloakx-1 1/1 Running 0 3h44m 10.244.2.33 worker-node2
      ingress-nginx pod/ingress-nginx-controller-95799797c-hzwwb 1/1 Running 0 16h 10.244.2.32 worker-node2
      ingress-nginx pod/ingress-nginx-controller-95799797c-r89z7 1/1 Running 0 16h 10.244.1.50 work-node1
      ingress-nginx pod/ingress-nginx-defaultbackend-76c5fb48d8-mrsps 1/1 Running 0 17h 10.244.1.47 work-node1
      kube-flannel pod/kube-flannel-ds-cvpjq 1/1 Running 2 (15d ago) 30d 10.0.7.37 worker-node2
      kube-flannel pod/kube-flannel-ds-qrhtl 1/1 Running 2 (14d ago) 33d 10.0.7.35 master-node
      kube-flannel pod/kube-flannel-ds-rt6sv 1/1 Running 2 (15d ago) 33d 10.0.7.36 work-node1
      kube-system pod/coredns-86d68db468-cdt6d 1/1 Running 0 18h 10.244.0.18 master-node
      kube-system pod/coredns-86d68db468-j86s8 1/1 Running 0 18h 10.244.1.42 work-node1
      kube-system pod/etcd-master-node 1/1 Running 2 (14d ago) 33d 10.0.7.35 master-node
      kube-system pod/kube-apiserver-master-node 1/1 Running 0 14d 10.0.7.35 master-node
      kube-system pod/kube-controller-manager-master-node 1/1 Running 0 14d 10.0.7.35 master-node
      kube-system pod/kube-proxy-g4pvk 1/1 Running 2 (14d ago) 33d 10.0.7.35 master-node
      kube-system pod/kube-proxy-k79vc 1/1 Running 1 (15d ago) 30d 10.0.7.37 worker-node2
      kube-system pod/kube-proxy-v6n4w 1/1 Running 2 (15d ago) 33d 10.0.7.36 work-node1
      kube-system pod/kube-scheduler-master-node 1/1 Running 0 14d 10.0.7.35 master-node
      metallb-system pod/controller-565ccc769f-vqxsr 1/1 Running 0 15d 10.244.2.140 worker-node2
      metallb-system pod/speaker-f94rd 1/1 Running 3 (15d ago) 30d 10.0.7.37 worker-node2
      metallb-system pod/speaker-wks5j 1/1 Running 4 (14d ago) 32d 10.0.7.36 work-node1
      metallb-system pod/speaker-xtxjc 1/1 Running 4 (14d ago) 32d 10.0.7.35 master-node
      rabbit pod/rabbit-rabbitmq-0 1/1 Running 0 23h 10.244.1.29 work-node1
      rabbit pod/rabbit-rabbitmq-1 0/1 Running 23 (5m58s ago) 167m 10.244.2.35 worker-node2
      taqeem pod/taqeem-app-d6dc8c5f-9r7fn 1/1 Running 0 27h 10.244.2.24 worker-node2

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
cert-manager service/cert-manager ClusterIP 10.105.201.170 9402/TCP 19d app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
cert-manager service/cert-manager-webhook ClusterIP 10.97.178.7 443/TCP 19d app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
default service/keycloak-keycloakx-headless ClusterIP None 80/TCP 5d16h app.kubernetes.io/instance=keycloak,app.kubernetes.io/name=keycloakx
default service/keycloak-keycloakx-http ClusterIP 10.111.159.20 80/TCP 5d16h app.kubernetes.io/instance=keycloak,app.kubernetes.io/name=keycloakx
default service/kubernetes ClusterIP 10.96.0.1 443/TCP 29d
default service/nginx ClusterIP 10.99.35.47 80/TCP 28d run=nginx
ingress-nginx service/ingress-nginx-controller LoadBalancer 10.106.226.129 10.0.7.71 80:30683/TCP,443:30241/TCP 17h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.107.206.120 443/TCP 17h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx service/ingress-nginx-defaultbackend ClusterIP 10.108.169.41 80/TCP 17h app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 33d k8s-app=kube-dns
metallb-system service/webhook-service ClusterIP 10.109.150.207 443/TCP 32d component=controller
rabbit service/rabbit-rabbitmq ClusterIP 10.102.154.189 5672/TCP,4369/TCP,25672/TCP,15672/TCP 9d app.kubernetes.io/instance=rabbit,app.kubernetes.io/name=rabbitmq
rabbit service/rabbit-rabbitmq-headless ClusterIP None 4369/TCP,5672/TCP,25672/TCP,15672/TCP 9d app.kubernetes.io/instance=rabbit,app.kubernetes.io/name=rabbitmq
taqeem service/taqeem-app-service ClusterIP 10.109.212.20 8080/TCP 27h app=my-app

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
kube-flannel daemonset.apps/kube-flannel-ds 3 3 3 3 3 33d kube-flannel docker.io/flannel/flannel:v0.23.0 app=flannel,k8s-app=flannel
kube-system daemonset.apps/kube-proxy 3 3 3 3 3 kubernetes.io/os=linux 33d kube-proxy registry.k8s.io/kube-proxy:v1.27.8 k8s-app=kube-proxy
metallb-system daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 32d speaker quay.io/metallb/speaker:v0.13.12 app=metallb,component=speaker

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
cert-manager deployment.apps/cert-manager 1/1 1 1 19d cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.13.3 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 19d cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.13.3 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector
cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 19d cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.13.3 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
ingress-nginx deployment.apps/ingress-nginx-controller 2/2 2 2 17h controller registry.k8s.io/ingress-nginx/controller:v1.9.5@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx deployment.apps/ingress-nginx-defaultbackend 1/1 1 1 17h ingress-nginx-default-backend registry.k8s.io/defaultbackend-amd64:1.5 app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system deployment.apps/coredns 2/2 2 2 33d coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns
metallb-system deployment.apps/controller 1/1 1 1 32d controller quay.io/metallb/controller:v0.13.12 app=metallb,component=controller
taqeem deployment.apps/taqeem-app 1/1 1 1 27h taqeemimage taqeemdevops786/evaluation-mnt-ms:0.1.1-SNAPSHOT app=my-app

NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
cert-manager replicaset.apps/cert-manager-75997f4b44 1 1 1 19d cert-manager-controller quay.io/jetstack/cert-manager-controller:v1.13.3 app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager,pod-template-hash=75997f4b44
cert-manager replicaset.apps/cert-manager-cainjector-769785cd7b 1 1 1 19d cert-manager-cainjector quay.io/jetstack/cert-manager-cainjector:v1.13.3 app.kubernetes.io/component=cainjector,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cainjector,pod-template-hash=769785cd7b
cert-manager replicaset.apps/cert-manager-webhook-6bc9944d78 1 1 1 19d cert-manager-webhook quay.io/jetstack/cert-manager-webhook:v1.13.3 app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook,pod-template-hash=6bc9944d78
ingress-nginx replicaset.apps/ingress-nginx-controller-95799797c 2 2 2 17h controller registry.k8s.io/ingress-nginx/controller:v1.9.5@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=95799797c
ingress-nginx replicaset.apps/ingress-nginx-defaultbackend-76c5fb48d8 1 1 1 17h ingress-nginx-default-backend registry.k8s.io/defaultbackend-amd64:1.5 app.kubernetes.io/component=default-backend,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=76c5fb48d8
kube-system replicaset.apps/coredns-5d78c9869d 0 0 0 33d coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=5d78c9869d
kube-system replicaset.apps/coredns-774c4869d9 0 0 0 8d coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=774c4869d9
kube-system replicaset.apps/coredns-7b4cb5d99b 0 0 0 9d coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=7b4cb5d99b
kube-system replicaset.apps/coredns-7fbd59d47 0 0 0 9d coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=7fbd59d47
kube-system replicaset.apps/coredns-86d68db468 2 2 2 18h coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=86d68db468
kube-system replicaset.apps/coredns-bfff4fc4 0 0 0 5d16h coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=bfff4fc4
kube-system replicaset.apps/coredns-d9445dc9b 0 0 0 26h coredns registry.k8s.io/coredns/coredns:v1.10.1 k8s-app=kube-dns,pod-template-hash=d9445dc9b
metallb-system replicaset.apps/controller-565ccc769f 1 1 1 32d controller quay.io/metallb/controller:v0.13.12 app=metallb,component=controller,pod-template-hash=565ccc769f
taqeem replicaset.apps/taqeem-app-d6dc8c5f 1 1 1 27h taqeemimage taqeemdevops786/evaluation-mnt-ms:0.1.1-SNAPSHOT app=my-app,pod-template-hash=d6dc8c5f

NAMESPACE NAME READY AGE CONTAINERS IMAGES
default statefulset.apps/keycloak-keycloakx 2/2 5d16h keycloak quay.io/keycloak/keycloak:21.1.0
rabbit statefulset.apps/rabbit-rabbitmq 1/2 9d rabbitmq docker.io/bitnami/rabbitmq:3.10.13

`

  • `kubectl -n describe po
    kubectl -n ingress-nginx describe pod/ingress-nginx-controller-95799797c-hzwwb
    Name: ingress-nginx-controller-95799797c-hzwwb
    Namespace: ingress-nginx
    Priority: 0
    Service Account: ingress-nginx
    Node: worker-node2/10.0.7.37
    Start Time: Sun, 14 Jan 2024 23:10:45 +0300
    Labels: app.kubernetes.io/component=controller
    app.kubernetes.io/instance=ingress-nginx
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/name=ingress-nginx
    app.kubernetes.io/part-of=ingress-nginx
    app.kubernetes.io/version=1.9.5
    helm.sh/chart=ingress-nginx-4.9.0
    pod-template-hash=95799797c
    Annotations:
    Status: Running
    IP: 10.244.2.32
    IPs:
    IP: 10.244.2.32
    Controlled By: ReplicaSet/ingress-nginx-controller-95799797c
    Containers:
    controller:
    Container ID: docker://2526e1120d5e99f986f167e125e19c002e691ae4865a1002739197c8a85d3f89
    Image: registry.k8s.io/ingress-nginx/controller:v1.9.5@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e
    Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:b3aba22b1da80e7acfc52b115cae1d4c687172cbf2b742d5b502419c25ff340e
    Ports: 80/TCP, 443/TCP, 8443/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP
    SeccompProfile: RuntimeDefault
    Args:
    /nginx-ingress-controller
    --default-backend-service=$(POD_NAMESPACE)/ingress-nginx-defaultbackend
    --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    --election-id=ingress-nginx-leader
    --controller-class=k8s.io/ingress-nginx
    --ingress-class=nginx
    --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    --validating-webhook=:8443
    --validating-webhook-certificate=/usr/local/certificates/cert
    --validating-webhook-key=/usr/local/certificates/key
    State: Running
    Started: Sun, 14 Jan 2024 23:10:45 +0300
    Ready: True
    Restart Count: 0
    Requests:
    cpu: 100m
    memory: 90Mi
    Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
    POD_NAME: ingress-nginx-controller-95799797c-hzwwb (v1:metadata.name)
    POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
    LD_PRELOAD: /usr/local/lib/libmimalloc.so
    Mounts:
    /usr/local/certificates/ from webhook-cert (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bj4vf (ro)
    Conditions:
    Type Status
    Initialized True
    Ready True
    ContainersReady True
    PodScheduled True
    Volumes:
    webhook-cert:
    Type: Secret (a volume populated by a Secret)
    SecretName: ingress-nginx-admission
    Optional: false
    kube-api-access-bj4vf:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional:
    DownwardAPI: true
    QoS Class: Burstable
    Node-Selectors: kubernetes.io/os=linux
    Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:

`

  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
    kubectl -n ingress-nginx describe service/ingress-nginx-controller
    Name: ingress-nginx-controller
    Namespace: ingress-nginx
    Labels: app.kubernetes.io/component=controller
    app.kubernetes.io/instance=ingress-nginx
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/name=ingress-nginx
    app.kubernetes.io/part-of=ingress-nginx
    app.kubernetes.io/version=1.9.5
    helm.sh/chart=ingress-nginx-4.9.0
    Annotations: meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
    metallb.universe.tf/ip-allocated-from-pool: first-pool
    Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
    Type: LoadBalancer
    IP Family Policy: SingleStack
    IP Families: IPv4
    IP: 10.106.226.129
    IPs: 10.106.226.129
    LoadBalancer Ingress: 10.0.7.71
    Port: http 80/TCP
    TargetPort: http/TCP
    NodePort: http 30683/TCP
    Endpoints: 10.244.1.50:80,10.244.2.32:80
    Port: https 443/TCP
    TargetPort: https/TCP
    NodePort: https 30241/TCP
    Endpoints: 10.244.1.50:443,10.244.2.32:443
    Session Affinity: None
    External Traffic Policy: Cluster
    Events:

  • Current state of ingress object, if applicable:

    • `kubectl -n get all,ing -o wide

    kubectl get all,ing -o wide
    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
    pod/keycloak-keycloakx-0 1/1 Running 0 5h48m 10.244.1.54 work-node1
    pod/keycloak-keycloakx-1 1/1 Running 0 3h47m 10.244.2.33 worker-node2

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/keycloak-keycloakx-headless ClusterIP None 80/TCP 5d16h app.kubernetes.io/instance=keycloak,app.kubernetes.io/name=keycloakx
service/keycloak-keycloakx-http ClusterIP 10.111.159.20 80/TCP 5d16h app.kubernetes.io/instance=keycloak,app.kubernetes.io/name=keycloakx
service/kubernetes ClusterIP 10.96.0.1 443/TCP 29d
service/nginx ClusterIP 10.99.35.47 80/TCP 28d run=nginx

NAME READY AGE CONTAINERS IMAGES
statefulset.apps/keycloak-keycloakx 2/2 5d16h keycloak quay.io/keycloak/keycloak:21.1.0

NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/keycloak-ingress nginx keycloak.testing.com 10.0.7.71 80 4h9m
`

  • kubectl -n <appnamespace> describe ing <ingressname>

  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

  • Others:

    • Any other related information like ;
      • copy/paste of the snippet (if applicable)
      • kubectl describe ... of any custom configmap(s) created and in use
      • Any other related information that may help
        kubectl describe cm ingress-nginx-controller -n ingress-nginx
        Name: ingress-nginx-controller
        Namespace: ingress-nginx
        Labels: app.kubernetes.io/component=controller
        app.kubernetes.io/instance=ingress-nginx
        app.kubernetes.io/managed-by=Helm
        app.kubernetes.io/name=ingress-nginx
        app.kubernetes.io/part-of=ingress-nginx
        app.kubernetes.io/version=1.9.5
        helm.sh/chart=ingress-nginx-4.9.0
        Annotations: meta.helm.sh/release-name: ingress-nginx
        meta.helm.sh/release-namespace: ingress-nginx

Data

log-format-escape-json:

true
log-format-upstream:

{"timestamp":"$time_iso8601","requestID":"$req_id","proxyUpstreamName":"$proxy_upstream_name","proxyAlternativeUpstreamName":"$proxy_alternative_upstream_name","upstreamStatus":"$upstream_status","upstreamAddr":"$upstream_addr","timeLocal":"$time_local","XForwaredFor":"$http_x_forwarded_for","request":{"requestMethod":"$request_method","requestUrl":"$host$request_uri","status":"$status","requestSize":"$request_length","responseSize":"$upstream_response_length","userAgent":"$http_user_agent","remoteIp":"$remote_addr","remoteUser":"$remote_user","referer":"$http_referer","latency":"$upstream_response_time s","requestTime":"$request_time","protocol":"$server_protocol","requestPlain":"$request"}}
log-level:

debug
proxy-connect-timeout:

10
proxy-read-timeout:

120
proxy-send-timeout:

120
use-forwarded-headers:

true
allow-snippet-annotations:

false

BinaryData

Events:

How to reproduce this issue:
@ajmalkhalil ajmalkhalil added the kind/bug Categorizes issue or PR as related to a bug. label Jan 14, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jan 14, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@longwuyuan
Copy link
Contributor

longwuyuan commented Jan 15, 2024 •
edited
Loading

@ajmalkhalil there is no information posted by you to analyze so any comments will be guess work.

@k8s-ci-robot k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jan 15, 2024
@longwuyuan
Copy link
Contributor

/retitle timeout connecting to upstream

@k8s-ci-robot k8s-ci-robot changed the title Bare metal [error] upstream timed out (110: Operation timed out) while connecting to upstream timeout connecting to upstream Jan 15, 2024
@ajmalkhalil ajmalkhalil changed the title timeout connecting to upstream timeout connecting to upstream for one app and other is working Jan 15, 2024
@longwuyuan longwuyuan mentioned this issue Jan 15, 2024
Closed
@ajmalkhalil ajmalkhalil changed the title timeout connecting to upstream for one app and other is working timeout connecting to upstream for one app and other is working- 504 Bad Gateway timeout Jan 15, 2024
@longwuyuan
Copy link
Contributor

longwuyuan commented Jan 15, 2024 •
edited
Loading

  • The issue description is not formatted as per markdown. So unable to do a good reading easily
  • The output of kubectl decribe ingress is not provided
  • The curl command or URL requested in browser is not provided
  • The result of the curl command or the response in browser is not shown
  • The logs of the controller pod are not provided
  • Reach out to other users of Keycloak on slack and ask them how they have installed and configured keycloak

@ajmalkhalil
Copy link
Author

@longwuyuan Above Format has been corrected, please help. Thanks!

@longwuyuan
Copy link
Contributor

  • I don't see any useful change in markdown formatting
  • But I already made some comments based on what data was readale

@github-actions GitHub Actions
Copy link

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

@github-actions github-actions bot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
[SIG Network] Ingress NGINX
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@longwuyuan @k8s-ci-robot @ajmalkhalil