502 error when accessing ingress

[Butters646]

Chart version:
Release 7.1.1

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)

KOPS on AWS

Helm Version:

Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

helm get release output

REVISION: 1
RELEASED: Fri Jun  7 14:44:46 2019
CHART: kibana-7.1.0
USER-SUPPLIED VALUES:
elasticsearchHosts: https://elasticsearch-master:9200
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
ingress:
  annotations: {}
  enabled: true
  hosts:
  - xxx.io
kibanaConfig:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
protocol: http
secretMounts:
- name: elastic-certificate-pem
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificate-pem

COMPUTED VALUES:
affinity: {}
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
elasticsearchHosts: https://elasticsearch-master:9200
elasticsearchURL: ""
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
fullnameOverride: ""
healthCheckPath: /app/kibana
httpPort: 5601
image: docker.elastic.co/kibana/kibana
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.1.0
ingress:
  annotations: {}
  enabled: true
  hosts:
  - kibana.xxx.io
  path: /
  tls: []
kibanaConfig:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
maxUnavailable: 1
nameOverride: ""
nodeSelector: {}
podSecurityContext: {}
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 500m
secretMounts:
- name: elastic-certificate-pem
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificate-pem
service:
  port: 5601
  type: ClusterIP
serviceAccount: ""
tolerations: []
updateStrategy:
  type: Recreate

HOOKS:
MANIFEST:

---
# Source: kibana/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-kibana-config
  labels:
    app: kibana
    release: "kibana"
data:
  kibana.yml: |
    server.host: kibana.xxx.io
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
---
# Source: kibana/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
    heritage: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      name: http
      targetPort: 5601
  selector:
    app: kibana
    release: "kibana"
---
# Source: kibana/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
spec:
  replicas: 1
  strategy:
    type: Recreate

  selector:
    matchLabels:
      app: kibana
      release: "kibana"
  template:
    metadata:
      labels:
        app: kibana
        release: "kibana"
      annotations:

        configchecksum: d6304589bb11d4439446b7290fbda4717d8e393c9e55733751380c6e3bf713e
    spec:
      volumes:
        - name: elastic-certificate-pem
          secret:
            secretName: elastic-certificate-pem
        - name: kibanaconfig
          configMap:
            name: kibana-kibana-config
      containers:
      - name: kibana
        image: "docker.elastic.co/kibana/kibana:7.1.0"
        env:
          - name: ELASTICSEARCH_HOSTS
            value: "https://elasticsearch-master:9200"
          - name: ELASTICSEARCH_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials

        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                http () {
                    local path="${1}"
                    set -- -XGET -s --fail

                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    fi

                    curl -k "$@" "http://localhost:5601${path}"
                }

                http "/app/kibana"
        ports:
        - containerPort: 5601
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500m

        volumeMounts:
          - name: elastic-certificate-pem
            mountPath: /usr/share/kibana/config/certs
          - name: kibanaconfig
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml
---
# Source: kibana/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: kibana
    heritage: Tiller
spec:
  rules:
    - host: kibana.xxx.io
      http:
        paths:
          - path: /
            backend:
              serviceName: kibana-kibana
              servicePort: 5601

Describe the bug:

When I try access the ingress that is created above I get a 502 error from nginx.

Steps to reproduce:

1. Install the secure version of the ES helm chart.
2. Install kibana helm chart with a pem ca cert that was used to create the ES chart.
3. Go to the ingress.

Expected behavior:

Able to access kibana from the above ingress

Provide logs and/or server output (if relevant):

Any additional context:

Kibana appears to be working and connects to the Elastic. All my pods are green. When I do a port forward to the service or the deployment it loads fine.

kubectl -n logging port-forward svc/kibana-kibana 5601

It's the ingress that isn't working. If I try a curl from inside a pod, I also get a connection refused.

curl kibana-kibana.logging.svc.cluster.local:5601
curl: (7) Failed to connect to kibana-kibana.logging.svc.cluster.local port 5601: Connection refused

[jfpucheu]

HEllo
add
server.host: 0.0.0.0

to kibana.yml config

jeff

[Butters646]

That worked, thanks Jeff!

You might want to add this setting to the kibana yaml file automatically when the ingress is enabled.

[Crazybus]

This is working for me (on GKE) without anything set for server.host. In your original comment I can see that you had it originally set to your domain name.

kibanaConfig:
  kibana.yml: |
    server.host: kibana.xxx.io

Can you check whether it works properly when removing this altogether? The default value is localhost and should work just fine.

Which ingress controller are you using? That could also potentially cause differences.

[jfpucheu]

Hello,
I'm not in GKE but in a custom kubernetes cluster using nginx ingress controller.
The flow coming from another pod (nginx ingress) that is not on the kibana pod , kibana is not accessible on localhost. I'm not using ingress from chart.( I will test)

Jeff

[Butters646]

I was trying that a troubleshooting step. Originally I had nothing in there. Didn't realize I left that in there.

I am using KOPS on AWS.

When my kibana.yaml file is this:

kibana.yml:
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
verificationMode: certificate

I get the 502 error. I am using nginx:
https://github.com/kubernetes/ingress-nginx/releases
version .19

[Crazybus]

Ahh I see the problem now. The default docker images comes with server.host: "0". As soon as you override kibana.yml you lose this default setting.

I'll whip up a PR now to set this via an environment variable to make sure it is always set.