consul-connect-inject-init terminates

[behnaz-surf]

Hi,

we were working with Consul 1.6 without any problem.
But just after upgrading to Consul 1.8 on k8s, we are facing these problems:

consul-connect-inject-init pod randomly throws error. I tried to see what is wrong and I found this log once:

kubectl logs ... consul-connect-inject-init
Registered service: x-sidecar-proxy
Registered service: x
cp: write error: Out of memory

I think it happens here:

      /bin/consul services register \
        /consul/connect-inject/service.hcl
      
      # Generate the envoy bootstrap code
      /bin/consul connect envoy \
        -proxy-id="${PROXY_SERVICE_ID}" \
        -bootstrap > /consul/connect-inject/envoy-bootstrap.yaml
      
      # Copy the Consul binary
      cp /bin/consul /consul/connect-inject/consul

On that time there was not any problem with my k8s cluster but there is a resource section that seems to be new in consul-connect-inject-init:

    name: consul-connect-inject-init
    resources:
      limits:
        cpu: 50m
        memory: 25Mi
      requests:
        cpu: 50m
        memory: 25Mi

and I suspect that might cause errors. I tried to override in values.yml it but it was not possible.

[kschoche]

Hi @behnaz-surf, thank you for reporting this issue, I'm sorry that you're running into this. This is an issue that we're actively working on in the consul-k8s project and consider a high priority to fix.

As the issue has been very difficult to reproduce in-house, would it be possible for you to describe a little bit more about your environment:
e.g. was the cluster under significant load?
also what type of k8s environment it is?

If you need to be unblocked you can change the following in consul-k8s/connect-inject/container_init.go :

   name: consul-connect-inject-init
    resources:
      limits:
        cpu: 50m
        memory: 125Mi    // <-- change from 25Mi to 125Mi
      requests:
        cpu: 50m
        memory: 25Mi

Build the consul-k8s container using make dev-docker in consul-k8s, it will have a tag of consul-k8s-dev and then be sure to set your imageK8S: in your helm values file to your new image.

If you have time to try this route and let us know if it does resolve the issue for you temporarily that would be really helpful.

Thanks!

[lkysow]

Hi! This was fixed in consul-helm 0.23.0 (but use 0.23.1 since that has an important TLS patch).

[behnaz-surf]

It was fixed by the update. Thank you very much for your fast support.