http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred}) #2435

evhfla · 2022-04-24T18:22:39Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

On brand new install of Kong using Helm Chart Version 2.8.0, getting the following two errors on an Azure Kubernetes Service Cluster, one error in the proxy logs and the other on the ingress logs.

Proxy - http2 requests not supported yet
Ingress - *checking config status failed: %!w(kong.APIError=&{500 An unexpected error occurred})

Subsequently installed it on minikube to make sure that the issue wasn't specific to an AKS cluster and got the same set of errors.

More detail error in Proxy:
3714 [lua] api_helpers.lua:511: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:424: /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: http2 requests not supported yet

Expected Behavior

No errors.

Steps To Reproduce

helm install kong kong/kong -n kong -f .\values.yaml --version 2.8.0

Value file:
env:
  prefix: /kong_prefix/
  database: "off"
  LOG_LEVEL: "error"
ingressController:
  ingressClass: "kong"
  enabled: true
  installCRDs: false

Kong Ingress Controller version

KIC 2.3

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"ba58f86b00f6b0f0b7694a75464aa7806f8bf6fc", GitTreeState:"clean", BuildDate:"2022-03-30T23:40:46Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

Also, getting the same thing using minikube:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

Kong Ingress Controller seems to be working though...so not sure what these errors mean and how concern I should be.

danopia · 2022-04-25T09:07:41Z

I'm also seeing this on a straightforward installation of the 2.8.0 helm chart.

I found a proxy PR to support http2 on the status endpoint:

fix(admin-status) use resty-http to get nginx_status kong#8528

This is an exact output from proxy:

2022/04/25 08:58:07 [error] 1110#0: *1656702 [lua] api_helpers.lua:511: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:424: /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: http2 requests not supported yet
stack traceback:
        [C]: in function 'capture'
        /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:268>

stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/lapis/application.lua:424: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:146: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:183: in function </usr/local/share/lua/5.1/lapis/application.lua:181>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:189: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:1423: in function 'admin_content'
        content_by_lua(nginx-kong.conf:360):2: in main chunk, client: 127.0.0.1, server: kong_admin, request: "GET /status HTTP/2.0", host: "localhost:8444"
127.0.0.1 - - [25/Apr/2022:08:58:07 +0000] "GET /status HTTP/2.0" 500 42 "-" "Go-http-client/2.0"

Kong overall is operating fine, but the log volume is annoying and it's not clear if this issue has any further impact.

rcmorano · 2022-04-25T09:27:35Z

JFTR, just got hit by this issue aswell. It also happens with chart version 2.7.0, works smooth on 2.6.4.

danopia · 2022-04-25T10:41:27Z

It looks like the effect is that this new feature is not operational:

The controller can now detect whether a Kong container has crashed and needs a configuration push.

39c385e#diff-d8590e8687e7807da0cf97910b88e0ddbfc670866e0973a15c56597b99df9244R59

workaround

These helm chart values seem to restore the broken feature and remove all the error logs:

# Temporary workaround: disable HTTP2 on admin endpoint
# https://github.com/Kong/kubernetes-ingress-controller/issues/2435
admin:
  tls:
    parameters: []

(It seems like KIC could be hitting /status on the status port instead of the admin port, as the status port already has http2 disabled)

jcam · 2022-04-26T14:49:29Z

@rainest do you know whether the above workaround actually leaves the system in a state that correctly is validating configuration? (If you don't know, I'm not asking you to do the work of checking)

As mentioned in the above PR: "since the /nginx_status location is marked as internal, I don't think it will be possible to fetch it using real network socket"... does that mean the status call we are making doesn't actually check the nginx status, or does that only impact metrics?

Just trying to get an idea of whether I can tell my product owners that the upgrade I just put in place will resolve the stability issues or not ;)

Thanks!!!

rainest · 2022-04-26T21:34:40Z

The workaround (disabling HTTP/2 support on the admin API) is fine. It doesn't meaningfully change the operation of the controller, which makes a separate request for each API call anyway rather than using HTTP/2 pipelining. It may technically degrade Manager performance, but I can't think of pages that are making more than 2-3 admin API requests at once, so I doubt the impact would be noticeable.

Kong/kong#8690 is the PR that's planned to fix this/will allow you to turn HTTP/2 back on.

The status endpoint (as of 2.8) provides both generic NGINX status information (connection count and such) and whether or not configuration is present. We only care about the latter, we just can't fetch it separately, and weren't using the endpoint before.

jcam · 2022-04-26T21:36:02Z

Awesome, thank you for the detail!

shaneutt · 2022-05-02T12:52:52Z

marking this as blocked on Kong/kong#8690

rainest · 2022-05-02T17:35:36Z

Minor related issue with the error formatting should be corrected by #2436

scseanchow · 2022-05-31T15:47:39Z

Blocked until Kong/kong#8690 releases (2.8.1)?

jayakumarj85 · 2022-07-13T14:27:04Z

I have set ADMIN_LISTEN environment variable to "127.0.0.1:8444 http2 ssl" and it was causing this error.

As a workaround, I had to set ADMIN_LISTEN to "127.0.0.1:8444 ssl" to get rid of these errors.

altais · 2022-08-25T10:19:33Z

Blocked until Kong/kong#8690 releases (2.8.1)?

Up, It seem it is released and that it can now be fixed

pmalek · 2022-08-25T10:33:27Z

It seems that Kong/kong#8690 didn't land in 2.8.1: Kong/kong@c5fd723.

In that case this will be available in 3.0 (it's already included in 3.0 alpha.1 build: https://github.com/Kong/kong/releases/tag/3.0.0-alpha.1)

mflendrich · 2022-09-06T15:56:41Z

Resolving this issue is a no-op on the ingress controller side.
When Kong releases the version 3.0, making KIC use Kong 3.0 will fix this issue.

Leaving this issue open for sake of discussion & comments if this solution does not work after 3.0 is out.

redradrat · 2022-11-11T18:05:27Z

not sure if this is supposed to be active but I'm still seeing this behavior on 3.0 🤔 I'm still getting the same lua exception log bloat for the HTTP2 probes. So I still have to disable the http2 listener config.

rainest · 2022-11-11T23:47:37Z

@redradrat can you double-check your Deployment?

Deploying https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/v2.7.0/deploy/single/all-in-one-dbless.yaml and adding http2 to the admin listen no longer generates this error for me:

15:44:00-0800 esenin $ kubectl get deploy -n kong ingress-kong -oyaml | grep image:                  
        image: kong:3.0
        image: kong/kubernetes-ingress-controller:2.7.0

15:44:04-0800 esenin $ kubectl get deploy -n kong ingress-kong -oyaml | grep -A1 "name: KONG_ADMIN_LISTEN" 
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl http2

15:44:36-0800 esenin $ kubectl logs -n kong deployments/ingress-kong -c ingress-controller | grep  "checking config status" | wc -l
0

15:44:43-0800 esenin $ kubectl logs -n kong deployments/ingress-kong -c proxy | grep "http2 requests not" | wc -l
0

evhfla added the bug Something isn't working label Apr 24, 2022

shaneutt added blocked priority/medium labels May 2, 2022

rainest mentioned this issue May 16, 2022

Installation fails with default release manifest Kong/charts#593

Closed

HansK-p mentioned this issue May 22, 2022

Proxy container lose its config when oom killed inside the kong pod with an ingress controller setup Kong/kong#7541

Closed

shaneutt added this to the KIC v2.7.0 milestone Aug 25, 2022

mflendrich removed this from the KIC v2.7.0 milestone Sep 6, 2022

rainest mentioned this issue Nov 11, 2022

chore(deploy) align listens with upstream #3165

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred}) #2435

http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred}) #2435

evhfla commented Apr 24, 2022

danopia commented Apr 25, 2022 •

edited

Loading

rcmorano commented Apr 25, 2022 •

edited

Loading

danopia commented Apr 25, 2022 •

edited

Loading

jcam commented Apr 26, 2022 •

edited

Loading

rainest commented Apr 26, 2022

jcam commented Apr 26, 2022

shaneutt commented May 2, 2022

rainest commented May 2, 2022

scseanchow commented May 31, 2022

jayakumarj85 commented Jul 13, 2022

altais commented Aug 25, 2022

pmalek commented Aug 25, 2022

mflendrich commented Sep 6, 2022

redradrat commented Nov 11, 2022 •

edited

Loading

rainest commented Nov 11, 2022

http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred}) #2435

http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred}) #2435

Comments

evhfla commented Apr 24, 2022

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Kong Ingress Controller version

Kubernetes version

Anything else?

danopia commented Apr 25, 2022 • edited Loading

rcmorano commented Apr 25, 2022 • edited Loading

danopia commented Apr 25, 2022 • edited Loading

workaround

jcam commented Apr 26, 2022 • edited Loading

rainest commented Apr 26, 2022

jcam commented Apr 26, 2022

shaneutt commented May 2, 2022

rainest commented May 2, 2022

scseanchow commented May 31, 2022

jayakumarj85 commented Jul 13, 2022

altais commented Aug 25, 2022

pmalek commented Aug 25, 2022

mflendrich commented Sep 6, 2022

redradrat commented Nov 11, 2022 • edited Loading

rainest commented Nov 11, 2022

danopia commented Apr 25, 2022 •

edited

Loading

rcmorano commented Apr 25, 2022 •

edited

Loading

danopia commented Apr 25, 2022 •

edited

Loading

jcam commented Apr 26, 2022 •

edited

Loading

redradrat commented Nov 11, 2022 •

edited

Loading