feat(balancer): expose balancer_health even when healthchecks are off

[hishamhm]

The Kong load balancer has concepts of upstream health, which is managed by the healthchecker, and balancer health, which is affected by the healthchecker but also affected by the balancer "health threshold" setting. This means a balancer can be unhealthy (and return HTTP 503) even when healthchecks for the upstream are disabled.

The endpoint /upstreams/<upstream>/health?balancer_health=1 is an "advanced mode" for viewing health, which shows the balancer health as opposed to upstream health. (It is hidden behind a query argument to avoid confusion, as for most cases the upstream health is sufficient and the "balancer" terminology referring to a mostly internal Kong object may also be unclear to end-users.) This endpoint, however, only returns the balancer health, when healthchecks for upstream health are enabled.

This PR changes this behavior so that using ?balancer_health=1 we can always see the balancer health, through a new attribute balancer_health, which always returns HEALTHY or UNHEALTHY (reporting the true state of the balancer), even if the overall upstream health status is HEALTHCHECKS_OFF. This is useful for debugging.

The original attribute health is preserved with the existing semantics for backward compatibility, returning HEALTHY, UNHEALTHY or HEALTHCHECKS_OFF.

[hishamhm]

Thanks @hutchic for giving me a ping about this! It's been a while! I've just rebased it against the latest master (it wasn't too much trouble in spite of the balancer refactors!) This feature was an old ask from @Tieske, who wanted more diagnostics from the balancer state even when healthchecks were disabled, based on his experience troubleshooting customer issues. It might still be useful!

[hbagdi]

Closing because of lack of activity. We would like to accept this contribution if anyone wishes to pick this back up.

[hishamhm]

lack of activity? I rebased it like 3 hours ago 😆

[hbagdi]

Oops, apologies. Could you please re-open this one?

[hishamhm]

I can't, the reopen button is grayed out.

[hishamhm]

@bungle thank you! ❤️

[ankujuniyal]

@bungle @hishamhm just want to understand how this query param work "health?balancer_health=1". In my case even the target exist in upstream im not getting data key and its value, Sometime is show some time it didn't show up.
Like
{"node_id":"908a59ea-1ff1-47eb-89a1-xxxxxxxxx","next":null}
but some time i get {"node_id":"908a59ea-1ff1-47eb-89a1-xxxxxxxxx","data":[***] "next":null}
same i can see in admin logs response byte size is very less

x.x.x.x - - [30/Mar/2023:16:45:37 +0530] "GET /upstreams/908a59ea-1ff1-47eb-89a1-xxxxxxxxx/health?balancer_health=1 HTTP/1.1" 200 62 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/111.0"

x.x.x.x - - [30/Mar/2023:16:18:23 +0530] "GET /upstreams/908a59ea-1ff1-47eb-89a1-xxxxxxxxx/health?balancer_health=1 HTTP/1.1" 200 621 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/111.0"

Kong version:- 2.4.0
lua_version:- "LuaJIT 2.1.0-beta3"

Do let me know how can I debug this.