memory leak on OAuth 2.0 Resource Server JWT verification

[zandbelt]

we seem to have traced a memory issue we’ve been having to mod_auth_openidc.

We use the module to provide local JWT validation in a resource server scenario within docker / Kubernetes. Our problem is that the memory consumption of apache worker processes seems to be growing continuously under load, from ~10MB up to gigabytes, finally getting killed by k8s when the memory limit is reached.

I was able to reproduce this effect in a local docker setup with very minimal configuration (see attached Dockerfile).

It basically boils down to:

OIDCOAuthVerifyJwksUri https://<keycloak url>/auth/realms/<realm>/protocol/openid-connect/certs
<Location />
    AuthType oauth20
    Require valid-user
</Location> 

Memory consumption is stable under load in this simple scenario when switching to basic auth instead of oauth20 - but with activated oauth20, every request seems to leak a few kilobytes of memory.

We’ve reproduced this in various environments:

Provided packages on centos 8: httpd 2.4.37-16, mod_auth_openidc 2.3.7-3
Provided packages on buster: apache2 2.4.38-3, libapache2-mod-auth-openidc 2.3.10.2-1
buster with stock apache2 and current deb package from github release page (2.4.1-1)
Across the different test setups, we’ve also tried mpm_worker, mpm_prefork, and mpm_event.