-
-
Notifications
You must be signed in to change notification settings - Fork 358
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decouple credentials fetch from actual requests #203
Comments
Thank you for the detailed issue description! Your analysis is on point. The logic was implemented in such a way because of a couple of reasons:
What's obviously less than ideal is that the cache has such a short TTL. I would expect that the even if EOL for the cache item is reached, that the cache item be preferred than having no key (assuming the request to the remote fails with a timeout). In general I think that it is a sensible request to make the fetching of credentials configurable wrt timeouts. Every system has different tolerances regarding latency and if - for example - you decide that a 2s latency is tolerable, then that should be supported by the software. The same would apply the other way around where the latency must be much smaller (e.g. < 50ms). It also makes sense to configure the TTL of the cache somewhere. As a workaround for now, yes, the file thing is probably best and easily implementable. Simply fetch the key before running In any case, thank you for the write up and making the context so clear. I think that we'll find a suitable solution to this issue quickly! |
Just wanted to check if there was any progress on this issue? |
Not from our side but we appreciate - as always - contributions in this area :) |
At the end in our company, we decided that it would be easier to just implement the
Here we use another "oathkeeper-jwks-updater" container just to update jwks file in a shared "emptyDir" volume. This way the file updates every 30s and is also cached by oathkeeper for another 30s so it can take from 30s to 60s after an update to take an effect. We have been using this solution for more than a week in prod and haven't found any problems with it. |
Awesome, thank you for sharing your knowledge! |
@aeneasr do you have any preferences on what might be done in regards to this? I am thinking about jumping on this issue.. Few suggestions:
|
SGTM - I think we can also fetch the JWK and if it is fast enough use the new keys, if it's too slow use the old ones? |
That is certainly worth considering, I'll prepare a PR with the approach @aeneasr recommended using a timer & go rutine.. we'll see how that goes :) |
Wohoo, nice! :) |
Is your feature request related to a problem? Please describe.
First of all thanks for the big update in 0.16, it made our setup much simpler. Our issue is that timeout for credentials fetcher is now fixed (we use auth0 with jwt authenticator). Unfortunately, requests to jwks.json are very slow (1-1.5s from countries where our developers reside) and occasionally we get 403 from oathkeeper, but the next request succeeds.
The actual scenario is (I believe):
{"error":{"code":403,"status":"Forbidden","reason":"An internal server error occurred, please contact the system administrator","message":"Access credentials are not sufficient to access this resource"}}
Describe the solution you'd like
An ideal solution would be some kind of background function that will fetch jwks periodically so actual requests from clients would always use cache (even if it's stale). The benefits would be:
Describe alternatives you've considered
I understand that the ideal solution might be difficult to implement so I see 2 alternative ones:
Additional context
If I understand correctly the 1s limit is set here
oathkeeper/driver/registry_memory.go
Line 183 in 1e03ee2
oathkeeper/credentials/fetcher_default.go
Line 94 in 1e03ee2
The text was updated successfully, but these errors were encountered: