Error when opening a dashboard behind secured reverse-proxy #22
Comments
|
@danlester Ok I found the error - but I don't know yet how to correct it. self.log.error("OAuth POST from %s != %s", referer, full_url)Log message: Note that the full_url does not have https when the referer has it... The certificate are set in a proxy in front of JHub and not in JHub directly. I think this is the source. And it probably should be tackled in JupyterHub core. |
|
Chances are that I need to configure the proxy as in that example for JupyterHub with nginx: https://gist.github.com/cboettig/8643341bd3c93b62b5c2#file-nginx-conf-L32 |
|
Thank you very much for this report. Is there something that you were trying to change/achieve that led to this error (since everything worked before for you) - were you just trying to move everything behind the ngnix server and install https? Please let us know a bit more of the nginx config if you are still having problems. Does the usual singleuser Jupyter server work for your users? |
All our services are served via a traefik proxy. And the error occurs on that production infrastructure when a user tried sharing a dashboard. On local installation everything run fine.
Unfortunately this is traefik and not nginx. I try redirecting every request to https but I don't think it is properly configured because I successfully execute the POST request on the insecure url - that of course failed at a latter stage as the oauth state was not correct. So definitely this is something to be done at the proxy level.
Single-user server and dashboard run fine if opened by their owner. The trouble comes when the authorize page is displayed to allow non-owner users to access dashboards. |
|
It sounds quite likely that the authorize page is where there is a problem - this is rarely hit in normal JupyterHub setups. In the simplest case, we could just drop the authorize page in the ContainDS Dashboards setup too - it's only really there to make it a bit more obvious what's happening, and is probably overkill for most situations. I'll take a look to see if I can reproduce this problem somehow anyway. |
|
My current results are:
|
|
I discover that JHub can generate and handle internal SSL certificates. Unfortunately if activated: A dashboard server is not starting. The error is: I presume this concerns
My knowledge on the subject is limited. But it seems wrong to skip it because of the reverse-proxy. |
|
Thanks for the update! To confirm, I can reproduce the original 'auth form must be sent from auth page' problem even using standard singleuser notebook servers (assuming that I force needs_oauth_confirm to always return True). When internal_ssl is True, I also see the dashboards fail to start (or more precisely, be able to be polled). This is because jhsingle-native-proxy doesn't currently respect the internal SSL settings. It would be a relatively easy fix to make this happen, just doing similar things to the NotebookApp when it starts its Tornado listener. However, I'm not sure why you think this will solve your original problem. In particular, does internal_ssl=True fix your original problem for standard singleuser notebook servers when you force oauth through the Authorize button page? I am also not convinced that internal_ssl will work correctly using named servers (even standard singleuser notebooks) since I notice that each new server generates new certs for the user, possibly invalidating those that any existing servers believe are in use - unless JupyterHub is holding these in memory for each spawner. I haven't checked fully. I haven't been able to replicate your setup when internal_ssl=True - i.e. I haven't been able to get this to work behind a reverse-proxy (in my case, I'm trying nginx) so if you can share more about your SSL config that would be great (if you haven't already solved the overall problem first...). So I think we need to see if:
Agreed we shouldn't skip the auth page just to solve this problem if we think it's important - but it's a genuinely open question as to whether this is useful for any particular JupyterHub installation. In any case, a fix (e.g. configuration update) would need to be on the JupyterHub side, or at least patched somehow from ContainDS Dashboards. |
|
The master version of jhsingle-native-proxy (to be released as 0.5.0 in a few days) respects the internal SSL settings and seems to work just like the regular notebook servers in a JupyterHub with internal_ssl=True. However, I'm still not sure if this solves the original 'auth form must be sent from auth page' problem. (I haven't been able to get internal_ssl=True working behind an extra proxy yet.) @fcollonval it would be fantastic if you're able to explain why you thought internal_ssl=True would help, and even better if you can test! |
|
Hey @danlester thanks for looking deeply into this. So the idea is quite dirty, but I hoped to get the request to the hub server switched back to https thank to the internal certificates. And so the scheme would be the same for the request and the If I got the JupyterHub structure correctly, the call stack looks like:
But this is only because I do not have admin rights on the reverse proxy. |
|
Great diagram! If you get a chance to try it out with the updated jhsingle-native-proxy, please let us know. Without using internal_ssl=True, I have been able to solve the problem with nginx. This may help you or others... In the nginx config I added: And made sure to start CHP of JupyterHub as follows: When nginx set X-Forwarded-Proto to 'https', CHP changed this to 'https,http' (until I started it with the --no-x-forward flag). It's possible that your Traefik (which you are using in place of my nginx) is already correctly passing https under X-Forwarded-Proto but that CHP is subsequently corrupting it by appending http. So could be worth trying the --no-x-forward flag. If not, maybe try adding a check to understand what is being passed through, by adding a log line to your JupyterHub code again: Ultimately, this may need changes to your Traefik config of course. I think it is probably a bit too obscure to change JupyterHub for this reason since it would normally be expected that you have admin access to Traefik. |
I used mermaid-js editor. Then copy paste the png export 😉
You rock this works. Thanks a lot for the help. |
fcollonval
changed the title
|
Thanks, this works in an apache https proxy environment too. In the site config:
In jupyterhub_config.py:
|
Describe the bug
A clear and concise description of what the bug is.
Best is to understand the problem is to look at the gif below.
To Reproduce
Clear steps to reproduce the behavior, including any of your own ipynb or py files if they led to the error.
Screenshots
Please add screenshots to help explain your problem.
The failure raises from https://github.com/jupyterhub/jupyterhub/blob/e5a6119505f89a293447ce4e727c4bd15e86b145/jupyterhub/apihandlers/auth.py#L277-282
But from the web dev tools, the
Refererheader matches the page URL.Configuration
Include as much jupyterhub_config information as you can - at least enough to understand which Spawner type you are using, and how your JupyterHub is deployed (e.g. The Littlest JupyterHub, or Zero to JupyterHub).
cdsdashboards 0.3.0
ldapauthenticator 1.3.0
jupyterhub 1.1.0
jupyterhub-base 1.1.0
spawner_class = (
"cdsdashboards.hubextension.spawners.VariableLocalProcessSpawner"
)
The text was updated successfully, but these errors were encountered: