Error adding new users from JupyterHub

[jhamrick]

Adding new users seems to succeed:

[I 2016-08-25 17:14:50.481 JupyterHub log:100] 200 GET /hub/admin ([email protected]) 68.75ms
[I 2016-08-25 17:15:02.518 JupyterHub docker_oauth:60] Created user foo with uid 1292
[I 2016-08-25 17:15:02.524 JupyterHub docker_oauth:60] Created user foo with uid 1292

but then when their server is started, it appears the user_id variable in the user state isn't set correctly, so it tries to fall back on the default user id which causes an error since it's in a docker container:

[I 2016-08-25 17:15:41.356 JupyterHub dockerspawner:333] Container 'jupyter-foo' is gone
[W 2016-08-25 17:15:41.356 JupyterHub dockerspawner:304] container not found
[I 2016-08-25 17:15:41.387 JupyterHub dockerspawner:333] Container 'jupyter-foo' is gone
[E 2016-08-25 17:15:41.394 JupyterHub user:237] Unhandled error starting foo's server: 'getpwnam(): name not found: foo'

Restarting the hub seems to refresh the database and allows the servers to be started, so it does seem the information is being saved into the database correctly -- so maybe there is some view of the database that isn't getting correctly refreshed?

[minrk]

Which JupyterHub version?

[ianabc]

I think I see the same behaviour in 0.6.1 but I might not be using the state dictionary correctly. I'm wedging in a uid during add_user with

       if isinstance(user.state, dict):
            user.state['user_id'] = self.user_id
        else:
            user.state = { 'user_id': self.user_id }
        self.db.commit()

restarting the hub allows me to login.

[ianabc]

I just tried with the current HEAD(s): jupyterhub/eaeec9f and jupyterhub/dockerspawner@14a336b but I get the same behaviour.

I can work around it by modifying _user_id_default in systemuserspawner.py to check user.state,

diff --git a/dockerspawner/systemuserspawner.py b/dockerspawner/systemuserspawner.py
index b249578..76c5932 100644
--- a/dockerspawner/systemuserspawner.py
+++ b/dockerspawner/systemuserspawner.py
@@ -115,7 +115,10 @@ class SystemUserSpawner(DockerSpawner):
         this will never be called, which is necessary if
         the system users are not on the Hub system (i.e. Hub itself is in a container).
         """
-        return pwd.getpwnam(self.user.name).pw_uid
+        if self.user.state['user_id'] > 0:
+            return self.user.state['user_id']
+        else:
+            return pwd.getpwnam(self.user.name).pw_uid

     def load_state(self, state):
         super().load_state(state)

This gets me in (after some redirect problems which I think are unrelated) but I don't really understand the traitlets system yet so I'm not sure that's a good idea.

[jhamrick]

I'm using 0.6.0.dev, apparently.

Good to know that putting that patch in SystemUserSpawner works -- I was thinking of trying that at least for a temporary workaround.

I think though there must be something that changed in JupyterHub or the Google OAuth plugin, because this definitely used to work!

I will be investigating in more depth now this morning.

[jhamrick]

Also, I find it a bit strange that when users are created, two lines appear from restuser:

[I 2016-08-25 17:15:02.518 JupyterHub docker_oauth:60] Created user foo with uid 1292
[I 2016-08-25 17:15:02.524 JupyterHub docker_oauth:60] Created user foo with uid 1292

which suggests somehow two requests are being sent, maybe? @minrk do you know what would cause there to be two log entries for that?

[jhamrick]

Ok, so I think the problem is the following:

1. user ORM object gets created without a user id. This creates a server for the spawner, but the id isn't set in the server because the user doesn't have an id yet.
2. authenticator creates the user and sets the id in the database, but it doesn't set it for the spawner.
3. spawner tries to launch the server, but doesn't know what the id is so throws an error.
4. after JupyterHub restarts, it's fine, because now the user's ORM object includes an id so the spawner gets created with an id as well.

Time to test out this hypothesis now...

[jhamrick]

Yes, that seems to fix it!

[minrk]

I'm not sure why it would create the user twice, that seems odd, but I'm glad you figured out a way to make it work.