Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spawn notebook servers using host network in Docker swarm using swarmspawner. #245

Open
sangramga opened this issue Jul 18, 2018 · 1 comment

Comments

@sangramga
Copy link

sangramga commented Jul 18, 2018

I need to spawn notebook server in Docker Swarm, and these containers must be in host network (instead of overlay network).

I am running Docker swarm with 2 nodes (one master and one worker). I run jupyterhub as docker service on master node as specified in Readme.I am using dockerspawner from Github master repo. Also I am using jupyterhub-0.9
My Dockerfile for jupyterhub:


FROM jupyterhub/jupyterhub:0.9

EXPOSE 8081

RUN /opt/conda/bin/pip install jupyterhub-hashauthenticator jupyter_client
COPY dockerspawner-master.tar.gz /etc/dockerspawner-master.tar.gz
RUN /opt/conda/bin/pip install /etc/dockerspawner-master.tar.gz

Command to launch jupyterhub as a docker service:

docker service create \   
  --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \    
  --mount type=bind,src=/etc/jupyterhub-swarm,dst=/srv/jupyterhub \    
  --name jupyterhub \    
  --network host \    
  --constraint 'node.role == manager' \    
  --detach=true \    
sangramga/jupyterhub-swarm:latest

I tried over-riding get_ip_and_port() method, but still the jupyterhub service is not able to detect the notebook container. I am unable to pinpoint host_ip since the users Notebook containers will be Spawning in Multiple hosts.

Find my jupyterhub_config.py file.

import os

c.JupyterHub.authenticator_class = 'hashauthenticator.HashAuthenticator'
c.HashAuthenticator.secret_key = 'secret'  # Defaults to ''
c.HashAuthenticator.password_length = 6          # Defaults to 6
c.HashAuthenticator.show_logins = True            # Optional, defaults to False
## The public facing port of the proxy
c.JupyterHub.port = 8000
## The public facing ip of the whole application (the proxy)
c.JupyterHub.ip = '0.0.0.0'
## The ip for this process
c.JupyterHub.hub_ip = '0.0.0.0'
#  Defaults to an empty set, in which case no user has admin access.
c.Authenticator.admin_users = {'tcupadmin'}
c.Authenticator.whitelist = {'notebook2','tcupadmin'}
c.JupyterHub.admin_access = True
network_name = "host"

from jupyterhub.utils import random_port
from tornado import gen
from dockerspawner import SwarmSpawner
from jupyter_client.localinterfaces import public_ips

class custom_spawner(SwarmSpawner):
    @gen.coroutine
    def get_ip_and_port(self):
        # because of the 'network_mode': 'host' settings above, the spawned containers
        # bind to the host, so to access them, we need to use the ip of the host
        return self.host_ip, self.port
@gen.coroutine
    def start(self, *args, **kwargs):
        self.port = random_port()
        # start the container
        ret = yield SwarmSpawner.start(self, *args, **kwargs)
        return ret

c.JupyterHub.spawner_class = custom_spawner
c.SwarmSpawner.network_name = "host"
c.SwarmSpawner.extra_host_config = {'network_mode': network_name}

notebook_dir = os.environ.get('NOTEBOOK_DIR') or '/home/jovyan/work'
c.SwarmSpawner.notebook_dir = notebook_dir
c.SwarmSpawner.image = 'jupyter/base-notebook:latest'
c.SwarmSpawner.debug = True

Following are Error Logs for jupyterhub Docker Service, when user tcupadmin is spawned

jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [E 2018-07-18 12:06:42.216 JupyterHub log:158] 500 GET /hub/user/tcupadmin/ ([email protected]
4.134) 10142.89ms
jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [W 2018-07-18 12:07:01.142 JupyterHub user:504] tcupadmin's server never showed up at http://127
.0.0.1:57446/user/tcupadmin/ after 30 seconds. Giving up
jupyterhub.1.lkm1x6ygr640@ip-172-30-0-173    | [W 2018-07-18 12:07:01.174 JupyterHub swarmspawner:121] Service jupyter-tcupadmin not found

After inspecting the user's notebook Service.
docker service inspect jupyter-tcupadmin shows the following output

"Endpoint": {              
           "Spec": {              
               "Mode": "vip",     
               "Ports": [         
                   {              
                       "Protocol": "tcp",                             
                       "TargetPort": 57446,                           
                       "PublishMode": "ingress"                       
                   }              
               ]                  
           },                     
           "Ports": [             
               {                  
                   "Protocol": "tcp",                                 
                   "TargetPort": 57446,                               
                   "PublishedPort": 30000,                            
                   "PublishMode": "ingress"                           
               }                  
           ],                     
           "VirtualIPs": [        
               {                  
                   "NetworkID": "hlemj2afn0ndkl1jny3lgpa2v",          
                   "Addr": "10.255.2.9/16"                            
               },                 
               {                  
                   "NetworkID": "wplqnqrchc6f49tntpukc7i6b"           
               }                  
           ]                      
       }

@Blizzke
Copy link

Blizzke commented Mar 6, 2019

I'm currently experiencing a similar problem and I was looking into just leaving the notebook on the regular hub network. Like this the proxying works. After the spawn I could use docker network connect to simply provision a second network interface in the container with the public IP. Trying to figure out how to run a command after a notebook is spawned at this point, perhaps I should create an issue here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants