[Request deployment] New Hub: Smithsonian

[jmunroe]

Important dates

• Target start date: April 15, 2023
• Required start date: May 1, 2023
• Any important dates for usage: N/A
• Pilot runs until Oct 31, 2023

Hub Authentication Type

GitHub (e.g., @MyGitHubHandle)

First Hub Administrators

Mike Trizna, @MikeTrizna
Rebecca Dikow, @rdikow
Alex White, @aewhite100

[GitHub Auth only] How would you like to manage your users?

Allowing members of a specific GitHub organization

[GitHub Teams Auth only] Profile restriction based on team membership

Organization: @Smithsonian

(no teams yet -- may be added in the future)

Hub logo image URL

https://logo.si.edu/wp-content/uploads/2018/07/logo_primary.svg

Hub logo website URL

https://www.si.edu/

Hub user image GitHub repository

TBD

Hub user image tag and name

TBD

Extra features you'd like to enable

• Specific cloud provider or datacenter (otherwise GCP)
• Dedicated Kubernetes cluster
• Scalable Dask Cluster

(Optional) Preferred cloud provider

AWS

(Optional) Billing and Cloud account

None

Other relevant information to the features above

Use smithsonian.2i2c.cloud as URL

Please deploy to us-east-2

GPUs are not required today, but may be in the future if that affect choice of zone.

For now, only use a machine type 4 CPU/32 GB.

If additional information is required, please ping @jmunroe first.

Tasks to deploy the hub

• 1. Deploy information filled in above
• 2. Engineer who will deploy the hub is assigned
• 3. If using GitHub Orgs/Teams Auth, Engineer is given Owner rights to the org to set this up.
• 4. Initial Hub deployment PR
• 5. Administrators able to log on -> Hub now in steady-state

[consideRatio]

The JupypyterHub is setup and ready for preview

Hi @MikeTrizna @rdikow @aewhite100! I'm with 2i2c and have setup your JupyterHub installation. I have some followup actions (1), a question (2), and some initial information (3, 4) below.

1. Help to finalize GitHub based authorization

   I've configured that members of the https://github.com/smithsonian GitHub organization are authorized access to three locations:

   • https://grafana.smithsonian.2i2c.cloud
   • https://staging.smithsonian.2i2c.cloud
   • https://smithsonian.2i2c.cloud

   For the authorization to work, a admin of the https://github.com/smithsonian GitHub organization needs to accept that members give out information if they belong to the organization. If either of you is an admin of the smithsonian GitHub organization, please attempt to login at each of the websites above and press the "Grant" button next to the smithsonian organization when presented with a request like this:
   

   In the case none of you are admins, please visit the websites and press a "Request" button next to the smithsonian github organization. Then, contact someone that is a GitHub Organization admin and ask them to approve the OAuth Application requests for: smithsonian-grafana, smithsonian-staging, and smithsonian-prod as documented by GitHub here.
   Following this, you and other members of the smithsonian GitHub organization should successfully be able to login!

2. Is Grafana dashboard access acceptable?

   I've granted access to https://grafana.smithsonian.2i2c.cloud for all members of the smithsonian GitHub organization. It provides dashboards of information such as memory use, total number of users, etc. Let me know if this is acceptable, or if you wish to restrict access of that to only a few users. I recommend granting everyone access as it can be used to help individual users learn about their memory use etc. The most sensitive information is perhaps to see if other users have been active or not and during what times.

   If you wish to restrict the access to a few users, please provide me with emails to grant access to. Either directly here or by emailing [email protected].

3. About the initial user environment

   Right now, users starting servers via JupyterHub at https://smithsonian.2i2c.cloud will use the quay.io/pangeo/pangeo-notebook:2023.02.27 Docker image. If you wish, you can build your own user image, for example by using the template in https://github.com/2i2c-org/hub-user-image-template.

4. About users sharing nodes (machines)

   Right now, all users will start their servers (docker image containers) on a 4 CPU node (machine) with up to 32 GB of memory. In other words, multiple users can start on the same machine. Each user is by default guaranteed a 1GB share of memory and won't risk experience memory issues if they stay below that. The less memory requested, the more users can fit on the node, the cheaper the cloud bill.

   A users can opt to request more guaranteed memory though, and should if they find themselves using more to avoid issues. And you as community representatives can communicate with us at 2i2c if you wish for the default to change, from for example 1GB to something else. This is how it can look when users start their servers.

   

   If a user has reqested some amount of memory, but is in practice using more, users may be kicked off the node if it starts running out of memory. I suggest that you check how much memory you and other users are using overall. In JupyterLab you can see how much memory is used in the button left statusbar:

   

   Using grafana, and specifically this grafana dashboard, you can learn how memory and CPU use change over time and such as well to make better informed choices. Below is an example of me starting a user server over a few minutes.

   

[consideRatio]

Hi @MikeTrizna @rdikow @aewhite100, have you had time to trial the jupyterhub setup?

It would be good to know that 1 and 2 in #2323 (comment) is resolved before my vacation starting April 29th.

[MikeTrizna]

Thanks so much, @consideRatio for setting this up!

Regarding question 1 above, I am an admin of the Smithsonian organization, but it looks like it is already configured to provide membership access. See the screenshot below.

We are able to use GitHub logins to access https://staging.smithsonian.2i2c.cloud/ and https://smithsonian.2i2c.cloud/, but not https://grafana.smithsonian.2i2c.cloud/. That one gives the following error:

[aewhite100]

Hi @consideRatio sorry for the delay. Very exciting to see the new setup! Yes we've been able to do some basic exploration and are happy to see everything up and running - 1 from above is working. The grafana dashboard is not accessible as @MikeTrizna mentioned.

Once we get those issues fixed, we think it would be best to restrict access to the grafana dashboard to us three for now. [email protected], [email protected], and [email protected].

Thanks for the help we hope to have more time to play with more custom user environments, especially those including RStudio and GIS capabilities. All of the other documentation you provided in 3 and 4 is very useful! We will have more time to mess with multiple users once we have our colleagues take a look.

[consideRatio]

Thank you for the follow up!

Mike the green checkmark you observed indicates the "Press Grant button" step is complete, excellent!

Grafana access

@MikeTrizna ah the grafana authentication issue stemmed from the Smithsonian github organization was registered with a capital letter, and Grafana has a bug by being case sensitive when reading configuration about allowed_organizations. I reported this in grafana/grafana#66876 and submitted a fix for it in grafana/grafana#66879 :)

Based on feedback from @aewhite100 above, I've now disabled the GitHub based authentication and sent a grafana username/password invite to the emails: [email protected], [email protected], and [email protected].

---

I see the initial setup of the JupyterHub for Smithsonian as complete now. Is there something else you wish to see done for this hub as part of the initial setup @MikeTrizna @aewhite100 @rdikow?

[damianavila]

I see the initial setup of the JupyterHub for Smithsonian as complete now.

I will call this one done by now, please submit tickets through support if you need further changes. Thanks!