Document our cluster design choices

[yuvipanda]

We have made many specific choices in how we have designed our clusters, and these should be better documented.

• Choosing Highly Available Kubernetes cluster masters (across GCP, AWS, Azure)
• General node configuration considerations
  • Determining type of disk (spinning disk, ssd) of the nodes
  • Determining OS type (ubuntu, container optimized OS, bottlerocket, etc) of the nodes
• Core Node Pool Configuration
  • Considerations for determining size of the VM
  • Considerations for determining number of nodes here
• Notebook node pool configuration
  • Considerations for determining the size (memory & CPU) of the VMs in notebook node pools
  • Profile Lists - offering multiple memory and CPU options for users on research hubs
• Dask node pool configuration
  • Using pre-emptible / spot node pools here for cost cutting
  • How users can select different kinds of dask nodes for different resource requirements
• Network Policy enforcement options
• Private Cluster support