basehub: reduce prometheus-server's configured disk size (currently 100G)

[consideRatio]

I saw that the pangeo-hubs disk, sized to 200 GB, is only using 5.8 GB and is peaking to ~19 GB of memory when starting up. This makes me think that the default size of the prometheus helm chart, configured to be 8GB, is sufficient and we shouldn't need to adjust it.

For mybinder.org-deploy with lots of pods and node that has allocated 2000 GB in SSD, we have a 30.5G /data folder and a 30Gi memory request. Overall, we have over-allocated storage space significantly I think.

Action points

• Agree on a disk size for new hubs
  • Suggestions: 8 GB (prometheus default), 20 GB, 50 GB, 100 GB (retain current default)
• Change the value so new clusters get it, and copy over the existing values to existing clusters support chart config as we can't reduce a PVCs disk space request

Related

• Increase memory & disk size limit for pangeo prometheus #1906 increased the disk size for pangeo-hubs:

  I also actually manually increased the size of the disk (with kubectl -n support edit pvc), but
  that wasn't the problem. However, we need to persist the change regardless, so here it is.

• 2df2992 added the 100GB size configuration defined in basehub.

[yuvipanda]

As I go through the manual process of moving data from pd-standard spinning disks to pd-balanced (#3112), I'll also try to size the disks appropriately wherever possible