[Platform Initiative] Hub Scale Cost Monitoring for GCP

[Gman0909]

Productboard link: https://2i2c.productboard.com/roadmap/7947557-2i2c-roadmap/features/26823459

Description

Institutional leads, as well as department leads in large organizations, need to be able to justify their budgets and ensure they are being spent with value in mind. Business intelligence depends on data, and we want to make sure we build towards a data reporting infrastructure that can make a hub or constellation of hubs' usage and cost more transparent, enabling better decision making come budget time, as well as offering a sense of security and transparency over a service that is often perceived as being a high risk for cost overruns.

To that end, we would like to give community and institutional leaders the ability to monitor the cost and usage of a hub, or groups of hubs, provided by 2i2c.

The solution should provide a dashboard that automatically updates to reflect up to date aggregated costs and usage reports for each hub in a constellation, or the single hub an administrator has admin rights over. Data should be able to be exportable in the form of reports.

Additionally, we should investigate adding an option to share the dashboard with individuals outside of those with administrative privileges.

Typical use cases:

• Hub admin wants to keep an eye on their hub's costs to ensure they fall within budgetary lines
• Institutional lead wants to know which departments are costing the most, to ensure they are staying within budget
• Admins and institutional leads want to make sure their hubs are getting the usage they expect, to ensure they're getting their money's worth
• Admins want to see how many active users are on a hub over a defined period of time
• An institutional lead wants to print a CSV report they can use to plot a chart of their hubs' costs across departments for an upcoming budget review.

Scope

We already have a document listing the things cloud providers charge you for. The things we care about, in priority order, are:

1. Home directory storage (NFS)
2. Object storage (scratch and persistent)
3. Compute (nodes)

Each of these costs should be attributable to either:

1. A common pool that serves all of the hubs (primarily, the core node pool and staging hubs)
2. A particular hub

Attributing to individual users, or specific subgroups inside a hub, are out of scope.

Definition of Done

Admins of any 2i2c hub can access dashboards and reports where they can monitor up-to-date cost information for their hubs, and export reports with that same information.

Tasks

Beta Give feedback

1. [EPIC] Support attributing costs to individual hubs automatically on HHMI #4474
   +0
2. Sync on how we scale community guidance for Grafana usage #4551
   1 of 1
   +0