[EPIC] Roll out a cost attribution system in all AWS clusters

[consideRatio]

This EPIC captures the AWS part of #4384:

Quick intro

The AWS cost attribution system is a simple Python webserver accessing AWS Cost Explorer API to summarize information about an AWS accounts billing data. This summarized billing data is then exposed via a k8s Service, making a Grafana instance able to access it as a datasource. There is a datasource plugin used for this in Grafana called Infinity, as it helps parse the JSON response provided by the webserver. The openscapes grafana instance currently has this system deployed, and the Grafana instance has a Dashboard defined within it that we can re-use.

Kinds of work involved

Most of the AWS specific parts have been done, so now working this epic will mostly involve less cloud specific parts, such as:

• terraform / eksctl
• grafana dashboard development like in jupyterhub/grafana-dashboards
• interaction and communication with openscapes and nasa-cryo community
• the deployer script (Python) and GitHub workflows
• public communication via a blog post

As seen in openscapes

You can inspect the dashboard live at https://grafana.openscapes.2i2c.cloud/d/b4dbba938e548f10/cloud-cost-attribution?orgId=1

Tasks

Beta Give feedback

1. [EPIC] Support attributing costs to individual hubs automatically on Openscapes #4453
   51 of 54
   +0
2. AWS cost attribution (1/4): write documentation on how to deploy the system to another community #4873
   +0
   
3. AWS cost attribution (2a/4): deploy to nasa-cryo #4874
   +0
   
4. https://github.com/2i2c-org/meta/issues/1467
5. AWS cost attribution (2c/4): convert Grafana dashboard into a .jsonnet template and version control it #4875
   +0
   
6. AWS cost attribution (2c.2/4): [Spike 2h] Figure out how to define an infinity query variable #4944
   +0
   
7. AWS cost attribution (2d/4): automate deployment of the grafana dashboard definitions #4876
   +0
   
8. [Spike, 1h max] Followup cost attribution chat #4919
   +0
   
9. AWS cost attribution (feedback 1): Distinguish total AWS account cost from the 2i2c cluster attributed costs #4925
   +0
   
10. AWS cost attribution (feedback 2): Add panel descriptions #4926
    +0
    
11. AWS cost attribution (feedback 3): Disable alerting tab #4927
    +0
    
12. AWS cost attribution (feedback 4): Clarify situation on what is to be rolled out for all AWS clusters #4928
    +0
13. AWS cost attribution: (feedback 5) Enable viewers to generate and share snapshots #4991
    +0
    
14. AWS cost attribution: automatically install infinity plugin and datasource for all our grafana instances #4962
    +0
    
15. AWS cost attribution (3/4): deploy to all other AWS clusters #4878
    +0
    
16. AWS cost attribution: Ensure new AWS clusters have cost attribution enabled #5047
    +0
    
17. AWS cost attribution: documentation about opting in to have hub specific infra for compute and home storage #5000
    0 of 3
    +0
    
18. AWS cost attribution (4/4): communicate about the functionality via a blog post and share it with the communities #4879
    +0
    
19. AWS cost attribution (unplanned addition, high prio): fix or find a path foward to enable cost attribution for AWS member accounts under 2i2c's AWS organization #4941
    +0
    
20. cloud cost dashboard: fix details and refactor to avoid repetition #4951
    +0
21. AWS cost attribution (unplanned addition): handle "Not found" error showing up #4948
    +0
    

[GeorgianaElena]

What's remaining to consider this epic done is:

• https://github.com/2i2c-org/meta/issues/1578
• AWS cost attribution (4/4): communicate about the functionality via a blog post and share it with the communities #4879
• AWS cost attribution: documentation about opting in to have hub specific infra for compute and home storage #5000

Also, we still didn't get an answer from communities in:

• earthscope: https://2i2c.freshdesk.com/a/tickets/2387
• jupyter-meets-the-earth: https://2i2c.freshdesk.com/a/tickets/2386
• nasa-ghg: https://2i2c.freshdesk.com/a/tickets/2389
• nasa-veda: https://2i2c.freshdesk.com/a/tickets/2389

[GeorgianaElena]

Also, we still didn't get an answer from communities

@jnywong and @jmunroe, are you ok with me leaving the keeping an eye on those support tickets task on you and not open a separate tracking issue for that? I figure we can open its own tracking issue if they require additional assistance.

[jnywong]

Hey @GeorgianaElena ! We are both keeping an eye on the FreshDesk so there's no need to open another GH issue for that.

Also, we still didn't get an answer from communities

There will be another couple of touchpoints to share docs with them and gather feedback in #4879 and https://github.com/2i2c-org/meta/issues/1569, so hopefully that stimulates engagement. Ultimately we can't force them to engage, but we can keep track and improve that over time.

[consideRatio]

I've unassigned myself from this effort, and want to highlight @GeorgianaElena's decisive leadership this recent sprint and is a natural candidate to be asked to pick it up officially!

[consideRatio]

Feature request not yet tracked: to have one or more of the panels in the dashboard convey costs from different node instance types, in practice breaking down the "compute component" further. It can be broken down based on node group name and/or instance type.

[GeorgianaElena]

I've re-assgined you again @consideRatio for posterity to this one since you were the one that led this effort.
I extracted the remaining two remaining bits of it (the openscapes feedback on Nov 19 and the blog post) into the big initiative tracking this! So closing this issue 🎉

I will also try to capture #4872 (comment) somewhere once I get more context about it.