Major documentation structure for helping our team debug issues

[choldgraf]

Context

In our latest incident on CarbonPlan @yuvipanda and I had some conversation about ways to lower the barrier for people to debug our cloud infrastructure when incidents occur. We agreed it would be helpful to have some basic documentation to help our team members get started with debugging common things.

Proposal

We've got an outline of major areas of documentation to write to make it easier for people to use our docs in the debugging/operations process. Here's that document:

https://hackmd.io/omFosDsjS3-2UEtAIZstcA

Here are the documents outlined there that we should create

• Add tutorial on setting up your local environment #1176
• Expand documentation about our support charts #1891
• Write documentation on where to look for config #1144
• Document our logging setup #1890
• Document our dask gateway setup #1893
• Document our cluster design choices #1892
• Document our cluster design choices #1892
• Howto: Enable shared directory
• Document setting up GPUs on our clusters and access for our hubs #996 (comment)
• Logging: Evaluate and cost consistent logging for our clusters #2272

We should flesh out the major sections in that HackMD, and update this issue as we do so with refs to PRs that implement things. We can close the issue once we've got docs that cover the major parts of that HackMD.

Updates and actions

• Reorganizing how-to section #1177
• Add tutorial on setting up your local environment #1176
• Revisit our documentation to see if the checkboxes are actually relevant. #1826

[yuvipanda]

A more structured way to think about this is in terms of the 'objects' we have and the 'actions' that can be performed on them.

A starter pack would be:

Object	Verbs
Cloud account	Provision, authenticate, view web console, etc
Kubernetes cluster	Authenticate, explore objects in kubectl, look at logs, etc
Hub	View full config of, change config of, create, decomission, deploy change to, look at logs for, find out community rep, etc
User image	Create repo, setup image, debug changes, etc

A lot of these have detailed documentation and trianing provided elsewhere (kubectl for example), so we should strike a balance between writing our own custom docs, linking to upstream docs, and improving upstream docs.

[yuvipanda]

I think it'll also help us identify required training that we can ask folks to take as part of onboarding so they have time set aside to learn how to use and navigate the tools we use. We stand on the shoulders of giants and we gotta use them!

[damianavila]

This exercise would also help us identify which pieces we can build to help in the process (ie. a quick way to get a cmd line ready to write kubectl commands to start debugging).

[yuvipanda]

We are working on structure for more docs to be written to make support steward work less stressful - https://hackmd.io/omFosDsjS3-2UEtAIZstcA.

[choldgraf]

I see that the two PRs attached to this one have now been merged. Can we close this one? If not, can we define some glanceable deliverables that we can use to know when this issue should be closed?

[yuvipanda]

@choldgraf i've lifted the list of documents from the hackmd to the issue body. We can close this when those are written!

[yuvipanda]

From #1314 (comment), we should also develop guides for GPU debugging.

[choldgraf]

added to the list at the top 👍

[choldgraf]

I'd like to unassign myself from this one - I am more than happy to help with documentation, but I don't think that I will have the time to spearhead any of the efforts on this one. I would be happy to be tagged-in as a support person though, or to review PRs etc.

[damianavila]

1. Breaking up into smaller issues (@damianavila)
2. Revisit our documentation to see if the checkbox are actually relevant (@sgibson91 and @yuvipanda).

[sgibson91]

Comment migrated to #1826 (comment).

[damianavila]

@sgibson91, I moved over your comment into the dedicated issue referenced above.

[yuvipanda]

I'm going to close this one, as I think whatever improvements that were made during this push are complete by now.