[Documentation] Python Automation script and workflow for github wiki

[profishional]

Description

Motivation

Create a GitHub wiki (a separate git repo relative to this repo) for the documentation across this GitHub repo. GitHub wikis have better visibility and navigation for users.

Background

GitHub wikis lack version control and are entirely separate relative to this source repository. To solve both of these issues, we use the documentation/markdown files from the main repo to update the Github wiki repository.

To maximize the benefit of the wiki, a hierarchical view is created in a sidebar. However, the mapping of the markdowns files to the sidebar poses some challenges.

The PR is an attempt at a very simple solution. A GitHub action that tracks changes to a set of files and triggers a python script. The script picks up pre-written comments in the set of files to generate a sidebar. From the sidebars linking reference, it copies over the files from the main repo to the wiki repo.

Pre-Requisites to Merge this in

• Create a repository secret. Based on the legacy access rules, tick the repo permission box.

Outstanding issues

• Improve the format of the sidebar, so categories don't repeat. Rolling the main README of a category into the category heading so that the heading is also a link and not just text.
• Change the internal links of the Markdown files in the wiki to point to other paths in the wiki by either:
  1. A regex change to the current wiki file names (aka wiki titles)
  2. Changing the wiki file mapping to align with the internal links.

Issue

Fixes #133

Remaining/new issues

Type of change

Please mark the relevant option(s):

• New feature, functionality or library
• Bug fix
• Code health or cleanup
• Major breaking change
• Documentation
• Other

List of changes

• (Apologies in advance) Some formatting changes from auto linting of markdowns files. I don't like this as it has blown out the diff. I can try to remove if it's too messy.
• Added a GitHub action that tracks change to .md files, sets env variables and runs a python script
• Added a python script in a new directory called tools. Finds the path of .md files, maps them to the specified format described in the files, generates a GitHub wiki sidebar file and respective wiki files. Pulls the current wiki, syncs the files and pushes them.
• Added comments at the bottom of .md files found from find . -name "*.md" | grep -v -e "vendor" -e "app" in the form of <!-- GITHUB_WIKI: category/subcategory/.../file_name -->.
• Brief description of maintaining the wiki format comment in the docs/development/readme
• Brief mention and linking to the wiki in the main readme.

Testing

• make develop_test
• LocalNet w/ all of the steps outlined in the README
  d

Screenshot of output from example wiki on a personal fork: example wiki

Required Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have tested my changes using the available tooling
• I have updated the corresponding CHANGELOG

If Applicable Checklist

• I have updated the corresponding README(s); local and/or global
• I have added tests that prove my fix is effective or that my feature works
• I have added, or updated, mermaid.js diagrams in the corresponding README(s)
• I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

[profishional]

Just FYI @Olshansk; I am not able to set the Assignee, label, priority, etc. (as per the PR templates instructions) during or after creating a PR.
This is an example of a cross repo PR that has the buttons:

This is what I see while trying to creating a cross repo PR for pocket:

This is what it looked like after creating the draft PR, usually the words become clickable links to set the assignee etc.:

Maybe there is a different way, but those are the ones I was familiar with.

[Olshansk]

@okdas Could you help configure the secrets in our github repo to enable this via automation? See .github/workflows/wiki_sync_process.yml

[okdas]

@okdas Could you help configure the secrets in our github repo to enable this via automation? See .github/workflows/wiki_sync_process.yml

I somehow lost access to this functionality, I'll work on either getting access or setting the secrets with the infra team.

[Olshansk]

@okdas Could you help configure the secrets in our github repo to enable this via automation? See .github/workflows/wiki_sync_process.yml

I somehow lost access to this functionality, I'll work on either getting access or setting the secrets with the infra team.

Updated the settings. Let me know if you've taken another look

[okdas]

Thank you for taking care of this @profishional!

I've explained why tokens might not be the best way to access wiki with an alternative solution in comments in code. From my research, this should work just fine! However, if something won't work as suspected, I'll research what permissions we are going to need for fine-grained PAT.

[Olshansk]

@profishional Let us know if you have time/capacity to tend to the changes we mentioned so we can get this PR over the finish line. If not, @okdas and I can push it ourselves

[Olshansk]

@profishional LGTM!

Can you merge with main and resolve the merge conflicts? Since this branch is from your own repository, I don't have the ability to update it.

[profishional]

Ok I've solved the conflict. But can't merge "only those with write access to this repo can merge PR" @Olshansk.

[Olshansk]

@profishional I can merge it for you. Can you update the changelogs first please? Again, I would do it on your behalf, but this branch isn't in our main repo.

EDIT: Ignore. I will make an exception because of how minimal the changes in each repo are.

[profishional]

I don't follow. You've merged it but you'd like me to update the Changelogs as another PR?

[Olshansk]

@profishional Sorry about that. I wrote the original message but then added an EDIT at the bottom:

EDIT: Ignore. I will make an exception because of how minimal the changes in each repo are.

We'd be updating every changelog saying "Added a GITHUB WIKI tag", so the changelog diff would be greater than the actual diff, which I saw as unnecessary overhead. I made an exception and merged the changes as is.

I also just created a followup ticket that's adding it in the missing READMEs here: #488.

I think this is really awesome btw!! https://github.com/pokt-network/pocket/wiki