Creating a branch of the Docs using RunKit

[bnb]

As a part of the @nodejs/website-redesign work, we've gone through some work to analyze a way to improve some of the interactivity of the documentation on the website once we re-launch with the work the Website Redesign team is working on.

Early on, RunKit approached us with an interest to help improve the docs and make them more interactive while degrading gracefully.

My personal biggest concern was offline mode and not blocking the Docs for those who have JavaScript disabled – both of which are entirely addressed by the demos that the RunKit team has provided.

They've also been going to great lengths to ensure that some of the edge cases we have are addressed entirely within the platform.

@tolmasky from the RunKit team has requested that we begin to spin up an initial setup of RunKit enabled docs to help load-test the RunKit servers. This is purely to help see if they're going to need to set up a dedicated server for Node.js or if their existing infra will suffice.

I offered to be a bridge to Core to begin to help process this request – not sure what the barriers on the Docs side will be, but happy to help connect the dots and do what we need to to make this work 👍

Also wanted to mention that the RunKit team has been incredibly willing to help out and bend over backward to enable the Website Redesign team to succeed in revamping the Node.js website and improving the UX for developers. Huge thank you to them for all the work they're doing 🙌

[amiller-gh]

Huge thank you to the RunKit team – the redesign couldn't move this fast without them. Also happy to help un-block where I can 👍

The goal here is to create a better API docs experience only through build changes. No source documentation should have to change to bring an improved experience to the site. (Although I'd love to emerge from this process with some proposals on how we can improve the format and drafting experience of existing API docs, those changes will be outside the scope of this project).

I think enabling the RunKit folks to quickly iterate on the existing docs infrastructure, and complete a proof-of-concept of the final integration, is the best way to move forward here. Once we have a robust example of something close to what we'd like to see in production, we can come back and evaluate the nitty-gritty of the build changes in a PR. The Website Redesign team and RunKit will work closely to ensure the UX for the updated API pages is 💯.

I know the RunKit team has already been poking at the API docs a bit already, but I'm not sure how in-depth you've played with the Node.js docs build. After a quick look, here is my understanding of the current docs build, (someone please correct me if I'm off-base, or provide more detail if needed!):

• Docs build is kicked off by the doc-only task in the Makefile
• The /tools/doc package manages generating the docs assets at /out/doc/api.
• Entrypoint is generate.js
• All files are run through preprocess.js before being processed by json.js or html.js depending on their file type. These two files are where most of the proprietary magic seems to happen.
• All static resources for docs live in the /doc directory.
• Each page is generated using the template html file located at /doc/template.html, with placeholder tokens (ex: The __CONTENT__ token ) being replaced as needed.

@tolmasky, do you think your team needs more context than this to get started on an initial proposal?

Are there any other concerns that RunKit and the @nodejs/website-redesign WG should be aware of as we start hacking? Ex: No online resources required for docs build, docs build time concerns, docs tests to validate against, etc?

[tolmasky]

Thanks @bnb and @amiller-gh! I believe this is indeed sufficient context for us.

As already mentioned, we are wanting to do a very limited initial trial with no design changes. Just in case anyone saw our design demos in Berlin ( http://nodejs-org-runkit-demo.com/buffer/ ), we are not doing anything like that. Instead we are hoping to find a few key existing source examples in the API docs that can immediately have the switch flipped to "runnable". To provide a little background of what we'd like to achieve from this:

1. Ensure we can provide these feature in the least obtrusive way possible. For past projects like lodash/ramda/etc we've usually just written a plugin for whatever markdown tool they're using. My assumption is we'll be doing the same here. Ideally, we'd like to make it so that instead of doing the following:

   ```javascript

   You instead can do something like:

   ```javascript runnable

   The nice thing about this approach is that every markdown processor we've tested so far ignores everything past the language, so GitHub will for example still render the markdown docs correctly (with syntax highlighting), but it also allows an easy way to incrementally "turn on" snippets as runnable. However, if this is deemed too intrusive for this initial run, we are happy to also find a way to hardcode which specific examples to turn on in the build files themselves.

2. Iterate and test a few nodejs-website-specific features. For example, we've discussed not allowing the RunKit embeds to show the 0.10 option to users. We'd like to test some non-obtrusive ways of pulling this off like simply looking at the window URL to determine the default engine (for example, if window.href has /9, make the the selected engine "9" and turn off all other non-LTS engines, etc.). Again, this is stuff that we can put into config files, or even into the markdown itself, but we're striving for a first-pass "zero RunKit stuff littering the codebase" approach.

3. Ensure we can appropriately handle traffic. We'll actually probably be deploying on our demo server (vs. the main RunKit server) so that 1) we're testing the latest bits we've written specifically for this project and 2) we can more easily monitor all this.

4. Get early feedback from Node contributors about all this functionality.

5. From this as Adam mentioned, I'd also like to also generate some suggestions about the best way to design this feature-set that takes into account how the node contributors want to interact with it themselves.

Thanks!

Francisco

[devsnek]

sounds fantastic!

i like the javascript runnable just because it reminds you as you're editing an example that it should be designed around having good logging/etc where other examples might not need that.

[rubys]

FYI: I'm in the process of converting this build process from using marked to remark. See: #21490 and #21697.

What I would like to offer in the way of help is that if somebody can define a mapping of input to output, I can write a unified plugin that will perform that transformation.

Currently:

```javascript runnable
console.log('Hello world!');
```

Parses as follows:

{
  "type": "root",
  "children": [
    {
      "type": "code",
      "lang": "javascript runnable",
      "value": "console.log('Hello world!');",
      "position": {
        "start": {
          "line": 1,
          "column": 1,
          "offset": 0
        },
        "end": {
          "line": 3,
          "column": 4,
          "offset": 54
        },
        "indent": [
          1,
          1
        ]
      }
    }
  ],
  "position": {
    "start": {
      "line": 1,
      "column": 1,
      "offset": 0
    },
    "end": {
      "line": 4,
      "column": 1,
      "offset": 55
    }
  }
}

As you can see, the necessary data is present.

[vsemozhetbyt]

Some corrections and caveats:

1. Preprocess phase was eliminated recently in tools: build doc/api/all.html by combining generated HTML #21568 and tools: build all.json by combining generated JSON #21637
2. Any changes in doc generation may also need to address tests in https://github.com/nodejs/node/tree/master/test/doctool

[vsemozhetbyt]

Also: if code block tags are changed into javascript runnable, we may need to check if this does not break code block linting by eslint-plugin-markdown.

[tolmasky]

@rubys thanks for the heads up, sounds like I should maybe just be working off of that branch?
@vsemozhetbyt ah thank you very much, will definitely make sure to look into that!

[rubys]

@tolmasky There is no guarantee that that branch will be approved; and even if it is, it may look different when it does land. The most I can do at the moment is make you aware of this work and to offer to help if there is some impact.

[ghost]

For things that should be discussed in the community community

Is it a typo in the cc-agenda label description?

[vsemozhetbyt]

@trvsapjiac Thanks, fixed)

[tolmasky]

@rubys makes sense. I'll make sure to follow up with you as I do this, I know it can be stressful to have someone in the same code during a big PR and the last thing I want to do is create future merge conflicts. Please feel free to direct me to change anything through this process if it will potentially make things harder for you.

[bnb]

Removed CC-agenda label since we discussed it last meeting 👍

[sagirk]

Awesome! Been watching @nodejs/website-redesign for a while now. I just wanted to say that I love the direction that this is going in! ❤️

[tolmasky]

Thanks for the kind words @sagirk !

Just wanted to give a quick update -- we're making good progress on this, we're familiarizing ourselves with the new remark code (which we've grown quite a liking too -- we're actually changing some of our own internal tools to use remark too!)

[tolmasky]

Just wanted to post an update. We have an initial working version pushed here: https://github.com/runkitdev/node/tree/runkit-embeds. Still a number of things to do, but this is a pretty good proof of concept that we can do this initial take with very minimal changes which was our primary goal. Just some thoughts/items:

1. Currently the code highlighting on nodejs.org is done on the client with the assets/sh_javascript.min.js script that is included. This makes the runkit swap a tiny bit more complicated than it needs to be, so I am investigating just having the highlighting happen at the build step. If we do this, then this PR will have the added benefit of actually making the node site have less resources and load faster after this PR, which would be a cool bonus.
2. We're still choosing the key examples to turn this on for.
3. This current version uses "stock" RunKit vs. the node-only server we'll be using when we go live. We just need to spin up the AWS machine to do that.
4. We've just done this off master for now, not sure if we need to do anything off any release branches or anything, but I figure we'll be told about that when we actually make the PR.

[tolmasky]

My latest commit removes the dependence on a client-side script to syntax highlight. This makes the runkit stuff easier, but has the added benefit of making the load on all the pages faster, as well as highlighting all the non-js code as well (for example, C++ code, etc.). It is now in a more or less finished state, we'll be spinning up the dedicated AWS machines this weekend probably, so I'll be creating the pull request sometime later this week. Thanks everyone!

[Trott]

@bnb What is wr-agenda? This is the only issue with that label.

[tolmasky]

I believe (but @bnb should correct me if I'm wrong), this was just because we were set to discuss this during one of the website redesign YouTube meetings. For a quick update here - the PR is basically ready for checking, we're just waiting on eslint to release the changes we submitted to them in npm (they've already accepted them into master about a week ago).