Doc-only tests in CI

[Trott]

Recently, a doc-only commit broke CI. Code in the doc is extracted and run during CI tests. Rod Vagg fixed the problem but it makes me wonder if it's time to revisit the "no need to run doc-only PRs through CI" practice.

@jbergstroem suggested that perhaps there could be a doc-only CI job that only runs the relevant tests that might be affected by doc-only changes. (As far as I know, that would basically just be a subset of make test-addons at the current time, but there may be other things I'm unaware of.)

Prior conversation along these lines: nodejs/node#4411 (comment)

[MylesBorins]

I also think it is really worth bringing up the question if we should be running automated tests against examples in the docs.

If we broke that out into it's own testing suite we could have a CI job to run only those tests, and just tack that CI job onto the current stack.

This could end up being very useful for LTS

[jasnell]

Having a docs-only CI that ran only the potentially affected tests would be ideal. Having it clearly documented that there are, in fact, tests that depend on the docs is even more ideal

[chrisdickinson]

I'd like to second what @thealphanerd suggested — instead of pulling out code examples from docs, generating tests from them, and running them, we should just turn those examples into real tests.

Doc CI is a thing that should happen separately and should probably address different concerns than "ensure the addon API works as expected."

[jbergstroem]

I think we should do both. Having a doc-specific test run could focus on other areas of testing.

[orangemocha]

fwiw, node-test-commit has a pure_docs_change option under NODES_SUBSET. Right now no tests are run. It was meant as a placeholder for docs tests.