Add basic site crawler to find 404s, etc.

[toolness]

This is a work-in-progress attempt to fix #269, or at least part of it.

Instructions

1. Generate the site using jekyll build. The _site folder should now contain the latest version of the site.

2. Run npm run crawl. It will let you know if it found any errors.

Notes

I decided to use node-simplecrawler because I've had experience using it in the past, and it seems reasonably fast and extensible.

Unlike 18F/content-guide#132, I decided not to go with html-proofer because it's Ruby-based, and I don't have a ton of experience with Ruby. I also heard rumors that we might consider migrating this site from Jekyll to Hugo; if that happens, we would likely remove Ruby entirely from this project, so I figured node might be a safer long-term option. We can always switch, though!

To do

Some of these can be filed as separate issues and dealt with in separate PRs.

• Consider making the crawler crawl more than just the index page and all the immediate resources it links to.
• Fix 404s reported by the tool, or log warnings if the 404s come from external sources. Currently these seem to be:
  • All the JS scripts referenced by ie-polyfill-scripts.html.
  • html5shiv.js, which is in an IE-only comment in head.html.
  • In the developer's guide, there's a broken link to CONTRIBUTING.md.
• Run the crawler as part of npm test (and therefore during CI).
• Don't report errors until we're done w/ the crawl; this way our list of referrers is accurate.

[toolness]

@donjo any idea what to do about these "less than IE9"-specific JS files that don't seem to exist? Do we even support < IE9 anymore? If not, I guess we can just get rid of the references...

[toolness]

Hmm, @donjo just mentioned on slack that our documentation page on accessibility mentions that we only support IE9 and above.

[toolness]

So I'm finding that a number of 404s are actually coming from external sources, like WHO_IS_USING_USWDS.md and the release notes, which are stored in the other repository. For these I'm thinking maybe we should log warnings but not errors, so that a broken link caused by one of those files doesn't break the whole build.

[donjo]

➕ to warnings for external source 404s. Agree.

[jseppi]

Looks good to me, and runs well!

[jseppi]

leaving off the ! would clean up the output a bit, and make it easier to copy the path

[jseppi]

Maybe check that _site has contents, otherwise the call to refs.every(...) later on (and potentially other things) will fail.

Here's output from running yarn crawl without first running yarn build:

/Users/jamesseppi/CODE/web-design-standards-docs/config/crawl.js:67
        const isWarning = refs.every(path => WARNING_PAGES.includes(path));
                              ^

TypeError: Cannot read property 'every' of undefined
    at notFound.forEach.item (/Users/jamesseppi/CODE/web-design-standards-docs/config/crawl.js:67:31)

[toolness]

Good suggestions @jseppi! Just incorporated them.

@donjo I think this is good to merge if you are OK with the functionality!

[donjo]

lgtm 🚢

[toolness]

woooooooot!