core(unminified-javascript): replace esprima with custom tokenizer

[patrickhulce]

Summary
Removes dependency on esprima for JS minification savings and replaces it with a custom scanner/token counter similar to our CSS one. Putting this up for discussion before hardening it and going down the edge case rabbit hole.

Pros

• Faster than esprima (down to ~750ms from multiple seconds)
• Removes another bundle dependency
• We control our own destiny with bug fixes, etc

Cons

• We maintain a pseudo-JS parser
• Not nearly as well tested (though given the problems we've encountered with esprima I'm not super confident anyway)

Anecdotal testing I've done so far is run on a couple sites and make sure the % savings are the same for each file.

Related Issues/PRs
fixes #5723

[brendankenny]

Personally think this is a good direction to go, particularly since we're only interested in testing such a superficial level of minification anyways.

It would be cool to be able to guess at local variable names and replace them with minified versions to really approximate the baseline level uglify/whatever minification of JS, but not as important and just more state to track.

Anecdotal testing I've done so far is run on a couple sites and make sure the % savings are the same for each file.

Post these once you're more seriously tackling this?

[paulirish]

hot. lgtm

[patrickhulce]

some stats from runs, byte numbers are LH-reported savings, the usual culprits have gone and fixed their minification savings I used to find (yay!), so if you've got good examples, happy to have them :)

Site	Before	After
The Verge	<2KB 0ms savings (0.5s)	<2KB 0ms savings (0.2s)
CNN	3KB 0ms savings (1s)	<2KB 0ms savings (0.5s)

Instead I used a few libraries during my testing, here's the results

Library	Full Unminified Size	Predicted Savings Before	Predicted Savings After	Real Savings
jQuery	75 KB	37 KB	37 KB	46 KB
Angular	297 KB	223 KB	210 KB	240 KB
React	15 KB	7 KB	7 KB	12 KB
Moment	31 KB	13 KB	13 KB	15 KB

[patrickhulce]

@brendankenny @exterkamp how is this lookin' to you folks?

fine if no opinion, just a bit on the big side to expect no comments ;)

[brendankenny]

LGTM! This is great. Bundle size goes way down.

Since we'll fully control this tokenizer (and likely not change it much), it does feel like we could run it over largish files and get repeatable results without much test churn, but I don't have a good suggestion for candidates for this (files under our control we have, but not ones we change so rarely that they'd function as good canaries...)

[brendankenny]

candidates for this

we could bring in a dev dependency at a fixed version that we won't ever be using, but could use the contents entirely for smoke testing the CSS/JS tokenizer...

[patrickhulce]

we could bring in a dev dependency at a fixed version that we won't ever be using, but could use the contents entirely for smoke testing the CSS/JS tokenizer...

I like it. What's the biggest, most complex front end dependency we can think of? :)

[patrickhulce]

Oh sorry! Was waiting on ideas for smoke test :)

Will do angular in a follow-up if no other suggestions.

[paulirish]

sg

[brendankenny]

Was waiting on ideas for smoke test :)

ha, oh yeah, sorry!