Propose backwards-compatible improvements to compression

[EvanHahn]

This proposes a method to shrink the size of npm packages by about 5% while maintaining backwards compatibility.

[EvanHahn]

I missed the meeting where this was discussed, but I took the following notes from the recording:

• A proof of concept would be useful.
• Concerns about how to run Zopfli came to the fore because the project is written in C, not JavaScript. There were three options proposed: port it to JavaScript, use it in WebAssembly, or request its addition to Node core.
• The RFC can be accepted without a full solution. In other words, it can be an intent to implement, not a full implementation.
• Overall, this proposal is worth evaluation.

I built a proof of concept that uses WebAssembly to repack an npm tarball and report any size savings. You can run it in your browser and see the source code.

I plan to attend the meeting on June 15. Please let me know if you have any questions before then!

See also:

• Universal Zopfli, a WebAssembly-powered port of Zopfli, used in my demo
• wasm-zopfli, another WebAssembly implementation
• zopfli.js, a 9-year-old port of Zopfli that uses Emscripten

[EvanHahn]

Finally got around to thinking more about this, and decide it's not worth adding to the npm CLI.

The pros:

• Newly-published packages would be smaller.

  It's hard to give exact figures here, but I tried recompressing the latest version of the top 250 npm packages with Zopfli. All packages got smaller.

  • Median bytes removed: 273.5
  • Median percentage removed: ~4.3%
  • Mean bytes removed: 4148.668 (~4 KiB)
  • Mean percentage removed: ~4.7%

  This was only after 1 Zopfli iteration. Increasing the number (for example, to 100) would further shrink packages.

• This would be backwards compatible.

  Zopfli creates valid gzip data. Unlike introducing new compression schemes, like Brotli, this proposal would be entirely backwards compatible.

The cons:

• It doesn't retroactively apply to old packages.

• It's more complex for the CLI.

  Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity.

• Publishing would be slower. Sometimes, much slower.

  For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes.

I still think this is worth doing, but probably "in userland". More valuable, I think, would be finding a way to introduce better compression algorithms like Brotli.

I'm going to close this issue after doing this research. Thanks for your time!