Evolve the next-generation image formats audit (AVIF)

[addyosmani]

tl;dr: Could we update the next-gen formats audit to provide AVIF savings estimates?

Background

WebP has been around for a number of years now with support recently hitting 92% of browsers worldwide. In the years since it was introduced, another promising format has landed in browsers - AVIF (supported in Chromium) (support is 62% globally), and coming to Firefox 88-90. AVIF offers up to a 50% improvement in compression over JPEG and some estimates suggest a 20% improvement over WebP.

Some developers are seeing savings from switching to AVIF of up to 25% (based on payload bytes) with up to a 90ms saving to LCP at P75.

Current Lighthouse audit

Our current serve images in next-gen formats audit source estimates potential savings based on conversion figures for JPEG compression ratios (8:1) and WebP (10:1).

Proposal

What if we evolved this approach to either:

1. Replace the current WebP savings estimates with AVIF savings estimates? OR
2. Display two potential savings columns: one for WebP and one for AVIF? (i.e encourage at least serving what most browsers support, but if you're already using <picture>, consider adding AVIF as a source too)

URL	Size	Potential savings: WebP	AVIF
cool-cats.jpg	75KB	30KB	50KB

What are the potential savings estimates for AVIF?

The AVIF team at Google have confidence in the 50% savings number (compared to JPEG) being accurate and the one we should use for tooling purposes (as of today) if we need a multiplier for estimates. Further evidence to back this number up includes:

• Netflix study on AVIF
• Netflix presentation given to the Alliance for Open Media in 2019
• Next quarter, they'll have a dedicated test framework for comparisons that may allow us to refine the savings estimates in the future, but for now the 50 number is fine to use.

lighthouse/lighthouse-core/audits/byte-efficiency/uses-webp-images.js

Lines 56 to 65 in 1ca0e7f

	static estimateWebPSizeFromDimensions(imageElement) {
	const totalPixels = imageElement.naturalWidth * imageElement.naturalHeight;
	// See uses-optimized-images for the rationale behind our 2 byte-per-pixel baseline and
	// JPEG compression ratio of 8:1.
	// WebP usually gives ~20% additional savings on top of that, so we will use 10:1.
	// This is quite pessimistic as their study shows a photographic compression ratio of ~29:1.
	// https://developers.google.com/speed/webp/docs/webp_lossless_alpha_study#results
	const expectedBytesPerPixel = 2 * 1 / 10;
	return Math.round(totalPixels * expectedBytesPerPixel);
	}

I think encouraging use of AVIF could help developers improve their Largest Contentful Paint when its an image. This may also give us a chance to revise the current audit text to include AVIF generally.

Would love to hear what folks think. cc @paulirish @connorjclark @patrickhulce

[brendankenny]

We should definitely revise the audit text to include AVIF (and calling webp "next-gen" may be a bit dated by now :).

Dropping webp advice would be kind of sad since it's so great that you can use it almost everywhere now. One option is to graduate it from "next-gen" to uses-optimized-images alongside the existing jpeg evaluation, though jpeg->webp is still a pretty valid suggestion if the dev don't want to mess with multiple encodings.

One extra layer here is that the static compression ratio estimate is only the backup for a measured compression estimate done in optimized-images, where we use jpeg and webp encoders already shipping in Chrome with moderate settings to estimate what could have been saved if an image was served with those codecs.

• is there an AVIF encoder shipping in Chrome today or only decoder?
• if yes, would the AVIF team be willing to expose it to the protocol in Audits.getEncodedResponse like webp/jpeg/png is today?
• if yes, how slow is the encoder? One issue with optimized-images is that it can already be pretty slow, so we keep a modest timeout on it (currently 5s) and fall back to the static estimates for any images not able to be re-encoded.

The AVIF team at Google have confidence in the 50% savings number (compared to JPEG) being accurate and the one we should use for tooling purposes (as of today) if we need a multiplier for estimates.

12:1 seems like it could be ok (if I'm understanding your suggestion right), though we want to lean very conservative with the estimate (see the webp comment you quote from uses-webp-images.js) since we obviously don't want to flag a bunch of already-well-optimized jpegs. We could always start with a slightly lower ratio and increase it as more data comes in, but it's also totally possible that ratio will hit the mark already.

[brendankenny]

#11464 also mentioned dropping JPEG 2000 references in the audit since it's pretty clear that's not happening and WebP is good to go if you need a "next-gen" solution in Safari today.

[brendankenny]

For reference:

• @patrickhulce's change adding the Audits.getEncodedResponse protocol method: https://chromium-review.googlesource.com/c/chromium/src/+/617991

• the relevant code today: https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/core/inspector/inspector_audits_agent.cc;l=65-69;drc=5c3a04375727c8861afb76a4b2a06bbb5ffe40c3

  String mime_type_name = StringView("image/") + encoding;
  ImageEncodingMimeType mime_type;
  bool valid_mime_type = ParseImageEncodingMimeType(mime_type_name, mime_type);
  DCHECK(valid_mime_type);
  return image_to_encode->EncodeImage(mime_type, quality, output);

From the look of this, if there are any plans to add AVIF support to canvas's toBlob or toDataURL() (so ParseImageEncodingMimeType and ImageDataBuffer::EncodeImage will have been updated), debugger protocol support would then be trivial (add an 'avif' to the supported encoding types coming over the protocol and it should take care of itself).

If that's not feasible today, we won't have customized, per-image advice and will probably miss more dramatic savings estimates in some cases, but we can still list good opportunities in using AVIF for next-gen encoding using a fixed ratio.

[addyosmani]

After discussing our encoding options further with the AVIF team:

• We don't currently have an AVIF encoder shipping in Chrome. An AV1 encoder for RTC shipped in M90 but it's not entirely clear if that included encoding support (and the extent to which it was stripped out). The writing code isn't in Chrome today for sure and would need to be added back. The tl;dr is that if we only wanted AVIF encoding for the Lighthouse audit here, that may be overkill.... :)
• Another option here, similar to what we're doing with framework tooling + image optimization would be to consider using WASM (e.g via Squoosh CLI) for AVIF encoding. That would potentially impact our binary size, but is an option.

If that's not feasible today, we won't have customized, per-image advice and will probably miss more dramatic savings estimates in some cases, but we can still list good opportunities in using AVIF for next-gen encoding using a fixed ratio.

I think a fixed ratio approach could also very much make sense here.

[patrickhulce]

Proposal:

• Apply the fixed ratio for AVIF:JPEG to our JPEG re-encoded estimate from OptimizedImages artifact, arrive at estimate A.
• Apply the fixed ratio for AVIF:WebP to our WebP re-encoded estimate from OptimizedImages artifact, arrive at estimate B.
• Take the average (max?) of estimates A and B and use that as our AVIF savings estimate for each image.

[addyosmani]

I would be supportive of Patrick's proposed direction for the implementation here. @brendankenny @paulirish @connorjclark wdyt?