docs(variability): add dedicated variance doc #7175

patrickhulce · 2019-02-06T20:44:35Z

Summary
First stage of our war on variance: Basic documentation in our repo.

Adds a new document discussing sources of variance and links to additional resources.

brendankenny

LGTM! Just a few comments

brendankenny · 2019-02-07T20:43:30Z

docs/variability.md

@@ -0,0 +1,84 @@
+# Score Variability
+
+Lighthouse is often a developer's first exposure to web performance measurement. Whether you're new to performance measurement or an old pro, it's always good to refresh ourselves on all the complexity behind measuring websites.


I like reminding the reader of "the complexity behind measuring websites", but the rest of these two sentences seem unnecessary before the summary below

aight, removed

brendankenny · 2019-02-07T20:47:26Z

docs/variability.md

+
+## Summary
+
+Lighthouse performance scores will change due to inherent variability in web and network technologies even if there hasn't been a code change. Run Lighthouse multiple times and beware of variance before drawing conclusions about a performance-impacting change.


comma before even

maybe s/variance/variability to tie this to the section title below this?

brendankenny · 2019-02-07T20:48:18Z

docs/variability.md

+
+## Sources of Variability
+
+Variability in performance measurement is introduced via a number of channels with different levels of impact. Below is a table containing several common sources of metric variability, the typical impact they have on results, and the extent to which different Lighthouse runtimes are able to mitigate their effect.


different word for runtimes here? I thought it was referring to CLI/extension/devtools

can we also link up the throttling.md ?

done and done

brendankenny · 2019-02-07T20:51:55Z

docs/variability.md

+## Sources of Variability
+
+Variability in performance measurement is introduced via a number of channels with different levels of impact. Below is a table containing several common sources of metric variability, the typical impact they have on results, and the extent to which different Lighthouse runtimes are able to mitigate their effect.
+


maybe put a quick note about the source descriptions here in addition to the one below the tables ("See the next section for descriptions of the variability sources" or something)

moved it after the descriptions, I think it makes more sense there anyway 👍

brendankenny · 2019-02-07T20:54:46Z

docs/variability.md

+
+### Web Server Variability
+
+Web servers have variable load and do not always respond with the same delay. Lower traffic sites with shared hosting infrastructure are typically more susceptible to this. applied throttling partially masks these effects by applying a minimum request latency in its network throttling. Simulated throttling is susceptible to this effect but the overall impact is usually low when compared to other network variability.


"Lower-traffic sites"

"Applied throttling" (capitalize at the beginning of the sentence or don't capitalize "Simulated throttling" at the beginning of the next)?

and do we want to put some kind of typographic marker on those terms? Italics applied throttling or bold or something else to make it clear it's referring to a specific setting, not just describing something?

brendankenny · 2019-02-07T20:56:03Z

docs/variability.md

+
+### Client Hardware Variability
+
+The hardware on which the webpage is loading can greatly impact performance. applied throttling cannot do much to mitigate this issue. Simulated throttling partially mitigates this issue by capping the theoretical execution time of CPU tasks during simulation.


same thing with applied vs Simulated here and below

brendankenny · 2019-02-07T20:57:47Z

docs/variability.md

+
+### Browser Nondeterminism
+
+Browsers have inherent variability in their execution of tasks that impacts the way webpages are loaded. This is unavoidable for applied throttling as at the end of the day they are simply reporting whatever was observed by the browser. Simulated throttling is able to partially mitigate this effect by simulating execution on its own and only re-using task execution times from the browser in its estimate.


maybe "Simulated throttling is able to partially mitigate this effect by simulating execution on its own, only re-using task execution times from the browser in its estimate."

(and -> ,)

brendankenny · 2019-02-07T20:59:35Z

docs/variability.md

+
+### Page Nondeterminism
+
+Pages can contain logic that is nondeterministic that changes the way a user experiences a page, i.e. an A/B test that changes the layout and assets loaded or a different ad experience based on campaign progress. This is an intentional and irremovable source of variance. If the page changes in a way that hurts performance, Lighthouse should be able to identify this case. The only mitigation here is on the part of the site owner in ensuring that the exact same version of the page is being tested between different runs.


should we move this higher? I feel like a lot of the people who will be most interested in this doc will also be the kind of people running A/B tests and loading ads or something equivalent (changing content, sometimes videos, etc)

brendankenny · 2019-02-07T21:00:48Z

docs/variability.md

+
+### Client Resource Contention
+
+Other applications running on the same machine while Lighthouse is running can cause contention for CPU, memory, and network resources. Malware, browser extensions, and anti-virus software have particularly strong impacts on web performance. Multi-tenant or parallel server environments can also suffer from these issues. applied throttling is susceptible to this issue. Simulated throttling partially mitigates this issue by replaying network activity on its own and capping CPU execution.


Multi-tenant or parallel server environments can also suffer from these issues.

is this referring to running the server on the same machine as the testing browser?

this is referring to running lighthouse in multi-tenant server environments (like travis/AWS/etc) or multiple lighthouse/browser instances at once

added more language to clarify

brendankenny · 2019-02-07T21:02:46Z

docs/variability.md

+
+### Isolate External Factors
+
+- Isolate your page from third-party influence as much as possible. It’s never fun to be blamed for someone else's variable failures.


should we add a note somewhere about reducing variability like this has the possible tradeoff of making the numbers less realistic? e.g. If you test without ads you'll be able to test your own code more reliably, but the TTI may no longer resemble what users see on your actual production site

yeah that sounds good, the docs I pulled these snippets from talked about this more at length so a reminder here is a good call 👍

paulirish

this is great. thanks for writing it up

paulirish · 2019-02-07T22:22:06Z

docs/variability.md

+
+## Sources of Variability
+
+Variability in performance measurement is introduced via a number of channels with different levels of impact. Below is a table containing several common sources of metric variability, the typical impact they have on results, and the extent to which different Lighthouse runtimes are able to mitigate their effect.


can we also link up the throttling.md ?

docs(variability): add dedicated variance doc

4cc62df

patrickhulce requested a review from paulirish as a code owner February 6, 2019 20:44

brendankenny approved these changes Feb 7, 2019

View reviewed changes

paulirish reviewed Feb 7, 2019

View reviewed changes

feedback

7d4b331

paulirish approved these changes Feb 7, 2019

View reviewed changes

brendankenny merged commit c4313e7 into master Feb 7, 2019

brendankenny deleted the variance_docs branch February 7, 2019 23:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(variability): add dedicated variance doc #7175

docs(variability): add dedicated variance doc #7175

patrickhulce commented Feb 6, 2019

brendankenny left a comment

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

paulirish Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019 •

edited

Loading

brendankenny Feb 7, 2019

patrickhulce Feb 7, 2019

paulirish left a comment

paulirish Feb 7, 2019

		@@ -0,0 +1,84 @@
		# Score Variability

		Lighthouse is often a developer's first exposure to web performance measurement. Whether you're new to performance measurement or an old pro, it's always good to refresh ourselves on all the complexity behind measuring websites.


		## Summary

		Lighthouse performance scores will change due to inherent variability in web and network technologies even if there hasn't been a code change. Run Lighthouse multiple times and beware of variance before drawing conclusions about a performance-impacting change.


		## Sources of Variability

		Variability in performance measurement is introduced via a number of channels with different levels of impact. Below is a table containing several common sources of metric variability, the typical impact they have on results, and the extent to which different Lighthouse runtimes are able to mitigate their effect.


		### Web Server Variability

		Web servers have variable load and do not always respond with the same delay. Lower traffic sites with shared hosting infrastructure are typically more susceptible to this. applied throttling partially masks these effects by applying a minimum request latency in its network throttling. Simulated throttling is susceptible to this effect but the overall impact is usually low when compared to other network variability.


		### Client Hardware Variability

		The hardware on which the webpage is loading can greatly impact performance. applied throttling cannot do much to mitigate this issue. Simulated throttling partially mitigates this issue by capping the theoretical execution time of CPU tasks during simulation.


		### Browser Nondeterminism

		Browsers have inherent variability in their execution of tasks that impacts the way webpages are loaded. This is unavoidable for applied throttling as at the end of the day they are simply reporting whatever was observed by the browser. Simulated throttling is able to partially mitigate this effect by simulating execution on its own and only re-using task execution times from the browser in its estimate.


		### Page Nondeterminism

		Pages can contain logic that is nondeterministic that changes the way a user experiences a page, i.e. an A/B test that changes the layout and assets loaded or a different ad experience based on campaign progress. This is an intentional and irremovable source of variance. If the page changes in a way that hurts performance, Lighthouse should be able to identify this case. The only mitigation here is on the part of the site owner in ensuring that the exact same version of the page is being tested between different runs.


		### Client Resource Contention

		Other applications running on the same machine while Lighthouse is running can cause contention for CPU, memory, and network resources. Malware, browser extensions, and anti-virus software have particularly strong impacts on web performance. Multi-tenant or parallel server environments can also suffer from these issues. applied throttling is susceptible to this issue. Simulated throttling partially mitigates this issue by replaying network activity on its own and capping CPU execution.


		### Isolate External Factors

		- Isolate your page from third-party influence as much as possible. It’s never fun to be blamed for someone else's variable failures.

docs(variability): add dedicated variance doc #7175

docs(variability): add dedicated variance doc #7175

Conversation

patrickhulce commented Feb 6, 2019

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce Feb 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce Feb 7, 2019 •

edited

Loading