misc: lantern trace collection script #9662
Conversation
|
|
||
| ## Lighthouse Version | ||
|
|
||
| Check what version of Lighthouse WPT is using. You should use the same version of lighthouse for the desktop collection. |
There was a problem hiding this comment.
Is this important? WPT uses LH 5.2.0 rn.
There was a problem hiding this comment.
shouldn't be a big deal as long as we're not turning on different trace categories or something between the versions. observed metric definitions haven't changed in forever.
There was a problem hiding this comment.
actually I take it back, it'll be a big deal because we want LCP and layout stability to be computed for these that's the whole point of getting new ones. I guess we'll have to derive them after the fact from the trace if we want to avoid delaying this an entire release :/
There was a problem hiding this comment.
so: i'll collect the LHR, devtoolslog, and trace for local unthrottled desktop runs. And the LHR and trace for WPT runs. put them all in a giant zip and upload somewhere.
then make the new golden lantern set by taking the median runs for the local runs. and just run trace-processor on the WPT traces to get the metrics.
There was a problem hiding this comment.
right-o, but since we don't have LCP metric yet, I would triple check that the traces you're getting have the LCP and layout stability events in them at least and we don't need to enable some other category to get them.
|
|
||
| ## Lighthouse Version | ||
|
|
||
| Check what version of Lighthouse WPT is using. You should use the same version of lighthouse for the desktop collection. |
There was a problem hiding this comment.
shouldn't be a big deal as long as we're not turning on different trace categories or something between the versions. observed metric definitions haven't changed in forever.
| apiUrl.searchParams.set('k', WPT_KEY); | ||
| apiUrl.searchParams.set('f', 'json'); | ||
| // Keep the location constant. Use Chrome and 3G network conditions. | ||
| apiUrl.searchParams.set('location', 'Dulles:Chrome.3G'); |
There was a problem hiding this comment.
Is this the id for "3G Fast"? They have a separate "3G" profile that is slower than what we target. Dulles_MotoG4:Moto G4 - Chrome.3GFast is the location value I think we want
| // Disables some things that WPT does, such as a "repeat view" analysis. | ||
| apiUrl.searchParams.set('type', 'lighthouse'); | ||
| apiUrl.searchParams.set('mobile', '1'); | ||
| // apiUrl.searchParams.set('mobileDevice', '1'); |
There was a problem hiding this comment.
these are the parameters that are sent with the request made through the UI
lighthouseTrace: 1
vo: 8508357c0fc2a8119c04362b28b5bf8778ec1110
vd: 2019-09-12T20:59:51 00:00
vh: 4dc4641070f0dccf7187f77ce883f504246a3fec
url: https://patrickhulce.com
where: Mobile_Dulles_MotoG4
browser: MotoG4 - Chrome
location: Dulles_MotoG4:Moto G4 - Chrome.3GFast
bwDown: 1600
bwUp: 768
latency: 150
plr: 0
runs: 1
fvonly: 1
lighthouse: on
mobileDevice: MotoG4
traceCategories: blink,v8,cc,gpu,blink.net,netlog,disabled-by-default-v8.runtime_stats
not sure which of those are necessary to get the job done but perhaps bwDown, bwUp, latency are all necessary too not just location?
There was a problem hiding this comment.
bwDown / bwUp sets the network stuff ... doesn't location = 3G do the same?
| urlIndex++; | ||
| } | ||
|
|
||
| log.log('done! archiving ...'); |
There was a problem hiding this comment.
fwiw I think I had to break up the entire trace set into 3-4 archives or things like google drive wouldn't take it. that could probably just be done manually later, but just a heads up
Co-Authored-By: Patrick Hulce <[email protected]>
|
Collection script is good and is running in GCP right now. When done, will follow up with a new PR updating the baseline + adding LCP. |
| @@ -24,7 +24,7 @@ rm -rf lantern-data/ | |||
| mkdir -p lantern-data/ && cd lantern-data | |||
| echo $VERSION > version | |||
|
|
|||
| gsutil cp gs://lh-lantern-data/lantern-traces-$VERSION.tar.gz lantern-traces.tar.gz | |||
| DIRNAME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" | ||
| LH_ROOT_PATH="$DIRNAME/../../.." | ||
| cd $LH_ROOT_PATH | ||
|
|
||
| if [[ -f lantern-data/version ]] && [[ "$VERSION" != "$(cat lantern-data/version)" ]]; then | ||
| echo "Deleting old lantern data." |
There was a problem hiding this comment.
might be nice to give the user a chance to cancel this before deleting, I keep some debug stuff in that folder :D
| echo "Deleting old lantern data." | |
| echo "Version out of date. About to delete ./lantern-data..." | |
| echo "Press any key to continue, Ctrl+C to exit" | |
| read -n 1 -r unused_variable |
There was a problem hiding this comment.
This would hang travis ci tho?
There was a problem hiding this comment.
could throw a $CI in there then :)
I only ran it with a few urls @ 3 samples. See example output summary file below. I don't think the format is terribly important, I just wanted a nice formatted way to query the data during later analysis.
Seeking feedback on the data being collected and the options for LH / WPT being used before running the full suite.
Just collecting the traces, since the devtoolsLog can be generated from a trace.Next steps would be to democratize the analysis of these traces :)