tests(build): use firehouse smoke test runner to test bundle

[connorjclark]

This patch

1. provides an interface (firehouse) to run smoke tests with options providing a hook for runLighthouse and ways to modify the expectations. This has already been done internally to test Lightrider - this is just polishing that work and bringing it home.
2. uses firehouse to run a few smoke tests on a nifty patched CLI that uses a bundled lighthouse-core.

I have a followup patch for building a standalone firehouse and rolling it to DevTools, with a layout test that uses firehouse with properly filtered expectations. It'll be a local test at first (run only during release procedures), but eventually it'll be used by whatever process we come up with to integration test our DevTools channel.

Related:

#9501 (and #9501 (comment))

Need to land bundle tests before:

#9605 (changes to bundling script)
#9193 (removes only tests that uses bundled code)

[patrickhulce]

very clever stuff :) 💭 💡

[patrickhulce]

haha I don't know what this means but it's hilarious :)

[brendankenny]

ha, I was working on a similar change, though it's kind of the inverse of this, which allows unification of some stuff. Maybe we can combine powers.

[connorjclark]

ha, I was working on a similar change, though it's kind of the inverse of this, which allows unification of some stuff. Maybe we can combine powers.

Is there a branch?

[brendankenny]

ha, I was working on a similar change, though it's kind of the inverse of this, which allows unification of some stuff. Maybe we can combine powers.

Is there a branch?

I'll get this together now

[brendankenny]

I'll get this together now

FWIW I don't think we should block #9193 on this, since I'd like to propose a bunch of changes here :) I probably won't finish my proposal today, though.

Really the purpose of this PR is to automatically verify that we don't break the bundle, which isn't a super emergency since we'll be doing things like releasing to devtools in the next week or so, which will involve manual testing of the bundle.

Alternatively, we could land this PR and consider my proposal to radically alter it afterwards. I don't have a strong preference.

[connorjclark]

To be clear, the changes you want to propose will be related to the bundle test, not firehouse (the smokehouse test runner), correct?

[connorjclark]

I think the bundled test is self-contained enough to just push this and consider your changes later. Also blocked by this is #9605 (although, can wait) and further integration tests (although, could break out the firehouse changes to continue on that front).

[patrickhulce]

LGTM, we can pickup debate of firehouse API in Brendan's followup. With this landed we'll have an actual consumer we care about that must be satisfied too :)

[brendankenny]

To be clear, the changes you want to propose will be related to the bundle test, not firehouse (the smokehouse test runner), correct?

both

LGTM, we can pickup debate of firehouse API in Brendan's followup. With this landed we'll have an actual consumer we care about that must be satisfied too :)

My main issue is that if smokehouse has grown so many layers that it was easier to write a new script that does the same thing, and the replacement script is really simple (loop over runs, compare against expectations), the root problem should be pretty obvious.

[connorjclark]

and the replacement script is really simple (loop over runs, compare against expectations), the root problem should be pretty obvious.

We could replace the core smoke test runner with the same, it'd just need a mechanism for batching, if we want to keep that.

[brendankenny]

and the replacement script is really simple (loop over runs, compare against expectations), the root problem should be pretty obvious.

We could replace the core smoke test runner with the same, it'd just need a mechanism for batching, if we want to keep that.

exactly :)

(also github save comment state is apparently flaky :)