Rethink how @embroider/test-setup pulls in Embroider

[simonihmig]

Based on #1328 and a lively discussion on the (host-absent 😉) Embroider Office Hours (amongst others @mansona @NullVoxPopuli @gossi @void-mAlex and me), I'll try to summarize our thoughts here:

Problem

Due to all packages of the monorepo being released in lock-step historically, @embroider/test-setup logic for pulling in Embroider dependencies in ember-try addon test scenarios was to use the same version as its own package.json. But with packages now released independently, this concept falls apart. Currently this leads to Embroider effectively being pinned to 1.8.3 (no carent/tilde modifier here!) for any addon that relies on v1 of @embroider/test-setup. Even when ignoring v2 of Embroider, we should get at least the latest 1.9.x release. This has unfortunate consequences, like for example not being able to get tests passing that use the more recent {{unique-id}}, even with the most recent Ember versions, as Embroider needs to have special understanding of it, and that landed only after 1.8.3...

Possible solutions

Add the caret modifier

This is what #1328 is doing. However, even when that would fix the short-term problem, it kinda defies the purpose of not having lock-step anymore. There is no correlation between @embroider/test-setup's version and that of the core packages (core itself, compat and webpack). So for whatever reason, it could become necessary to do some breaking change releases for it, causing it to try to pull in v3 of the core packages, that doesn't even exist. This could happen accidentally, maybe not even catched by CI.

Also this implies that all core packages are always released in lock-step, which might not be true (example: wbepack 6 comes out, we need to bump @embroider/webpack, but not necessarily core or compat.

Have the version hardcoded

We could hard-code the (caret) versions of the core packages in @embroider/test-setup. This way it is our call when to bring in new major versions, and it wouldn't be wrongly coupled to its own version (see possible problems listed above).

This would also remedy the lock-step constraints, e.g. we could use v3 of webpack and v2 of core.

Have a "release channel" like dynamic behaviour

Similar to ember-source-channel-url we could pull in the versions to be used for the core packages dynamically. That could be as simple as doing an HTTP request to fetch some JSON file (e.g. static file in a github repo, we only need version specifiers, no hashed builds uploaded to S3 shenanigans like Ember canary).

Besides sharing the same benefits as the hardcoded versions, this would have the added benefit of not having those versions locked by the version of @embroider/test-setup that people have in their lock file (which they might not update often), instead we could define the versions in a central place (and even roll-back if needed). It would also allow to expand test coverage to different versions of Embroider (e.g. v1 vs. v2), but instead of having hard-coded versions in the ember-try config use more "symbolic" references like stable (today: v1) and next (v2), again similar to release/beta/canary for ember-source.

This would change behaviour, as when changing versions this could make CI fail from one day to the other. But that's already the case as mentioned for ember-source, and it would just reflect the truth, rather than having outdated versions pretending everything to be fine...

Implementation-wise we could introduce an (overloaded, no breaking change) argument as in embroider{Safe,Optimized}(channel: 'stable' | 'next'), which when omitted defaults to stable, making it match current behaviour. People wanting to have more test coverage could then add new scenarios like embroiderSafe('next') to their ember-try config, or eventually we come up with an update to the blueprint (tbd).

Things to consider:

• to support offline use, we could use versions specified in the installed @embroider/test-setup as a fallback, to not cause failures when e.g. people are working in the train (especially German trains, we don't have internet outside of cities 🙈)
• in constrained environments (no access to public internet), we might support alternative URLs to pull that from (similar to process.env.EMBER_SOURCE_CHANNEL_URL_HOST)