Adds test that validates that older supergraphs can be read by new ve… #2297
Conversation
…rsions Because newer versions can't generate old subgraphs, this commit creates a new scripts that allow to easily generate a supergraph for the "current" version. The intent is that with each new version, we'll generate the corresponding supergraph and commit it to source control. As such, any following version will have older versions of the supergraph which it can test the reading of. This is imperfect as this only test a relatively specific supergraph, and making change to it would be painful since we've have to backport the change to older versions and re-generate new versions of the supergraph (possible, but painful). But it is better than nothing as it will quickly catch any obvious breaking in the reading of older supergraphs.
👷 Deploy request for apollo-federation-docs pending review.Visit the deploys page to approve it
|
|
This pull request is automatically built and testable in CodeSandbox. To see build info of the built libraries, click here or the icon next to each commit SHA. |
|
Going to mark this draft to convey the fact that I'm not entirely convinced this is a good approach. But essentially, the problem we have is that newer versions of the query planner should be able to read and use older supergraphs, but we don't really have any test for that. Particularly as we're about to bump the join spec version to 0.3 in the next release, having tests that ensure we don't completely break when reading 0.2 versions would be nice. But setting up that kind of tests is a bit painful as we need to keep around a bunch of older supergraphs that newer versions can test against and:
In any case, this PR is some attempt at having "something". It tries to tackle the problem of the 1st bullet point above: it makes it easy to generate a test supergraph from the current version and to keep it around. In theory, if we don't forget to generate a new test supergraph on new versions (this part is still manual), we'd accumulate older supergraphs that never versions can test against (which is why this patch is against Overall, I'm on the fence on whether it's worth having something like this in-tree. Maybe this kind of backward compat is better left entirely to our offline harness. |
| ` | ||
| } | ||
|
|
||
| const reviews = { |
There was a problem hiding this comment.
I would separate the generation code and the tests into separate files, but it's up to you.
There was a problem hiding this comment.
Oh I see you address it in the comment below. Again, it's fine if you don't think it's worth the hassle.
|
|
||
| // Ensure we're not given a completely broken argument. This also reject any `-alpha` or `-rc` versions on purpose: no | ||
| // point in keeping test supergraphs for those versions around. | ||
| if (!(/[0-9]+\.[0-9]+(\.[0-9]+)?/.test(version))) { |
There was a problem hiding this comment.
We don't actually check that the file we are trying to generate is the branch that we're currently on. Probably nbd, but I thought I'd call it out as a potential footgun.
There was a problem hiding this comment.
Yeah; I wasn't actually sure how to do something useful here without being complicated. In theory, I wanted to use the current branch version instead of asking for a version manually, but as far as i can tell, we only bump versions just before releases, so the version of the "current branch" is usually the version of "whatever was the last release", and so it's not exactly the version we'd want for this.
I figured passing the version manually might end up less frustrating that something that tries to be automatic but does the wrong thing, but I'm open to suggestions (maybe someday we get this integrated to the release process...). Anyway, I'll note that we still need to manually commit the result of this script, which give more change to double check for mistakes.
…rsions
Because newer versions can't generate old subgraphs, this commit creates a new scripts that allow to easily generate a supergraph for the "current" version. The intent is that with each new version, we'll generate the corresponding supergraph and commit it to source control. As such, any following version will have older versions of the supergraph which it can test the reading of.
This is imperfect as this only test a relatively specific supergraph, and making change to it would be painful since we've have to backport the change to older versions and re-generate new versions of the supergraph (possible, but painful). But it is better than nothing as it will quickly catch any obvious breaking in the reading of older supergraphs.