Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core refactor part 2 #1057

Merged
merged 25 commits into from
May 14, 2021
Merged

Core refactor part 2 #1057

merged 25 commits into from
May 14, 2021

Conversation

jvarho
Copy link
Collaborator

@jvarho jvarho commented May 9, 2021

Moves most of the request routing over to core.

I had to make test changes as well due to:

  1. There were simply tests where the test data was wrong, but the old logic worked differently and happened to work.

  2. The 404 logic was tested two ways (render and serve) with the same manifests. I dropped the non-static 404 tests. I hope that's ok, AFAIU Next.js will always generate a static 404 since at least 9.5. (Note: static 500 logic should be implemented at some point.)

  3. Rewrites had "no-op rewrite" support (Dynamic rewrites 404ing #733) which I was not able to reproduce with Next.js server and dropped. Those add complexity, but if we want to diverge from Next.js I can try to reimplement them. However, with the new types of rewrites added in 10.2 that may not be possible consistently.

  4. Rewriting query parameters worked inconsistently when locale support was/not used.

I did not yet fix the routing priority issues I found, but they are flagged in a couple of comments. This branch reproduces the same (buggy) logic of going through routes by type rather than specificity. Before fixing that the build logic should be refactored, as doing the correct thing efficiently will require manifest changes.

Next steps:

  • Refactor build logic to core.
  • Refactor custom headers to core.
  • Refactor response handler routing.
  • Refactor lambda and s3 logic to separate libraries (for sharing between apigw/cf).
  • Refactor image handler to separate(?) library.
  • Implement apigw/lambda backend.

Unrelated TODO:

  • Core library could use more direct tests.
  • 10.2 rewrites should now be easier to implement.
  • Locale and basepath logic could do with a once-over. It works, but there's probably room for simplification.
  • I noticed redirects have no cache-control, 308 should probably be cached by cloudfront.

@codecov
Copy link

codecov bot commented May 9, 2021

Codecov Report

Merging #1057 (4b12434) into master (e61dc3c) will decrease coverage by 0.09%.
The diff coverage is 99.59%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1057      +/-   ##
==========================================
- Coverage   83.19%   83.10%   -0.10%     
==========================================
  Files          74       79       +5     
  Lines        3053     3012      -41     
  Branches      867      839      -28     
==========================================
- Hits         2540     2503      -37     
+ Misses        460      456       -4     
  Partials       53       53              
Impacted Files Coverage Δ
packages/libs/core/src/preview.ts 100.00% <ø> (ø)
...es/libs/lambda-at-edge/src/routing/locale-utils.ts 85.00% <ø> (-3.24%) ⬇️
...ackages/libs/lambda-at-edge/src/routing/matcher.ts 100.00% <ø> (ø)
packages/libs/core/src/data.ts 96.66% <96.66%> (ø)
packages/libs/core/src/api.ts 100.00% <100.00%> (ø)
packages/libs/core/src/basepath.ts 100.00% <100.00%> (ø)
packages/libs/core/src/index.ts 100.00% <100.00%> (ø)
packages/libs/core/src/locale.ts 100.00% <100.00%> (ø)
packages/libs/core/src/match.ts 100.00% <100.00%> (ø)
packages/libs/core/src/page.ts 100.00% <100.00%> (ø)
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 033488e...4b12434. Read the comment docs.

@jvarho
Copy link
Collaborator Author

jvarho commented May 10, 2021

After these latest commits all the tests pass and lib/core has >99% test coverage. Coverage only falls because this removes lines overall. Some e2e tests fail to build, but I can't see how it's because of any changes I've made when the other jobs go fine.

I will continue with the next steps, but this can be merged unless one of the changes mentioned above is a problem.

@dphang
Copy link
Collaborator

dphang commented May 11, 2021

After these latest commits all the tests pass and lib/core has >99% test coverage. Coverage only falls because this removes lines overall. Some e2e tests fail to build, but I can't see how it's because of any changes I've made when the other jobs go fine.

I will continue with the next steps, but this can be merged unless one of the changes mentioned above is a problem.

The yarn build step failed here but the workflow still continued, not sure why. The actual problem looks to be with the yarn cache:

error https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.6.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, stat '/home/runner/.cache/yarn/v6/npm-graceful-fs-4.2.6-ff040b2b0853b23c3d31027523706f1885d76bee-integrity/node_modules/graceful-fs/clone.js'"
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "@aws-sdk/client-s3 > @aws-sdk/middleware-retry > [email protected]" has unmet peer dependency "react-native@>=0.56".
warning "@vercel/nft > [email protected]" has incorrect peer dependency "acorn@^6.0.1".
warning " > [email protected]" has unmet peer dependency "builtin-modules@^3.1.0".
[4/4] Building fresh packages...
lerna ERR! yarn install --frozen-lockfile exited 1 in '@sls-next/serverless-cli'
lerna ERR! yarn install --frozen-lockfile exited 1 in '@sls-next/serverless-cli'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Usually rerun will fix it but GH Actions doesn't support just re-running a single job right now =(

@jvarho jvarho force-pushed the core-refactor-part-2 branch 2 times, most recently from c40d5a0 to 1cbcf6c Compare May 11, 2021 04:19
@jvarho
Copy link
Collaborator Author

jvarho commented May 11, 2021

Now the tests got far enough to show an actual bug somewhere: https://github.com/serverless-nextjs/serverless-next.js/runs/2555973177?check_suite_focus=true#step:9:1104

Looks like perhaps vercel/next.js#21943 which a nextjs update in the relevant e2e tests would fix. I should have time to look into it tomorrow.

@jvarho
Copy link
Collaborator Author

jvarho commented May 11, 2021

Had time sooner after all. My interpretation, based on testing local deploy with a couple of versions and logging statements:

  • In current master these tests pass more or less by coincidence.
  • With my changes the correct next.js exported page gets called with the correct data in req/res but it buggily returns a 307.
  • Upgrading to 10.0.9 fixes the issue, as does 10.2 or the latest 10.1.

For testing purposes I decided to upgrade one of the two locale tests to 10.0.9 and one to 10.2.0, so those should now pass. Might be better to decide which versions are supported, spell it out in README and upgrade all the e2e tests to the latest versions of those major.minor releases?

@dphang
Copy link
Collaborator

dphang commented May 11, 2021

Had time sooner after all. My interpretation, based on testing local deploy with a couple of versions and logging statements:

  • In current master these tests pass more or less by coincidence.
  • With my changes the correct next.js exported page gets called with the correct data in req/res but it buggily returns a 307.
  • Upgrading to 10.0.9 fixes the issue, as does 10.2 or the latest 10.1.

For testing purposes I decided to upgrade one of the two locale tests to 10.0.9 and one to 10.2.0, so those should now pass. Might be better to decide which versions are supported, spell it out in README and upgrade all the e2e tests to the latest versions of those major.minor releases?

Previously I was trying to support the latest two minor versions, so when a new version of Next.js is released, I manually copied the current e2e tests over with prev- prefixed to them and then bumped the current versions up to the latest. So that way we could at least have a bit of backwards compatibility (some people). But I guess it makes it harder to maintain as well...

@jvarho
Copy link
Collaborator Author

jvarho commented May 11, 2021

Previously I was trying to support the latest two minor versions, so when a new version of Next.js is released, I manually copied the current e2e tests over with prev- prefixed to them and then bumped the current versions up to the latest. So that way we could at least have a bit of backwards compatibility (some people). But I guess it makes it harder to maintain as well...

That makes a lot of sense. I don't think two versions should be hard to support – except for the latest features of course. But I was unsure what the target was – the readme starts off by talking about version 9.0...

@dphang
Copy link
Collaborator

dphang commented May 11, 2021 via email

@jvarho
Copy link
Collaborator Author

jvarho commented May 11, 2021

Eh, back to the drawing board I suppose. The tests I looked at in detail were fixed by the version upgrades, but others are still failing.

@jvarho
Copy link
Collaborator Author

jvarho commented May 12, 2021

Ok, so upgrading the next.js versions fixed one issue, but uncovered another. There is a front end change that drops the default locale under some situations. I've reported the inconsistent behavior as a bug in next.js, but there are many locale related open issues, so I guess that functionality is still a bit in flux. I modified the tests to pass with the current behavior, not sure what the best solution is.

An api rewrite test fail was an actual bug in my code, which I fixed. So even though the e2e tests are annoying to get to run, they are useful.

@dphang
Copy link
Collaborator

dphang commented May 13, 2021

Ok, so upgrading the next.js versions fixed one issue, but uncovered another. There is a front end change that drops the default locale under some situations. I've reported the inconsistent behavior as a bug in next.js, but there are many locale related open issues, so I guess that functionality is still a bit in flux. I modified the tests to pass with the current behavior, not sure what the best solution is.

An api rewrite test fail was an actual bug in my code, which I fixed. So even though the e2e tests are annoying to get to run, they are useful.

Thanks for fixing the tests. Yeah, I don't personally use the locales feature too much, so not sure what is the best either. If you can make it work with the latest version, that is probably the best compromise for now. If users have issues with the latest of this component, they should probably try to upgrade Next.js first.

@dphang dphang merged commit dd6576e into master May 14, 2021
@delete-merged-branch delete-merged-branch bot deleted the core-refactor-part-2 branch May 14, 2021 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants