Implement new client-side router

[timneutkens]

Client-side router for app directory

This PR implements the new router that leverages React 18 concurrent features like Suspense and startTransition.
It also integrates with React Server Components and builds on top of it to allow server-centric routing that only renders the part of the page that has to change.

It's one of the pieces of the implementation for https://nextjs.org/blog/layouts-rfc.

Details

I'm going to document the differences with the current router here (will be reworked for the upgrade guide)

Client-side cache

In the current router we have an in-memory cache for getStaticProps data so that if you prefetch and then navigate to a route that has been prefetched it'll be near-instant. For getServerSideProps the behavior is different, any navigation to a page with getServerSideProps fetches the data again.

In the new model the cache is a fundamental piece, it's more granular than at the page level and is set up to ensure consistency across concurrent renders. It can also be invalidated at any level.

Push/Replace (also applies to next/link)

The new router still has a router.push / router.replace method.

There are a few differences in how it works though:

• It only takes href as an argument, historically you had to provide href (the page path) and as (the actual url path) to do dynamic routing. In later versions of Next.js this is no longer required and in the majority of cases as was no longer needed. In the new router there's no way to reason about href vs as because there is no notion of "pages" in the browser.
• Both methods now use startTransition, you can wrap these in your own startTransition to get isPending
• The push/replace support concurrent rendering. When a render is bailed by clicking a different link to navigate to a completely different page that still works and doesn't cause race conditions.
• Support for optimistic loading states when navigating

Hard/Soft push/replace

Because of the client-side cache being reworked this now allows us to cover two cases: hard push and soft push.

The main difference between the two is if the cache is reused while navigating. The default for next/link is a hard push which means that the part of the cache affected by the navigation will be invalidated, e.g. if you already navigated to /dashboard and you router.push('/dashboard') again it'll get the latest version. This is similar to the existing getServerSideProps handling.

In case of a soft push (API to be defined but for testing added router.softPush('/')) it'll reuse the existing cache and not invalidate parts that are already filled in. In practice this means it's more like the getStaticProps client-side navigation because it does not fetch on navigation except if a part of the page is missing.

Back/Forward navigation

Back and Forward navigation (popstate) are always handled as a soft navigation, meaning that the cache is reused, this ensures back/forward navigation is near-instant when it's in the client-side cache. This will also allow back/forward navigation to be a high priority update instead of a transition as it is based on user interaction. Note: in this PR it still uses startTransition as there's no way to handle the high priority update suspending which happens in case of missing data in the cache. We're working with the React team on a solution for this particular case.

Layouts

Note: this section assumes you've read The layouts RFC and React Server Components RFC

React Server Components rendering leverages the Flight streaming mechanism in React 18, this allows sending a serializable representation of the rendered React tree on the server to the browser, the client-side React can use this serialized representation to render components client-side without the JavaScript being sent to the browser. This is one of the building blocks of Server Components. This allows a bunch of interesting features but for now I'll keep it to how it affects layouts.

When you have a app/dashboard/layout.js and app/dashboard/page.js the page will render as children of the layout, when you add another page like app/dashboard/integrations/page.js that page falls under the dashboard layout as well. When client-side navigating the new router automatically figures out if the page you're navigating to can be a smaller render than the whole page, in this case app/dashboard/page.js and app/dashboard/integrations/page.js share the app/dashboard/layout.js so instead of rendering the whole page we render below the layout component, this means the layout itself does not get re-rendered, the layout's getServerSideProps would not be called, and the Flight response would only hold the result of app/dashboard/integrations/page.js, effectively giving you the smallest patch for the UI.

---

Note: the commits in this PR were mostly work in progress to ensure it wasn't lost along the way. The implementation was reworked a bunch of times to where it is now.

[mmmeff]

Awesome work guys, thanks!