Consider enabling third party Javascript embeds to safely use the navigation API

[colinclerk]

We run a third-party Javascript widget and would love if the appHistory API could safely be used by third-parties like us.

Today, we cannot safely use the History API on sites we're embedded in. If we do, we're likely to conflict with the developer's router, which wraps the History API and may have strict expectations for the state argument.

One example is Next.js - if a third party uses history.pushState directly, it can lead to this error being thrown:
https://github.com/vercel/next.js/blob/canary/errors/popstate-state-empty.md

Today, instead of using the History API directly, we ask developers to pass us their router's implementation of push and replace, which ensures we do not inject a state argument that breaks their router.

We like having push and replace access for two main reasons:

1. When a user is done interacting with our widget, we navigate to a callback URL passed by the developer. With push, we can navigate to that callback URL without a full page reload.
2. Our widget has many subpages. In addition to passing us push and replace, we ask the developer grant us a wildcard route like /foo/* that always runs our widget's code. We use push and replace to navigate between subpaths of the wildcard route.

Is appHistory open to proposals that would enable these third party use cases?

[domenic]

This is a tricky area, and I'm really glad you opened it for discussion!

We're super-interested in making this new API more usable by multiple scripts on the page. For example, the specific issue you mention around Next.js seem like something we're aiming to solve. In particular, a lot of the reasons that using history.pushState() today breaks routers and frameworks is because window.history doesn't give enough information about current and past history states, so those routers/frameworks need to be the entry point for all possible single-page navs so they can do proper bookkeeping.

Since app history gives much deeper introspection (e.g. looking at past entries; useful and comprehensive navigate and currententrychange events; ...), routers and applications built on top of app history should be much more resilient to third parties calling appHistory.pushNewEntry(), or (if we can pull off the layering correctly) even history.pushState(). So I'm optimistic that third-party widgets should be able to use app history in this way without breaking apps and frameworks.

Another thing worth mentioning is that for your

When a user is done interacting with our widget, we navigate to a callback URL passed by the developer. With push, we can navigate to that callback URL without a full page reload.

it might be possible, in a world where all cool SPAs use appHistory, for you to just do location.href = callbackURL. Then, the cool SPA will just intercept that using the navigate event, and MPAs will do a full navigation. This ties into the general "you might not need pushNewEntry()" message.

---

However, in some other aspects our experience so far points to their needing to be central coordination. In particular, routing and the navigate event. If there are multiple navigate event listeners all trying to intercept navigations, things can get quite tricky. I believe some frameworks solve this via "nested routers" or "per-component routers"; our current thinking is that such complexity is too much to tackle up front, and we'd rather have a framework take control of the navigate event and delegate to third-parties as necessary, at least until app history "v1" is solid and interoperable everywhere. I think this is tied to your case of

Our widget has many subpages. In addition to passing us push and replace, we ask the developer grant us a wildcard route like /foo/* that always runs our widget's code. We use push and replace to navigate between subpaths of the wildcard route.

i.e. our current thinking is that it'll work best if the application or framework adds the navigate event handler, which does something like

// Using the proposal at https://github.com/WICG/urlpattern/blob/master/explainer.md
if (widgetURLPattern.test(e.destinationEntry.url)) {
  widgetLibrary.handle(e);
  return;
}

It seems possible to have multiple non-coordinated navigate event handlers, but pretty fragile; see the note at the bottom of this section.

What do you think of this answer?

[colinclerk]

Thanks for the quick response! I very much appreciate the willingness to explore, and thank you for pointing me to these links.

I want to try pushing back on this point:

In particular, a lot of the reasons that using history.pushState() today breaks routers and frameworks is because window.history doesn't give enough information about current and past history states, so those routers/frameworks need to be the entry point for all possible single-page navs so they can do proper bookkeeping.

I think this overlooks that the History API itself encourages a single bookkeeper, which stems from the combination of:

1. a free-form, developer-defined state object for pushState
2. a global popstate event

If multiple scripts are using pushState and listening to popstate, it's inevitable that each script's popstate listener will receive state objects that it didn't write. If the listener isn't expecting this, it leads to errors like the Next.js one above.

I wonder if we can approach this challenge head-on.

One idea is to decouple appHistory and state. appHistory would remain scoped to the window, but multiples scripts (controllers?) can maintain their own state for each entry. Something like:

• Add appHistory.registerController(), which returns a controller object
• appHistory.currentEntry.state => appHistory.currentEntry.getState(controller)
• appHistory.updateCurrentEntry({ newState }) => appHistory.currentEntry.setState(controller, { newState })
• appHistory.pushNewEntry(options) => appHistory.pushNewEntry(controller, options)
• Add initiatingController to AppHistoryNavigateEvent

This would force developers to acknowledge that there can be multiple controllers, provide a way to determine when another controller triggered the navigate event, and ensure developers only work with states they generated.

Would this help with any multiple script scenarios you have in mind?

[domenic]

The point about the shared nature of the state object is a good one. Currently that is carried over into appHistory, but we've discussed in the past adding more structure (e.g. making it a maplike with get()/set()/has()/keys()). See also #17, which is related (but from a very different angle).

To some extent, the proposal already makes what you suggest possible, by giving each AppHistoryEntry a unique key. (But see #7 for some complications there.) So you can accomplish all of what you suggest just by having your component maintain its state, not in appHistory.currentEntry.state, but in myStateMap.get(appHistory.currentEntry.key).

Making that first-class through an API like your "controllers" seems possible, and perhaps desirable, but my initial instinct is toward more minimalism. Hmm...

[colinclerk]

Can you help me understand how the proposal might help as-is?

I see that we can maintain our own state-map using the entry keys, and that is a welcome addition over our current solution where the only key we have is the URL.

But, I think we'd still want to avoid calling appHistory.pushNewEntry directly, since our parent (e.g. next.js) might be expecting something in the state object, and we wouldn't know how to generate that something.

For what it's worth, I had debated proposing another idea that pushes for more minimalism. In short, strip state out of the API entirely, and force every script to maintain their own set of states mapped to the keys. This would end up encouraging sessionStorage for state management instead of the appHistory api directly.

[domenic]

But, I think we'd still want to avoid calling appHistory.pushNewEntry directly, since our parent (e.g. next.js) might be expecting something in the state object, and we wouldn't know how to generate that something.

Well, Next.js should be able to use the navigate event to notice such changes, and associate state appropriately. See the example in #5 starting "I also suspect that sometimes you want to do something like this, but with state, not URLs".

This does rely on Next.js to do the right thing, but they're pretty incentivized to do so: that way they can catch any navigations, coming from third-parties or framework users, using any API at all.

[tbondwilkinson]

I think this is the problem that "substacks" would solve.

If you could so something like create your very own view on top of the global stack, you could get this basically for free.

I.e. if you could push your own entries with its own state, and calls to this stacks appHistory.entries() returns ONLY your entries, this becomes really trivial to implement.

I think we'll always need a centralized router that has global view of the history stack. But I do think that giving parts of the application a "view" on top of the stack that is scoped, is actually pretty powerful.

An API to actually enable this though is tricky. What does the sub-appHistory API look like? Is it a copy of the existing appHistory API? Or is it subtly different? Does it support the same events like navigate? Can it be a secondary router?

Lots of questions once we go down this route. My preference is probably that we create a sub-stack control that has read and write access, but does NOT have the same router-like controls.

[colinclerk]

@domenic At risk of oversimplifying:

Is it fair to say the appHistory design only allows one script to safely use the built-in state, while other scripts must maintain their own state-map using the appHistory keys?

If so, I think this conflicts with the goal making appHistory more usable by multiple scripts on the page.

[colinclerk]

@tbondwilkinson I'm not sure independent entry stacks works, since only one can be correlated with the back/forward buttons.

Do you think decoupling the state stack from the entry stack might accomplish your goal? There's only one entry stack, but each script can maintain it's own state stack.

I feel like we might all be circling around the same notion here... somehow each script needs to listen for other script's taking action against appHistory (likely the navigate event), and maintaining it's own state stack.

[domenic]

Is it fair to say the appHistory design only allows one script to safely use the built-in state, while other scripts must maintain their own state-map using the appHistory keys?

Yes, I think that's a reasonable summary.

If so, I think this conflicts with the goal making appHistory more usable by multiple scripts on the page.

I don't think so. Multiple scripts can easily maintain their own state map.

Put another way, consider as a baseline no state property on app history at all. This is now friendly to multiple scripts, as they will all maintain their own state map.

Now, we add a state property. All that friendliness remains; it's just that now the main application (not third party scripts) has a slight additional convenience they can use.

[tbondwilkinson]

Back/forward buttons will always correlate with the global history stack. No application logic should ever use the back() forward(), those should really be reserved for the user who has a global view of the page.

What I'm suggesting with sub-stacks is that that stack would be a view OVER the current stack (just as appHistory is a view OVER the global history that includes things like iframe entries) and would filter out any URL parameters/state entries, that did not have to do with your specific component.

[colinclerk]

@tbondwilkinson - Hmmm - are you thinking flip the API on its head a bit?

Instead of the developer filtering navigate events, somehow the browser only sends them the appropriate events? Perhaps some kind of registry...

---

Thought experiment: is there a need or benefit to the first-vs-third or current-vs-sub hierarchy? Perhaps treating each script on an equal level leads to improved composability of frontend scripts.

Per this:

This does rely on Next.js to do the right thing, but they're pretty incentivized to do so: that way they can catch any navigations, coming from third-parties or framework users, using any API at all.

Would a router be more likely to do the right thing if it thinks of itself as one of many scripts instead of the primary script?

---

@domenic I feel okay with the notion that third parties need to maintain their own state map. That's effectively what we're doing today, except the key is location.href and we don't have navigateInfo or access to the full entry stack. So it's a clear improvement already, just wondering aloud if there's something better.

[tbondwilkinson]

Possibly but I think we run the risk of a "who's on first" situation. If you're trying to install multiple routers on a page, who's responsible for behavior like figuring out what to do about queued navigations? There are definitely some decisions that MUST be made by one and only one piece of logic, or at least it's far simpler.

So the question for me is less "how can we get multiple routers to play nicely together" but rather, "how can a component be agnostic of what router the application is using? how can a component get the information it needs by ONLY using the native history methods" and I think we haven't adequately answered that question yet.

One valid solution, you're right, is to say that any component can install a navigate event handler, and we just have to trust that they're using it for just the purposes they need it for. But having to, for instance, ignore a bunch of history state that isn't related to your component is a drag - we've seen that to be the case before when iframes end up in the global history stack and the page has to ignore it.

[tbondwilkinson]

Another way of thinking about this re:routers is that it's okay if each component has its own router choice, but that router shouldn't get to weigh on history changes that aren't related to that component. I don't think router implementors wants to think about whether they're the only router on the page, or one of many.

[colinclerk]

It would be a hugeeeee deviation from what's proposed so far, but the idea of registering to paths maybe isn't totally insane.

There's some prior art in cookies: https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#Path_attribute

[colinclerk]

You can imagine events "bubbling up" through path-registered navigate handlers, sort of like they bubble up through DOM ancestors.

Maybe not totally insane?

[tbondwilkinson]

I struggle with this idea is because I think the API becomes really hungry and requires a lot of changes. I keep hoping for a light-weight version of this idea that doesn't require such radical things as a bubbling navigate event.

Maybe it's sufficient to just register: URL parameters + a state key that gives you a unique subspace. And the sub-appHistory will only notify if those URL parameters and/or state changes, and that will be what you get from entries().

[jakearchibald]

@tbondwilkinson

No application logic should ever use the back() forward(), those should really be reserved for the user who has a global view of the page.

I agree with this. Maybe appHistory shouldn't have those methods? @domenic?

With the current shape of the API, I think I'd make sure my entries had state like { myApp: {…} }, and if a state didn't have a top-level myApp key I'd assume it wasn't state belonging to my part of the app.

BroadcastChannel solves these kinds of problems well by having instances, but that doesn't really work for appHistory since it needs to work across page loads. I guess you could create a sub-stack from a key, but maybe it's best left to web developers to build on top of appHistory.