The menu is omakase

[rubys]

Rails doctrine: The menu is omakase (translation: I'll leave it up to you)

Preface

I'm going to try to make the case that instead of focusing on Module requirements, we should focus on Loader requirements and let a thousand flowers bloom. Part of making my case, I will introduce a number of new use cases that may have here-to-fore seemed out of reach.

Parallel Evolution

Once upon a time, when Rails was young, XML ruled the web, at least for API interfaces. Unsurprisingly, XML support was baked into early versions of Rails.

Times change, and now JSON is in vogue.

In a parallel universe, Node was created. The Web Platform (including JavaScript) at the time didn't have modules, buffers, promises, streams, or even a URL definition. By necessity, Node created each of these.

Rails eventually jettisoned XML support. Oh my god, I can hear you thinking, that will break users. Except it didn't. The code was extracted into the activemodel-serializers-xml gem. Developers that wanted to continue to access this function simply had to install and include the new gem.

Node now has two definitions of URLs, one (non-standard) definition of streams, two definitions of the file system interface (one with, and one without promises), and is headed to having two module definitions. And now it looks like we will be heading towards having both process.nextTick() and queueMicrotask.

That's suboptimal, and that's putting it midly.

Brittleness

The current state of Node is that very little can be changed as any change that might break somebody undoubtedly will break a lot of people. This is particularly true for the module subsystem. Consider the following return statement. It is not legal JavaScript, but it clearly works.

This makes implementing the modules features list somewhat impossible. It is impossible to be 100% backwards compatible with CommonJS and 100% compatible with browsers. At least not simultaneously.

So, let's make that a non-goal.

Chroot jails

If you contemplate what import or require do, they do for JavaScript what the file system does for POSIX applications. Many years ago, people realized that one could modify the environment in such a way that subprocesses could operate from a different root directory.

And thus, chroot was born. This begat containers and docker and kubernetes and the cloud.

Imagine an architectural boundary (say, perhaps Workers), across which different code could use a different loader. Different loaders could expose different globals, different modules, and even different semantics entirely for import statements.

This could make it easier to mock and polyfill - without changing code.

And if we create the ability for modules to hide and even proxy other modules, we can enable the creation of secure sandboxes - all without affecting the performce of the underlying modules for people that don't need this level of isolation.

If we would ever want to collapse once again down to a single implementation of fs (the Promisfied version, naturally), and still allow people to opt-in either temporarily or permanently to the classic fs implementation, then the Node core would need to be use a different loader than code in user space.

Issues

"Out of scope" and "technically challenging" would be valid criticisms of this proposal. To which I observe, while definitely hard, implementing this proposal is easier than the impossible task of merging of incompatible module semantics with the added constraint of not breaking anyone.

Imagine a loader that is bug-for-bug compatible with CommonJS (possibly with babelify thrown in). And another loader that is meticulously spec compliant. And one or more that quixotically attempt to find the sweet spot between the two.

Those aren't the problem that this proposal intends to solve. Instead it proposes to focus on the APIs needed to enable the creation of these loaders. And the user interface by which one specifies the initial loader - I'm of the opinion that command line arguments alone aren't going to cut it.

Upsides

So far, I've focused on issues and complexities. That's not the right place to start. The right place to start is with the type of future we want to enable. Specifically:

• A future where there is only one core definition of URL. One core definition of fs.
  • A corollary to this is a future where we can reasonably discuss retiring APIs that no longer are relevant or have been effectively obsoleted by another API.
• A future where node and the web platform continue to evolve separately but together, as opposed to continuing to diverge over time.
• A future where developers can opt-in early to new node core features without upgrading node itself, or opt to continue to use specific features after an upgrade, either temporarily or permanently.

Prerequisites

• An obvious prerequisite for this is a pluggable loader.
• Not as obvious, but I strongly believe that this will require the ability to define a persistent configuration, i.e., more than command line options and NODE_OPTIONS provides. At the same time, I would suggest that we co-opt and take ownership of NODE_ENV, enabling separate configurations by environment. Note that configurations in Rails are emitted by generators, and consist of running code. Normally nobody needs to touch these files except for two cases:
  • They have a specific problem that need to address. The generated code has comments that help people find the configuration option and make the correct choices.
  • When upgrading to a new release of Rails.
• Part of this persistent configuration will be the ability to do the equivalent of the current -r or --require for modules that only consist of side effects (doing things such as extending existing classes or modifying globals). This will make it easier for applications to do such things as opting-in to queueMicrotask() with Node version 8, and opting-in to continue to have access to process.nextTick() with Node version 14.

[ljharb]

Rails doesn't actually leave things up to users tho - it makes strong, opinionated default choices like haml, coffeescript, sprockets, webpack, yarn, etc, that have huge consequences on its ecosystem. That everything is configurable is great - our module loaders should be as well - but to underestimate the consequences of node's choices of defaults is to take no lessons from rails, among others.

Backwards compat is inextricably a goal for JS, and for the web, and imo also for node. Making that a non-goal is risking destroying node and its ecosystem. Things that can't be easily migrated to, won't be.

In general, I like your upsides/"future goals" section - but "modules" aren't a feature in the same sense as, say, "bigints" or "promises" - they're the glue that ties everything together (just like CJS already is), and are not something that should be casually opted in to.

[rubys]

Rails doesn't actually leave things up to users tho - it makes strong, opinionated default choices like haml

Can I stop you right there? HAML is certainly a popular choice, but despite rumors to the contrary, not the default choice.

Things that can't be easily migrated to, won't be.

While I certainly agree strongly with this statement, I disagree with your apparent conclusion that everything that ever was once a part of core needs to forever be a part of core. Can we discuss why we need to support two definitions for URLs indefinitely?

but "modules" aren't a feature in the same sense as, say, "bigints" or "promises"

I may have inadvertently used the term modules at two different meta-levels. Sorry for the confusion. I am suggesting that features such a "bigints" or "promises" could be introduced first as modules (or to use the web term: polyfills).

Separately, much of what I describe builds on a separate feature: pluggable loader. The merits of that requirement can be discussed in its separate issue.

[devsnek]

we've discussed making the loader a module before... its not really something that is possible from a technical standpoint.

[ljharb]

not the default choice.

I stand corrected, but you can put whatever things in that list you want, and it's still forcing a choice on the majority of users - who won't configure things for themselves. The defaults matter, a lot.

Can we discuss why we need to support two definitions for URLs indefinitely?

can you clarify about "definitions for URLs"? Certainly URL should be preferred; but file paths aren't URLs (despite attempts to make it one with file://) and shouldn't be a URL instance imo.

I am suggesting that features…

I definitely agree that it would be great if all features (aside from ESM) added to node were first available as properly versioned packages on npm, and then brought into core when they're ready.

I don't think there's any controversy or pushback against making loaders fully configurable, at least at an app level. The majority of the debate has been about the defaults, from my perspective.

[mcollina]

@rubys this is extremely well written, and as a former Rails users I can definitely agree with the sentiment. I think the plan forward could be:

1. ship the minimal kernel
2. ship the pluggable loader
3. remove the flag
4. based on popularity, define the defaults, even if they are not spec compliant.

---

On a more philosophical level, I would love a future where node and the web platform continue to evolve separately but together, as opposed to continuing to diverge over time. I do not know if it is achievable, as a huge part of the Web Platform foundation was built without taking Node.js needs into account. For that dream to become true, it should not be Node that adopt all what the Web Platform standards dictates but rather we should sit at the standards table and explain why we cannot do certain things, and look for a compromise.

[devsnek]

@mcollina

I would love a future where node and the web platform continue to evolve separately but together, as opposed to continuing to diverge over time.

https://github.com/nodejs/open-standards 😄

[rubys]

the majority of users - who won't configure things for themselves

In a language as dynamic as JavaScript or Ruby, reconfiguring is as simple as a require or import. I will suggest that a majority of users install and use packages. Let's try to leverage that.

Continuing with the haml/rails example: configuring Rails to use HAML is as simple as adding the Haml-rails gem to your Gemfile.

can you clarify about "definitions for URLs"

See https://nodejs.org/docs/latest/api/url.html. Node ships both the WHATWG URL API and a Legacy URL API. I'd like to see only one shipped and one be extracted to a module. I wouldn't be surprised if the legacy url API is more popular at the moment, but would suggest that it be the one to be extracted out to a module.

Imagine a future where require('url') produced a helpful error message. And npm install legacy-url made require('url') work again.

The majority of the debate has been about the defaults, from my perspective.

I don't debate that defaults matter. I just think things would be better if perhaps a small percentage of the energy that goes into those debates were redirected towards making node easier to configure and head in the direction of embracing customization then we can afford to worry less about the defaults in the future.

[SMotaal]

I wouldn't be surprised if the legacy url API is more popular at the moment, but would suggest that it be the one to be extracted out to a module.

I think @rubys raises a very very strong point, which I have not been able to put in words, let alone express since I joined.

Apologies in advance if my take goes in a different (or just a more extreme) vector.

I think that the idea that "legacy consumer" is constant unchanging and thus breakable by any and all divergence is a fear that sometimes limits our ability to consider the finer details of legacy code:

1. Continues to use CJS — if unchanged

• Is require going to change: no
• Is import going to be used: only with tooling
• Will ESM affect how how existing "sic" CommonJS code executes: no
• Does it make sense switch out tooling (like older babel-node) used for legacy code in favour of an new ESM loader alternative (like a --loader) ... etc: no
  • Does it make sense for newly implemented consumers to use a --loader to work with such unchanged packages: yes

2. Moves to ESM (and refactored CJS) modules — if changed

• Is require going to change: no
• Is import going to be used: probably
• Is a human being doing this migration: if it matters, they will be there
• Can there be complicated decisions involved: hopefully very few
• Can ESM provide 1:1 parity with CJS features: not currently
  • Do some modules need to be CJS: very possible
  • Is there a human being handling this migration: if it matters, they will be there

3. It is a dependency for a mixed bag of *1'*s and *2'*s — changed or not

• So, if it works, it still works. If it needs to change, it must change. If it cannot be reasonably changed without humans, it needs humans, and if they don't show up, then…
  • maybe it is just outdated
  • or it is no longer a priority
  • or needed but substitutable (with up-to-date alternatives)
  • ultimately it should be deprecated now, then removed in due

---

On planet earth, we can't afford to be uncomfortable discussing things like that

[SMotaal]

we've discussed making the loader a module before... its not really something that is possible from a technical standpoint.

@devsnek I can make a few cases for the opposite, but obviously I am talking not about ModuleWrap aspects here (that's all you).

To me, a modular (pluggable) loader is one that has a few fixed "template methods" that mutate a few fixed "structures" all of which expect a known start and end condition, but allow modularity to determine what happens in between.

A bad module loader design is one that very loosely allows plugin implementations to be:

• inefficient — unchecked redundancy by not offering the right set of optimized helpers
• unstable — lacking recoverability from both predictable and unpredictable fault
• ineffective — limited usability (no clean API) and traceability (ie non-contextual overload)
• unpredictable — without well defined (and testable) abstract operations
  • nondeclarative — needlessly defaults to programmatic configuration

A good one is one that knows that all those aspects have trade-offs, and as such predetermines it's priorities and consistently uses them to make future decisions, as more and more plugin abstractions are explored, experimentally implemented, refined (or redefined), and finally implemented.

This takes a lot of different mindsets — naturally improving the odds with a few "specialized" teams adhering to such roadmaps.

---

My two cents on the technical feasibility in node (I dare, and feel free to laugh with me)...

Coming from the (amazing) ModuleWrap implementation, I am confident enough to assume it is impossible to do.

However, working with V8::Module and the easy-to-uncouple aspects of ModuleWrap wrap is a starting point to creating a loader in a module. From there, and borrowing endlessly from the invaluable ModuleWrap lessons and strategies IMHO is a reasonably suitable path for designing a modular loader with the predetermined fact that it is a (node) module in itself.

[bmeck]

@SMotaal

I'm unclear on what "plugin" means here, but I assume it is just a userland loader.

All of the things you mentioned are not things that can be stopped purely by design. A userland loader can without regard to any designs by Node.js achieve these problems and we shouldn't point to Node.js' implementation as bad by allowing and trusting that users take some level of care while writing behaviors.

• inefficient - a userland loader using bubble sort instead of a better sort
• unstable - a userland loader having bugs
• ineffective - a userland loader having complex configuration (e.g. older webpack)
• unpredictable - a userland loader does something surprising

I'm not sure what these points are trying to state because if users are allowed to code their own loaders, they can create these for themselves.

A good one is one that knows that all those aspects have trade-offs, and as such predetermines it's priorities and consistently uses them to make future decisions, as more and more plugin abstractions are explored, experimentally implemented, refined (or redefined), and finally implemented.

I'm also confused by "abstraction" here, I think you mean hooks or interceptors. We should try to keep those to an absolute minimal amount that we can and push as much as we can reasonably do so to users in the future so that we can see how it is used. Often times we will not be able to exactly adopt how userland does things, but we can see patterns and tradeoffs that the community enjoys and see what changes can be done to get closer.

[SMotaal]

Strictly speaking about "plugins"… let's use the more abstract term "extension" to avoid coloring it with some of the baggage associated with the mainstream "…-plugin-…" trend.

So, "extension" is still a plugin, but without the presumption of these kinds of controversial mechanisms, ie not dealing with format or delivery aspects (for now).

A userland loader can without regard to any designs by Node.js achieve these problems and we shouldn't point to Node.js' implementation as bad by allowing and trusting that users take some level of care while writing behaviors.

@bmeck Absolutely, and I don't claim there is a way to prevent that. Instead, one could argue that some efforts can encourage good practices, while some (including doing nothing at all) can encourage the opposite.

So the idea of "by example" comes to mind, where an extension that does the exact same abstract operation as another, which is provided as a good example, that extension will likely the follow same template, especially if comments are used to communicate the rationale. At the same time, if any example (generally speaking) includes bad shortcuts for the sake of brevity, without explicitly stating and even maybe hinting towards the longer and more appropriate way to get it done, that example will only encourage disappointing outcomes.

Considering the set of features, we can assume that if we had a extensible loader, our team will likely include a few baseline extensions. Those extensions will allow us to consider and refine all those aspects I pointed out. They will also be really good examples that are likely to lead to good userland loaders, be it because they can switch out the example parser with their own, or because they were able to "remix" ideas from multiple extensions to accomplish more comprehensive feats, they will likely have better outcomes.

Of course this is only one way to look at the challenge of would encouraging all around more pleasing outcomes.

[bmeck]

So, "extension" is still a plugin, but without the presumption of these kinds of controversial mechanisms, ie not dealing with format or delivery aspects (for now).

I don't understand this. What is an "extension" if it isn't associated with any mechanisms?

[SMotaal]

I'm also confused by "abstraction" here, I think you mean hooks or interceptors. We should try to keep those to an absolute minimal amount that we can and push as much as we can reasonably do so to users in the future so that we can see how it is used. Often times we will not be able to exactly adopt how userland does things, but we can see patterns and tradeoffs that the community enjoys and see what changes can be done to get closer.

Funny you mention this, because on more occasions than I care to count, people involved with various popular tools told me they "are holding off now to avoid creating too much noise and will follow Node's lead," or something to that effect. First, there is the obvious mutual sense of responsibility towards the larger ecosystem. More importantly, those statements also come with an implied confidence in the fact that with the amount of talent and skill coming together into the group's effort, it very likely that they would be able to take a bigger bite of the technical challenges and deliver a well rounded (and innovative) solution to many of the old (and new) challenges of modules. At the very least, if the sentiment is not that positive, there is confidence that if optimal solutions can be delivered, ones that do so with the benefit of internal optimizations of the runtime have the upper hand.

[bmeck]

Funny you mention this, because on more occasions than I care to count, people involved with various popular tools told me they "are holding off now to avoid creating too much noise and will follow Node's lead," or something to that effect. First, there is the obvious mutual sense of responsibility towards the larger ecosystem. More importantly, those statements also come with an implied confidence in the fact that with the amount of talent and skill coming together into the group's effort, it very likely that they would be able to take a bigger bite of the technical challenges and deliver a well rounded (and innovative) solution to many of the old (and new) challenges of modules. At the very least, if the sentiment is not that positive, there is confidence that if optimal solutions can be delivered, ones that do so with the benefit of internal optimizations of the runtime have the upper hand.

I'm also confused here. How does this tie back to wanting to keep these to a minimum? Having innovative solutions is not something that is good or bad by its own nature. Having an increased volume and scope in which we alter how the platform behaves is my concern since that makes solutions harder to ensure they can retain the internal optimizations a runtime could provide, and in fact having conflicting configurations can create issues.

[SMotaal]

I don't understand this. What is an "extension" if it isn't associated with any mechanisms?

I am just trying to avoid talking about them as plugins in the same sense as say babel plugins... etc. In other words, to not make any claims regarding the actual pluggable design. (a separate topic)

[bmeck]

I'm getting more confused on the topics, what are we talking about? Prior to your usage of the term "plugin" or "extension" we have used the term "loader" to describe when users take control of aspects of loading resources.

[SMotaal]

Let me think on how to better approach this, reflect a little on where the disconnect is happening. (also need to walk the dogs)...

But instead of considering increased volume and scope... etc., I will leave you with a different question.

Without the benefit of some internal optimizations that userland loaders could align with for better performance, how possible is it from your perspective that someone can take one of the popular CommonJS loaders and implement it's new ESM counterpart ensuring it is at least as fast if not 10x faster than the original?

[bmeck]

Without the benefit of some internal optimizations that userland loaders could align with for better performance, how possible is it from your perspective that one can take one of the popular CommonJS loaders and implement it's new ESM counterpart ensuring it is at least as fast if not 10x faster than the original?

I don't think speed should be our focus right now. Make it work, then make it fast. Part of making things work, is making as little impact and overlap as possible to leave space to optimize.

I don't think speed is going to be 10x faster or even 2x faster than CJS for any form of ESM for a long time due to how hard it is to optimize and that VMs themselves haven't fully optimized ESM implementations. We have spent 10 years figuring out CJS, and we are struggling just to ship a first implementation of ESM. Also, focusing on how much we can optimize things is hard without benchmarking which we can't really do except in theory for now. I know that adding threads actually slows down Loaders while you must wait for threads to spin up. I think we could improve that speed, but certainly it will actually be slower than it is today if we take that approach. However, by forcing the isolation to begin with, we can move things back onto the main thread but it would be harder to go the other way around. When we design things, reserving design space that allows for more potential optimizations is more important to me than seeing a benchmark on any given initial unoptimized implementation.

[SMotaal]

I don't think speed should be our focus right now. Make it work, then make it fast. Part of making things work, is making as little impact and overlap as possible to leave space to optimize.

Great point... I agree... So maybe it helps to not assume that agreement is a given. What I am trying to say is that obviously we have points where we agree, but without queuing those into our discussion, some (I for one) can just generalize the sense of disagreement to the expressed opinions in general (which is obviously not the case).

Do we agree that ModuleWrap already introduced a number of internal optimizations over V8::Module?

It is that level of internal optimization that should either be planned for or even implemented to some extent so that Node's own resolution (not userland) can be fast enough than if it were all done by the ESM loader (js side).

So internal optimizations are not off the table, but one can say we have a very conservative attitude towards them, erring on the side of avoiding by reserving design space that allows for more potential optimizations once ESM stabilizes.

[SMotaal]

At the end of the day... honestly, I am on your side 😉

[rubys]

@jasnell outline for moving legacy URL out of core (over time):

• Copy existing "core" url module out to npm. This does not include WHATWG URL.
• Rename core url mode to url-deprecated or some such. This internal name does not need to be widely advertised.
• Change the cjs loader to recover from "module not found" for known deprecated core modules (initially only url) by issuing a deprecation warning and substitute the internal deprecated version.

This is an improved user experience over deprecating url and removing it a release or three later. One will be able to continue to use the legacy URL (potentially indefinitely) by merely installing a module. The hope is by starting with something like url we can gain experience with this process, and hopefully begin to train the user community as to how major releases of node will evolve over time.

[mcollina]

Definitely +1 to @jasnell proposal.

[bmeck]

Given the lack of discussion and the more structured phases process we are going through, I'm closing this. If you wish to reopen we can move the discussion to the proper phase issues related to topics of how to configure things.