Browser Build/Bundle API

[mattdesl]

Hey Evan, amazing work with esbuild so far.

I noticed the build/bundle API is not supported in the browser via esbuild-wasm, however now with plugins it seems like it could be something that devs could work around. For example, import foo from 'some-module', if it's not marked external, would expect plugins to resolve and load some-module, if that fails, then it can emit an error. Same for relative paths, which could be stored in memory rather than on disk.

In many cases a plugin would just need to replace 'some-module' with an unpkg prefixed URL and the build will still work.

These changes should allow tools like the Svelte REPL to be built on top of esbuild, which can support "files" (in-memory), URL imports, and bare imports (resolved via unpkg or similar).

https://svelte.dev/repl

[evanw]

That's a good idea. It should be possible to do this now that plugins exist. I'll work on this at some point.

[ggoodman]

Hi @evanw, I've begun hacking on an idea and find myself wishing this sort of thing were possible both in the browser environment and other environments like a CloudFlare worker.

Even in Node, I find myself wishing that the plugin api's reach was a little bit further. Specifically, I would like to run esbuild in an environment where plugins are the only way to interact with the file system.

I would like to leverage esbuild to act as the engine powering a service that receives build requests over some arbitrary transport and produces builds on the fly. Using esbuild in its service mode, I would also use the logic from the velcro resolver to have a persistent builder service and a persistent resolver / loader service, respectively.

I've found myself coming up against a few rough edges, notably:

• References in entrypoints are not resolved via resolve plugins, meaning I have to reach for the stdin input.

• Velcro works in terms of urls, so I create an in-memory file-system rooted at file:/ and an entrypoint might be file:/index.tsx. My stdin config looks like:

  {
      contents: 'import * as React from 'react'\n...',
      loader: 'tsx',
      sourcefile: 'memory:/index.tsx',
  }

• Despite configuring a sourcefile on stdin, the args.importer in the onResolve hook is <stdin>. Easy enough to work around but could possibly grab the value explicitly passed in for less surprise.

• In the generated metafile, all urls are prefixed by the plugin's namespace except the stdin file which, in this case takes its path from stdin.sourcefile.

• In the generated metafile, I can't see to work out how path resolution is working. I end up getting the single outfile's path being "../../../../../stdin.js".

Ideas

1. Propagate a configured stdin.sourcefile value through to the .importer of onResolve plugins instead of <stdin>.
2. Allow entrypoints to be resolved and loaded by plugins. Perhaps a mechanism could be provided to explicitly fall back to built-in logic.
3. Consider surfacing the namespace and filename separately for entries in the meta.json output to break ambiguity between url paths and namespaced paths.
4. Consider allowing plugins to control all filesystem I/O (and knowingly suffer the overhead of the extra message passing).

All this being said, I've found work-arounds for everything and it's really AMAZING what can be done so easily :D Thank you again for all your work.

[evanw]

In the generated metafile, I can't see to work out how path resolution is working. I end up getting the single outfile's path being "../../../../../stdin.js".

Paths in error messages and the metafile are relative to the current working directory.

1. Propagate a configured stdin.sourcefile value through to the .importer of onResolve plugins instead of <stdin>.

Yup, makes sense. I think this is probably just an oversight.

2. Allow entrypoints to be resolved and loaded by plugins. Perhaps a mechanism could be provided to explicitly fall back to built-in logic.

I've thought about this before but there's an issue I would need to solve. Someone else filed this too: #546. I wrote some more thoughts over there.

3. Consider surfacing the namespace and filename separately for entries in the meta.json output to break ambiguity between url paths and namespaced paths.

I'll consider this. It would be a very different structure though since JSON doesn't have non-string keys. So it'd be a big breaking change.

4. Consider allowing plugins to control all filesystem I/O (and knowingly suffer the overhead of the extra message passing).

I will consider this. It would indeed be a lot of overhead. It would also not work with other plugins, which could still access the file system anyway, so it wouldn't be a general-purpose mechanism. FWIW with the plugin system you should be able to avoid all of esbuild's behavior that's specific to the file system by just not working in the file namespace (once the entry point thing is solved).

[ggoodman]

It would be a very different structure though since JSON doesn't have non-string keys. So it'd be a big breaking change.

Maybe instead of changing the keys, new properties can be added so that entries look like this:

{
  "inputs": {
    "velcro:https://cdn.jsdelivr.net/npm/[email protected]/index.js": {
+     "filename": "https://cdn.jsdelivr.net/npm/[email protected]/index.js",
+     "namespace": "velcro",
      "bytes": 2108,
      "imports": []
    }
}

That should be enough to break the ambiguity from the perspective of a consumer of the meta.json.

[ggoodman]

As you explore what the plugin API might look like in esbuild-wasm, I'd love to know what you think about resolving 'ambient' files that esbuild does on consumers behalf but that might be impossible in a WASM environment.

Specifically, I'm thinking about tsconfig.json detection and loading.

A related thought is that esbuild has already solved for the intricacies of resolving via graphs of tsconfig.json files and its own world of path mapping. How can a plugin defer to some of this logic?

I'd love to see a method exposed via the plugin handler function argument that lets us invoke such logic (acknowledging the cost of the extra message passing).

I can understand the perspective on the overhead of all this back-and-forth but believe that even in pathological cases, esbuild will remain orders of magnitude faster than the status quo. It still blows my mind that the same amount of work is SO much slower with the traditional tools.

[ggoodman]

@evanw another challenge with the stdin API is that we can't inject multiple entrypoints. This would be really helpful for projects whose files are not coming from disk.

[evanw]

This has been fixed in version 0.8.21.