WebGPU as low level graphics API

[olanod]

Soon will come the time to decide how to pain something on a screen, are there discussions about this topic in the community? didn't see any. Seeing what WebGPU is supposed to do seems like a natural fit for WASI to use this API instead of defining its own or doing something WASIVulkan/WASIMetal ... I imagine on top of, next to it, or even bellow, other APIs could be defined right? like stuff to draw 2d graphics web 2d canvas style, or create windows, or not sure if it would need to go lower level like defining a compositor, a framebuffer, etc.
I imagine Rust WebGPU implementation should make it easy for wasi runtimes like wasmtime or wasmer to come up with a working prototype since they are rust based, there are idiomatic rust wrappers, works natively on windows, mac and linux and there are a WebIDL definitions.

[sunfishcode]

I agree. As WASI grows, it's going to want a lot of the same functionality as the Web. And especially once we have webidl-bindings (or whatever that evolves into), there will be a straightforward way to directly use APIs like this.

WASI does have its own needs and use cases outside of the Web, and I expect it won't always make sense to use Web APIs. I haven't looked at WebGPU in detail myself, however graphics APIs are an area where using the Web API would have major advantages.

[sbc100]

Sounds plausible. I can also imagine direct Vulkan bindings being created.

Seems like it will depends on a couple of factors:

1. How much complexity would there be in the WebGPU -> Vulkan layer need to implement such a thing.
2. How much existing code out there is written for Vulkan vs WebGPU APIs.

[MarkMcCaskey]

I think one of the benefits of using some of the web technology is that they're the only, as far as I'm aware, ones thinking about untrusted use of the GPU. There have been some nasty security vulnerabilities with WebGL and one of Vulkan's selling points, as I understand it, is that it doesn't do runtime checking in release mode. I recall a number of years ago that malformed Vulkan without the validation layer could get the GPU into a bad state or crash the operating system or something in certain circumstances. Given that, I think exposing raw Vulkan will only make sense if the Vulkan drivers are built with malicious use in mind.

Vulkan seems like a good choice for building something on top of though!

More generally though, it's really tricky. Certain things like denial of service by asking the GPU to do a ton of work are still an issue with WebGL as far as I know.

I'm not an expert here by any means, but I am excited about getting graphical programs working with WASI!

[olanod]

@sbc100 1. I don't think there will be much complexity, I don't know either API but as I understand it WebGPU is being designed with heavy influence from Vulkan/Metal, Google and Mozilla. And the mentioned native rust implementation already works using dx12, vulkan or metal depending on the platform. 2. Of course code written for WebGPU is almost non existent since it's so fresh and still being defined but is totally worth considering it as the one backend to rule them all since it will rule the web for sure 😎

[aardappel]

It may be worth it to make the first WASI graphics API some kind of lowest common denominator API that works well on desktops, mobile, and the web alike, similar to OpenGL ES 3 / OpenGL 3 / WebGL 2. The point being that such an API would map relatively 1:1 onto these 3 types of platforms, and that a lot of code / game engines out there (in non-web land) already targets these APIs so would need few to no modifications to work (as opposed to WebGPU, which would require whole new engine ports).

Also note that WASI doesn't need to restrict itself to one graphics API, we can spec a WASI-Vulkan later, and/or WebGPU if it makes more sense. OpenGL may seem like an old-fashioned API, but it is easy to work with, and the amount of code out there that works with it dwarfs the other API options right now.

Once we have OpenGL, it may be possible to implement (a subset of) LibSDL to work on top of both the Posix and GL parts of WASI, further extending the amount of code that can run on top of this with little modification.

[programmerjake]

If we decide to supply OpenGL, I think providing it through EGL is probably best, since engines/applications can reuse their EGL code, we can follow a spec that's already been thought out and in wide deployment (Android, Linux, and more), and it allows relatively easy customization through EGL extensions. It additionally allows binding OpenGL, OpenGL ES, OpenVG, and more.

[tangmi]

I'd like to make another argument for WebGPU, which is a safe Metal-like modern graphics API that is being implemented in Rust with pluggable backends (as well as getting native support in browsers). This should mean that WASI could target Vulkan, DX12, DX11, Metal, OpenGL/ES, and WebGPU itself through a WebGPU interface.

It'd be lovely to see a WASI backend targeting the web and exposing a WebGPU-like interface via either native browser support or a polyfill via wgpu. It could mean truly "write once, run everywhere" (which hasn't always been true with regards to graphics, even with OpenGL).

Edit: Something we may also want to consider is how the graphics API will interact with composition APIs (e.g. DirectComposition on Windows, Core Animation on macOS/iOS, SurfaceFlinger/Control on Android, the DOM (?) on Web, etc.).

[doug-moen]

WebGPU is the only universal GPU API that is safe, portable, high level, high performance, and supports modern GPU features like compute shaders. Portable means: guarantees that code has the same behaviour on all platforms, backed by a conformance test suite. It works in web browsers, Windows, Mac, Linux, and mobile. Mozilla and Google are both implementing C library interfaces that target multiple platforms, and they have pledged to use a common WebGPU.h header file to promote portability. Mozilla's Rust library is layered on top of their C interface.

This looks like the next big thing. Game engines are paying attention. I'm planning to dump OpenGL and switch to WebGPU when it is ready, because I don't want to create my own abstraction layer that supports multiple back ends. OpenGL won't get me compute shaders on MacOS or the web. Vulcan will never run on the web. WebGPU is the only API that gives me compute shaders on the web, so why not just use it on all platforms?

[MarkMcCaskey]

Following up on WebGPU and OpenGL:

There's an interesting argument against OpenGL in the Glium post-mortem, namely that "Drivers are still crippled with bugs" [so abstracting over OpenGL in a cross-platform way isn't practical].

I've heard that WebGPU is more like Vulkan than it is like OpenGL; can anyone with context on WebGPU speak to this?

Security and being properly portable are my biggest concerns so WebGPU seems pretty appealing.

[tangmi]

I've heard that WebGPU is more like Vulkan than it is like OpenGL; can anyone with context on WebGPU speak to this?

WebGPU is the newest of the "modern" graphics APIs (e.g. Vulkan, DX12, Metal, Mantle), which I distinguish from the older APIs (OpenGL, DX11) as having explicit command queues opposed to implicit ones. As far as similarity, WebGPU compares closest to Metal (probably since Apple is the one that originally proposed it)--both don't require manual memory management while DX12 and Vulkan do. Like Metal (and OpenGL), I think WebGPU is a wonderfully friendly API that requires little boilerplate to be productive.

To me, the most compelling argument for WebGPU on portability is that it seems straightforward (although not trivial) to implement some standard OpenGL on top of WebGPU as a library (there's already a handful of OpenGL on Vulkan libraries, like GLOVE and Zink), while the opposite is neither straightforward nor trivial (especially given the problems addressed in the glium post-mortem).

[programmerjake]

To me, the most compelling argument for WebGPU on portability is that it seems straightforward (although not trivial) to implement some standard OpenGL on top of WebGPU as a library (there's already a handful of OpenGL on Vulkan libraries, like GLOVE and Zink), while the opposite is neither straightforward nor trivial (especially given the problems addressed in the glium post-mortem).

That's one of the major reasons we at (libre-riscv)[https://libre-riscv.org/3d_gpu/] are just writing a Vulkan driver and not an OpenGL driver for our open-source GPU.

[jpryne]

If someone is willing to layout the structure of a WebGPU-WASI implementation in Rust, and at least one or two example functions, I will devote my time to fleshing out remaining functions. IOW, plant the seed & I will water it.

[sunfishcode]

@jpryne An interesting thing about WebGPU is that it has a WebIDL API, meaning that once everything is ready we can use interface types to talk to it from wasm.

One of the first tasks for WASI is to evaluate that API to see if it's compatible with WASI -- that it doesn't make any assumptions about running inside a browser or having JS available, and that it fits within a Capability-based sandboxing model (roughly speaking, all functions should operate on a handle returned by the library, rather than having global impact).

If someone wants to start prototyping an implementation, one possible path is to hook up Rust WebGPU and the idiomatic Rust wrappers to the wasi-common standalone implementation of WASI in Rust. To reduce the amount of boilerplate needed, it may be desirable to build a tool to auto-generate some of the interface from the WebIDL source mentioned above, possibly with help from wasm-bindgen.

There's a lot in there, but feel free to ask questions!

[jpryne]

Thanks for your suggestions. I think, in the absence of examples to study, I should be starting with editing the documentation, (i.e. Todo: Update the IR Reference)

[bhack]

Is this only about GPU or could be related on any other kind of accelerator?

[doug-moen]

@bhack: The WebGPU standard only covers GPU. WebGPU corresponds to the intersection (common subset) of Vulkan, DX12 and Metal. Other kinds of accelerators (tensor processing units) are out of scope.

[tangmi]

If you have some time, @sunfishcode, help me understand what work needs to be done:

• API evaluation: I've looked at the WebGPU API and it seems compatible with how you've described WASI's sandboxing model. The entire API goes through a context object (Device) (sometimes indirectly through types that have a reference to the Device). As an implementation detail of wgpu, multiple instances of Device share the same gfx instance and resource registries (I think this just means two devices in the same process can't create resources with the same identifier?), but I dont think this intereferes with WASI's sandboxing.
• hostcalls: You mentioned boilerplate code needing the be written (or generated) in wasi-common. Are you referring to the hostcalls macro invokations in wasi_common::hostcalls::*?
• Windowing: wgpu needs a window context to present to, but I'm unsure how that should be exposed to WASI. I think for WASI to be successful at filling the 2D application niche, it needs to support more complex windowing APIs (i.e. OS composition layers), but for this work maybe it's fine to define a simple "create this window and wgpu device"?

Sorry if this is a little disorganized--I'm trying to wrap my head around how all the pieces fit together. From my understanding, one needs to:

1. Write (or create a tool to generate) a file at wasi_common::hostcalls::webgpu, containing essentially wgpu-native modified to input/return WASI types.
2. Implement wasi_common::hostcalls_impl::webgpu that mostly forwards to wgpu-native or wgpu-rs.
3. Have wasi_common::hostcalls_impl::webgpu call to wasi_common::sys::*::hostcalls_impl::webgpu to handle platform-specific stuff (device creation/window creation?)
4. Add the webgpu api to rust-wasi (?) so that users can consume the API (e.g. wasi-misc-tests?)

Please let me know where I'm hand-waving too much! If I can understand what needs to be done, I'm interesting in taking a shot at it!

[sunfishcode]

I apologize; I somehow missed your message earlier. Some pointers:

• There's been some preliminary discussion of Windowing APIs here.
• And yes, the rust-wasi crate, since renamed to just wasi, would be a good place to host Rust API bindings for this.

[AldaronLau]

@sunfishcode Has anyone attempted to start a specification for a Windowing / WebGPU / Framebuffer module? I'd be interested in contributing in this area, but I don't want to do duplicate work.

[sbc100]

I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface). Basically it's low level bindings to Khronos Group APIs, the windowing API is somewhat derived from the now defunkt OpenKODE API; Graphics from OpenGL and Vulkan, Audio a mix of OpenSL and OpenAL.

Oh my, the mention of OpenKODE certainly brings back memories! I was involved with that at the time too. Certainly a lot of overlap with what wasi is doing today.

[jpryne]

"I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface)..."

Btw, "waki" is a Japanese word meaning "sword", (as in "wakizashi",) along with other things. Let me know if you'd like a logo...

[Kagetsuki]

"I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface)..."

Btw, "waki" is a Japanese word meaning "sword", (as in "wakizashi",) along with other things. Let me know if you'd like a logo...

Waki is arm pit. Wakizashi is 脇差, which is 腋 "arm pit" and 差 which means like to put something through something and hold it there (like hold by sticking through a belt). The short sword known as wakizashi is just how it's called because it was a short sword designed to be fastened under your arm/arm pit.

[trusktr]

Here is a TypeScript program compiled to WebAssembly with AssemblyScript, using WebGL APIs outside of a browser in Node.js (the JS glue code calls the webgl-raub package):

assemblyscript-webgl-node.mp4

The WebGL code is written in AssemblyScript, and currently starts like this:

import {WebGLRenderingContext} from '../node_modules/aswebglue/src/WebGL'

const gl = new WebGLRenderingContext(...)

// call gl.whatever as you would in JS/TS

This means we at least have a WebGL interface in AssemblyScript currently working in Node or browsers.

Next we would need to make the WebGL (or WebGPU) interface in WASI, but the AS code would remain the same.

[trusktr]

Oops, I forgot links!

ASWebGLue: https://github.com/battlelinegames/ASWebGLue

LUME Glas (port of Three.js to AssemblyScript that will likely fork direction): https://github.com/lume/glas (part of the LUME project, http://github.com/lume/lume)

AssemblyScript's #gamedev Discord channel where we chat about ASWebGLue and AS game dev: https://discord.gg/A5n6qdeYfR

LUME's #glas channel where we talk about ASWebGLue and Glas: https://discord.gg/6XvnkMb

The Node-AssemblyScript-WebGL example isn't pushed up yet, but will be soon...

[Toasterson]

Has somebody picked this up to make a demo? Otherwise I will make one with the rust-wasi crate as outlined here, so anything that has moved since the last comment would help me here.

[MaxGraey]

I am also interested in a webgpu submodule for WASi. But what confuses me is that there are two types of WASI notation and tools for codegen - witx (https://github.com/WebAssembly/WASI/tree/main/tools/witx) and wit (https://github.com/bytecodealliance/wit-bindgen). As I understand the second one is newer, but it is under the Bytecodealliance umbrella and I understand it is still under development. It would be great if someone from Wasm WG could clarify which tools to use and how stable they are.

[ShadowJonathan]

I also believe this should have someone from api design to look at, to enumerate the webgpu api module's endpoints.

[sunfishcode]

@MaxGraey witx is the tooling that wasi_snapshot_preview1 uses. It's been around a while, we've gotten a lot of feedback and experience with it. We learned a lot, including that we'd really like to do several things very differently. It's also significantly out of date at this point with where interface types and module linking have evolved to.

wit is a new format being designed to be part of interface types, which is under the WebAssembly CG umbrella. As a part of interface types, it'll be kept current with interface types evolution. And, besides having its own syntax instead of S-expressions, it also avoids the need for awkward @interface and @witx annotations, so it's much less verbose than witx. And the WASI subgroup is already starting to move to this new WebAssembly format.

wit-bindgen is an implementation of a bindings generator that uses this wit format. It's new, and under active development, and it's already much more powerful and easier to use than current witx tools. For example, in the witx tools, "string" is only supported for arguments. In wit-bindgen, "string" is also supported as a return type, record field, tuple element, and more.

At this time, I recommend people writing new interfaces to use the wit format, and the wit-bindgen tools. There are some limitations, and not everything is stable yet, but they're already far ahead of the witx tooling in almost every way, and they're being actively developed.

[MaxGraey]

@sunfishcode Thanks! That's basically what I thought. It's just that there is very little information on the topic. It wasn't even clear whether wit was the official direction of development. It is now clear what it will be. It would be nice to add a full documentation / specification for wit. It would also be nice to mention on official WASI channels / resources about witx deprecation and WIT and the tools for it should be used.

[trusktr]

Related: what about WebGL? I know WebGPU is superior, but there are still many apps written with WebGL, and WebGL isn't going away any time soon.

Soon, it will be possible to port Three.js apps (WebGL-based) to lume/glas. It'd be sweet to be able to compile them to native too. :)

[atx-barnes]

Any update on this thread? Very interested in support for WebGPU

[Toasterson]

I had to move to other work for multiple reasons. Feel free to pick it up.

[sagehane]

"I've been silently working on a separate thing dubbed WAKI (WebAssembly Khronos Interface)..."
Btw, "waki" is a Japanese word meaning "sword", (as in "wakizashi",) along with other things. Let me know if you'd like a logo...

Waki is arm pit. Wakizashi is 脇差, which is 腋 "arm pit" and 差 which means like to put something through something and hold it there (like hold by sticking through a belt). The short sword known as wakizashi is just how it's called because it was a short sword designed to be fastened under your arm/arm pit.

Sorry for going completely off-topic but: tl;dr, at least in common usage, waki on its own often refers to armpit but the waki in wakizashi often means "a subsidiary/complementary sword".

脇 does mainly mean "armpit" but it also means "something that's not the main focus". 脇役 (in which 役 means "role") means "side role/character" for example.

Although the exact origin of the meaning is disputed (and your interpretation is indeed one of them), it's also argued that it referred to "a sword that was hidden/concealed under a person's clothes".

And by the 16th century or so, it became commonplace for bushi (armed people) to carry two swords: 大刀 (big sword) for longer reach and a 小刀 (small sword) for indoor usage. The two were also referred to as 本差 (honzashi, main sword) and 脇差 (wakizashi, sub sword). Both swords would be strapped near the waist so it clearly didn't refer to the body part in this case. This is probably the usage most familiar to a layman.

Source: My uncle used to work at Japan

Annnnd... yeah, I would love WebGPU support in WASI, yeah... Yeah...

[maxammann]

I'm developing a map renderer in Rust called maplibre-rs based on WebGPU which can compile to WASM and run in the browser.

I'd be very interested in being able to run it in a WASI runtime. I guess for me it would be a toy project though.

Though, it definitely has potential in the server space. Maplibre-rs also supports a headless mode.

A WASI binary deployed on a server could render PNG raster tiles :) I'm not sure where WASM is going, but that would be a cool usecase for this feature.

[anyputer]

Consider if Wayland is viable for windowing. It is security-minded and can be a good portable abstraction, see WSLg for Wayland under Windows. Windowing/GUI toolkits can mostly (if not completely) reuse their Wayland code for WASI, maybe requiring some changes in libwayland-client, and in the WebGPU implementation. For best portability between different Wayland compositors, the Wasm runtime could expose a restricted set of Wayland protocols. There can be a new Wayland protocol for providing access to WebGPU in the browser or on native. I think this is worth exploring.

[i509VCB]

Consider if Wayland is viable for windowing. It is security-minded and can be a good portable abstraction, see WSLg for Wayland under Windows. Windowing/GUI toolkits can mostly (if not completely) reuse their Wayland code for WASI, maybe requiring some changes in libwayland-client, and in the WebGPU implementation. For best portability between different Wayland compositors, the Wasm runtime could expose a restricted set of Wayland protocols. There can be a new Wayland protocol for providing access to WebGPU in the browser or on native. I think this is worth exploring.

Requring WSLg to be installed would more than likely be a non-starter for WASI. For the scope of a windowing library maybe something with an api surface like winit could be considered which is a relatively new toolkit and does run on most platforms.

On macOS there is Owl which does exist for Wayland on macOS, but it ultimately has to follow the rules Cocoa sets.

[MendyBerger]

This is my attempt to get this going.
https://github.com/MendyBerger/wasi-webgpu