Describe separate the compilation schemes

[nomeata]

this doesn’t really change the meaning of the document, but describes
the intermediate step a with a bit more detail. This can be seen as a
compromise that I can live with being merged into master.

[matthewhammer]

Noted minor typos; added a question about release times.

[matthewhammer]

For symmetry with the text above, you may mention that: "incremental compilation via Wasm seems incompatible with the interpreter, but will be useful for latter versions of the compiler."

[nomeata]

I guess the question is: How will it work with the intepreter? I.e. is there the assumption that the .as files are always available next to the compiled .wasm thing?

[matthewhammer]

So the negation is true: 0.5 will support these things? Sorry, I'm forgetting the consensus opinion about the timeline for these features, though I know we've discussed that before.

I don't see this question answered in the AS planning google doc, but maybe I missed it?

[nomeata]

I don’t think we need to enshrine it in the design docs. But yeah, 0.5 would do “Source imports” I think

[rossberg]

We probably should put everything dedicated to AS into a single "actorscript" section with subsections. Spreading it over multiple sections introduces a bit of extra complexity.

(FWIW, we should do the same with the Dfinity section.)

Do we also envision the possibility of an DIDL section? Or should that be described somewhere else?

[nomeata]

We probably should put everything dedicated to AS into a single "actorscript" section with subsections. Spreading it over multiple sections introduces a bit of extra complexity.

I think there is merit in separating the information needed for the type checker (and, e.g., a SDK) from the information needed for the code generation. We can even make a stable specification for the former.

(FWIW, we should do the same with the Dfinity section.)

What is the DFINITY section? The one for the old system API? That should be gone with the new System API anyways.

Do we also envision the possibility of an DIDL section? Or should that be described somewhere else?

Clearly somewhere else, as it is language-independent, and needs input from the SDK team. Also, it would be a section on a deployable canister module, not an AS unit, right?

[rossberg]

We may still need a custom section to declare which memories/tables/globals are supposed to be persistent. Though come to think of it, we might differentiate that with appropriate import naming conventions.

[nomeata]

Really? I thought the point of orthogonal persistence is that, well, the whole state (“store” as per WebAssembly spec) is persisted.

But this is out of scope for this discussion. If there is a need, a PR against the end2end spec would be appropriate.

[rossberg]

Do you envision this in addition to the "natural" scheme? I assume the benefit you have in mind is performance? Do we know that's not a premature optimisation? And would it be possible to support optimised calling conventions as an augmentation to the "natural" format, e.g., by exporting additional entry points?

[nomeata]

It’s not (just) performance, it is also simplification.

The “natural” scheme (and I should call you out for using a term that implicit carries judgmental value, but at least you are putting it in quotes ;-)) has lots of complications for the backend – eta-expansion of closures, re-boxing of modules on the importing side, specifying a stable name mangling of field names.

So I expect to implement this scheme will be implemented before the “natural” scheme, which would already give us a usable model for incremental compilation. And the “natural” scheme can follow later.

[rossberg]

Hm, surely, supporting two schemes is not a simplification?

[nomeata]

No, but starting with the simpler, more flexible scheme is :-)

[rossberg]

So you are suggesting it as a temporary solution? Why put it in the design doc then?

[nomeata]

No, I am suggesting this as a viable, useful and realisitc solution for 0.5 and 1.0, leaving the “natural” solution (which I am still not sold on) in the document as a vision for post 1.0 as a compromise.

[nomeata]

(Selling me on this would require a convincing response to, say, #452 (comment). Or a concrete usecase for who would use these “natural” entry points – especially since you admit that asc itself would be allowed to export and use additional faster entry points.)

[rossberg]

I see what you're doing there ;). But I'm afraid I'll insist that this doc describes the finalised module representation, and that it's what we should have implemented by 1.0. If you think we can move faster by first doing a temporary hack that's fine, too, but it can't be more than that.

Look, Wasm defines a notion of module where the entry points to a module are represented as exported functions. That is what every tool and framework in the eco system understands. It is what relevant developments like WASI build upon. It is the minimum foundation to enable interoperability. (The only alternative is sharing a single table across all modules, and that introduces tighter coupling and hence is a less modular approach. Edit: it also causes more trouble for orthogonal persistence.)

AS isn't an island. To be practical we'll eventually need to support linking with foreign libraries in user land. And clearly we want no N^2 FFI and tooling problem. Hence, ultimately the question isn't whether we support the native platform mechanism, but only whether we also maintain something home-brewed in addition. Clearly, it wouldn't be a simplification but a complication to maintain two different mechanisms. The only justification for such a complication would be a significant performance improvement, but currently that's hypothetical and hence a premature design optimisation. Should we ever identify this as a bottleneck, and should whole program compilation not be good enough to address it, then we can still add a secondary mechanism to the design.

[nomeata]

Well, then we are at an impasse.

An AS module exports AS values of various types. Some of them have don’t have function types, some dot. That does not make them WebAssembly functions! AS function ≠ WebAssembly functions – the former can be closures, they can be computed, they can be put in heap data structures, some may not even be WebAssembly functions (e.g. if an argument has type None, they might feasibly be absent).

Maybe at some point there is an realistic interop story for WebAssembly. And maybe we want to use it. That’s already two maybes that are likely not true for our 1.0.

But even if these maybe become true, there will still be a significant mismatch between anAS function and a WebAssembly interop function! Bridging this mismatch will likely

• incur an implementation overhead (already hinted here: eta-expanding non-manifest functions, for example)
• incur a performance overhead (marshaling arguments from and to our uniform representation)
• require developer guidance (i.e. how to marshal these arguments, if that is not completely type-driven)
• still require special handling by non-AS code, such as initializing the AS runtime and somehow coordinating memory management.

So the viable path I see is a separation between

• AS modules (for consumption by other AS code) without any external usability guarantee, using the “flexible” format, and
• Interop modules (for consumption by other WebAssembly code), using a “natural” interop format, with some stability,
  and I expect the latter to be it’s own kind of AS unit, because of additional restrictions on what we can export (e.g. only types that have meaning in the interop world), and maybe additional requirements from the user (e.g. annotations on how to present/expect arguments).

This is not a N^2 FFI story. There is only a single FFI story. But it is an FFI story, not a “all AS code can interop” fairy tale story.

There are some languages that can pull of the seamless integration with a low-level interop language. Rust code, for example, is particularly easy to call from and to C. But that is because it is a low-level language! A high-level language (with RTS and GC, including Ocaml, Haskell, but also like AS) cannot provide that, and usually has dedicated, separate support for “creating a native library”.

And even Rust code does not provide a native library by default, but you have to tell the compiler to do so (via crate-type = ["dylib"], for example), and requires annotations like #[no_mangle].

So what you want to achieve here seems to be overly ambitious for what we need to deliver, and also seems to be guided by your vision for a bright WebAssembly world, rather than what we actually need.

[crusso]

I'm actually with Joachim here. I don't see how we can provide a useful notion of cross language wasm module without specifying an ABI. The Rust/C interop presumably only works (if it does at all) because of an agreement to use the C ABI at the boundary and the ability of Rust users to write unsafe code that does the appropriate thing with C values. It we really wanted to support this then we'd have to use a common ABI at the boundary, would we not?

[kritzcreek]

Name mangling isn't as much of a concern for us because WASM already provides proper name spacing, right? What if we said that the only stable ABI we provide is functions on basic WASM types + func references. So if you define a List type in AS, you can't trust that whatever representation that has at runtime will be stable across AS releases. Instead you'd need to expose a function from AS that converts that list into an Array (or the WASM equivalent of that) and that's the function you'd call from WASM.

[nomeata]

functions on basic WASM types

But because of uniform representation, an ActorScript function func Word64 -> () still takes a pointer to the heap, not a WebAssembly i64.

Array (or the WASM equivalent of that)

That doesn’t exist

[rossberg]

@crusso, thanks for bringing up the notion of ABI, because that's exactly what I'm alluding to. One way to understand Wasm is that it has a basic, canonical ABI built in already, which includes functions over basic numeric datatypes. All ongoing efforts for richer Wasm interfaces (WASI, webidl-bindings, export bindings) are an extension of this. Playing along nicely is clearly not sufficient for more general interop, but necessary -- and hence simply future-proofing and canonicalising the design, no more no less.

@nomeata, you keep reiterating points that I already addressed. None of your examples of differences between (exported) AS functions and Wasm functions are inherent, they're all overcome by one simple implementation technique -- which I trust somebody as capable as you would implement in less than a day. Your assertions about simplicity only hold under the (unreasonable) assumption that we'll never care about interop -- or even backwards compatibility; they are inverted otherwise. The other "overheads" you list are not relevant as far as merely following an "ABI" is concerned.

I'm fine with keeping the "flexible" implementation going for a while, but not with labelling that a design. The "implementation-dependent" tag it has all over it obviously is a big red flag in that regard. That's a no-go if you want to be able to deploy library binaries instead of just having them as intermediate build targets. Can we try to think ahead a step or two?

[nomeata]

I keep reiterating them because I don’t think they are addressed. I asked for a concrete use-case, and have not seen one. At some point you said that value interop is not part of this. But without value interop, there is no useful interop to me. And I don't see how value interop will play nicely with our requirements for uniform representation, and our custom requirements for memory management, including the freedom to, say, add a shadow stack at some point for GC.

At some point you said that “well, ActorScript can use additional entry-points”, presumably for functions following the AS calling convention, rather than the WebAssembly interop. But then you are basically saying that you want the compiler to produce entry points that it doesn't actually use in most cases.

Maybe the problem is that there is a difference between an “compiled module” (e.g. .o file in the C world) with the purpose of incremental or separate linking, and a “self-contained shipable library” (e.g. an .so file in the C world), with a proper name, versioning? The latter likely contains multiple of the former, and a typical project will want to use a large number of source files without turning them into individual .so files.

If we transfer that our world, then the “shipable library” is more on one level with “program” and “actor”, and maybe needs a design that is detached from “importable ActorScript file”?

Can we try to think ahead a step or two?

I am trying! And the result is always: Producing a shipable library that supports lower-level interop is a separate thing that needs dedicated design (possibly including separate syntax, specific type restrictions etc.) and does not have to nor should affect the design for incremental or separate compilation within a project.

[rossberg]

Maybe the problem is that there is a difference between an “compiled module” (e.g. .o file in the C world) with the purpose of incremental or separate linking, and a “self-contained shipable library”

That may be part of the impedance mismatch, although we surely want to support offline linking with foreign libraries eventually as well.

More generally, the crucial thing to realise is that our platform is an online environment, where binary code is going to be shared on a network. In such an environment, backwards compatibility is essential, as providers of a piece of code cannot control the versions of tools client code is using, nor can they request clients to simply recompile. Code you have to interact with may not even be maintained anymore.

This is similar to the web, but in stark contrast to offline environments like those you may be used to.

And that's why future-proof design is far more important. In such an environment, future-proofing avoids evolutionary dead ends and fatal backwards compatibility traps. That has been an overriding consideration in many of (JS's and) Wasm's design decisions, and has contributed significantly to their ability to evolve.

Future-proofing does not mean implementing a complete solution. On the contrary, it's often most relevant when you do not know the full solution yet, but have partial knowledge about it. Then you avoid incompatible design decisions and prepare the space for seamless extension later. It's like building a driveway without paving the road behind it yet.

[nomeata]

No we are getting philosophical :-)

offline linking with foreign libraries eventually as well.

Why eventually? We already have as-ld (although the “as” is a misnomer, there is nothing as-specific about it). So offline linking with foreign libraries is not a problem.

But thanks for introducing the term “foreign library”. This starts to address my main issue, namely that I think it is unfeasible to conflate foreign linking (according to some lower-level foreign interop spec) and AS-internal linking, because for the former, a less powerful, less expressive common denominator programming model needs to be used (and hiding that from the programmer is unlikely to work nicely), while the latter can be completely seamless for the AS programmer.

Fundamentally, I don’t share the vision for an “online platform”, where you import pre-compiled libraries “from the cloud”, maybe even in a way where these libraries can update behind your back, and am happy with an clear stratification between local development (you download all the libraries you use, probably pinned by version, and build a self-contained canister locally) and in-cloud execution (the developers uploads all the stuff). This seems simpler, more robust, more flexible ot me. And, it seems, it leads to different technical design decisions.

And even if we go towards a separate linking story, then I resist signing off a spec that enshrines 5% of the necessary detail, namely “values of AS function type must be Wasm functions” (causing the implementer to do additional, and in isolation pointless, work), without defining the remaining 95% of the ABI (which would require specifying the heap structure and memory management as well…)