feat(crypto): Port to Wasm (in a JS host) and to NodeJS #675
Conversation
This patch updates to code to raise a compilation error if the `js` feature is used for another architecture than `wasm32`.
Ask `rustc` to generate a dynamic system library, which will be useful to generate a Wasm module.
|
Fixes #681 |
|
The code has moved into its own crate: |
|
Yeah, it's two approaches for the same problem. I used Cargo features because it's just shorter to write |
Hywan
changed the title
|
The priorities of the company have shifted a little bit. For the next days, I'll focus on NodeJS bindings. I'll come back to Wasm+JS bindings after that. With the team, we have agreed to review and to merge this PR, then I'll open a new PR for NodeJS only, and another one to finish Wasm+JS only. |
Codecov Report
@@ Coverage Diff @@
## main #675 +/- ##
==========================================
- Coverage 72.72% 71.21% -1.51%
==========================================
Files 109 116 +7
Lines 15974 16291 +317
==========================================
- Hits 11617 11602 -15
- Misses 4357 4689 +332
Continue to review full report at Codecov.
|
…y code. `wasm_bindgen_future::future_to_promise` expects a `Future<Output = Result<JsValue, JsValue>>`. We reimplement this function by expecting a `Future<Output = Result<T, anyhow::Error>>` where `T: Into<JsValue>`. That way, we apply the type conversions to `JsValue` inside this helper rather than in the call site. Additionally, all errors are managed automatically without having to deal with `JsError` or `JsValue`. It makes the code simpler to read and easier to write from my point of view.
Why? Because `napi` and `wasm-bindgen` are too different. At this step, the most notable bugs are the way `napi` is handling its proc macros. There is too much conflicts when used with `#[cfg_attr]`. It makes the code repetitive and harder to read and to understand (and also to compile, we must be very careful). But on the short-term, quickly, we will see more notable differences between `wasm-bindgen` and `napi`, e.g. with array (in `wasm-bindgen`, we can downcast array items into particular types, with `napi` it's going to be a very different code). Instead of fighting the proc macros bugs now, and having to split the code later inside the same crate, we believe it's a good idea to split the code now into 2 crates. At first we will see obvious code duplications, but on the short-term, the code is likely to be more and more different.
The why and how
We want all the clients (as in client-server terminology) to use the same Rust code base for the Crypto part, namely: Element iOS, Element Android and Element Web.
Element iOS and Android are using (or soon) the
matrix-sdk-crypto-fficrate from this repository. What is important for us in this issue is: Element Web is relying on thematrix-js-sdklibrary.This
matrix-rust-sdkprovides the Rustmatrix-sdk-cryptocrate, which already compiles to Wasm (short for WebAssembly, i.e. thewasm32-unknown-unknowntarget). However, no symbol is exported or imported, which means that it cannot be used. Note: I'm not minimising the crazy efforts that were done to reach this milestone, it implies writing Vodozemac etc. Kudos!The idea of this PR is to push on the previous efforts by exposing Wasm functions and types to the host. Which host? Well, we want to target JavaScript for now so that the resulting Wasm module can be used by the
matrix-js-sdk, and thus, by Element Web.The plan is then to provide a JavaScript API through WebAssembly to the Rust
matrix-sdk-cryptocrate. This JavaScript API should ideally come with TypeScript definitions. To achieve that, we use thewasm-bindgenproject. To facilitate the compilation, optimisation, and bundling, we use thewasm-packproject.Edit: We need to support NodeJS too (see #699). We have started by adding NodeJS support within the same codebase, within the same
matrix-sdk-crypto-jscrate, but now, the code is split into 2 crates, please join me to welcomematrix-sdk-crypto-nodejs. Why? Becausenapiandwasm-bindgenare too different. At this step, the most notable bugs are the waynapiis handling its proc macros. There is too much conflicts when used with#[cfg_attr]. It makes the code repetitive and harder to read and to understand (and also to compile, we must be very careful). But on the short-term, quickly, we will see more notable differences betweenwasm-bindgenandnapi, e.g. with array (inwasm-bindgen, we can downcast array items into particular types, withnapiit's going to be a very different code).Instead of fighting the proc macros bugs now, and having to split the code later inside the same crate, we believe it's a good idea to split the code now into 2 crates. At first we will see obvious code duplications, but on the short-term, the code is likely to be more and
more different.
The timeline
At the time of writing, it's been 6 days I've started my work at Element/Matrix and I'm still not entirely familiar with the codebase.
It's not trivial to define a progression timeline, but since the
OlmMachineis the main entry point of the Crypto API, I reckon focusing on its implementation progression would provide a good intuition of the overall status.OlmMachine:async new,user_id,device_id,identity_keys,async display_name,tracked_users,async outgoing_requests,async mark_request_as_sent,async bootstrap_cross_signing,async get_missing_sessions,async encrypt_room_event,async encrypt_room_event_raw,async invalidate_group_session,async share_group_session,async receive_unencrypted_verification_event,get_verification,get_verification_request,get_verification_requests,async receive_sync_changes,async request_room_key,async decrypt_room_event,async update_tracked_users,async get_device,async get_identity,async get_user_devices,async import_keys,async export_keys,async cross_signing_status,async export_cross_signing_keys,async import_cross_signing_keys,async sign.How to test
pkg/{*.d.ts,*.wasm}files will be generated incrates/matrix-sdk-crypto-js/js/pkg/for now.Status
The focus of the company has shifted a little bit. The priority is on NodeJS now. The idea is to merge this PR with its current state, then iterate on NodeJS in another second PR, and finish the Wasm+JS project with a third PR.
See #1016 for the next PR