Skip to content

Bacalhau project report 20221014

Jump to bottom
lukemarsden edited this page Oct 14, 2022 · 10 revisions

Amazing progress by the team this week, feels like we are really rolling!

FIL+ ⚙️

Kai has made significant progess discussing FIL+ integration with the community and now has a concrete design in his head, which will soon be written up! This has been a non-trivial undertaking to figure out a path forwards, so great work on that Kai!

Here's a sneak peak (clearer diagram coming soon 😂)!

image

Kai has also been a prolific bug-fixer this week, nailing:

We also have a bit of a mad idea: if the SPs are gonna get paid extra to run Bacalhau jobs via FIL+, what if we set up profit sharing between them and the users? Imagine if we were the cloud that paid you to run jobs that are good for humanity.

It would incentivise users to come up with jobs that get blessed. Which the SPs would like because it would generate more data. Users show up and do useful work on public data and get FIL magically deposited in their wallet. Let us know what you think!

Lotus now running with Devstack 🪷

Will has landed his first major contribution to the core codebase with a change that includes Lotus in the Bacalhau devstack. This will be extremely valuable as we deepen our Filecoin integration going forwards.

Inputs and outputs with WASM & WASI 🌅

WebAssembly System Interface (WASI) specifies a standard set of syscalls that act as a target for other programs. It is only in preview (and in fact is being rewritten) but is currently the state of the art in terms of running arbitrary code as WASM.

We now whole-heartedly support accepting WASI workloads. We now also support specifying arbitrary input and output volumes for WASM jobs as well. Files can be opened from the WASI interface and read or written as expected.

WASM jobs are therefore now capable of reading data from input volumes, writing logs via STDOUT and STDERR, returning an exit code, and writing arbitrary files to output volumes.

There is no support for listing directory contents yet - jobs have to know what files they are expecting. In a future commit, we could add support for specifying input arguments to the program to work around this. Full directory listing support will require upgrades to the wazero runtime.

With latest Bacalhau main checked out, you can now:

  1. Add inputs to IPFS:
ipfs add -r ~/bacalhau/testdata/wasm/csv/inputs
  1. Run WASM:
bacalhau wasm run ~/bacalhau/testdata/wasm/csv/main.wasm _start -v QmeVbdq3vx4P8LJMWmjPaYAzxjMg6nNNJdw9rYFrMvqqy1:inputs

where QmeVbdq3vx4P8LJMWmjPaYAzxjMg6nNNJdw9rYFrMvqqy1 is the CID from step 1.

Examples pipeline green again ✅

We had a lot of regressions on the Examples pipeline as a result of some (great!) UX changes. These are now green again thanks to a heroic effort from Phil!

He also fixed a URL downloading bug, sped up the tests, and landed docs for Stable Diffusion!

image

CoD in space!! 🚀 😁

There are also several new examples in the docs site, nice work Vedant and Phil!

Job Pipelining 👷

We're now referring to the DAG work as job pipelining. And Enrico has made excellent progress on the design doc!

What's more, we've settled on starting work to prototype Bacalhau-Airflow integration! I'm really excited about this as it will open Bacalhau up to a community of Airflow pipeline authors who -- with minimal effort -- will be able to port their existing pipelines to run on our open, distributed network!

Enrico is also hacking away at getting our OVH machines online, and has updated our Service Provider docs. If you're a Service Provider, interested in becoming a Compute Provider, please give them a try and give us feedback on #bacalhau on the Filecoin Slack!!

Reliability and scalability design docs 📖

Walid shipped these design docs:

What's next? ⏭️

Next week is our final push on UX polish and examples before the Lisbon CoD summit -- so it will be all hands on deck for that.

There might also be some cool new demos ready in time for that 😉

Clone this wiki locally