Status Update: 2022-05-25

[haixuanTao]

In this discussion, we describe our recent progress and the main topics that we would like to discuss in today's meeting. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[haixuanTao]

Refactoring of Pylot

• Remove previous latency metrics that are now handled by opentelemetry.
• Simplify Python code.

[haixuanTao]

Adding Opentelemetry Metrics with System Metrics

• Adding Opentelemetry metrics (a module different of Opentelemetry tracing) enable us to export distributed metrics of all kind target verious backend: Influxdb, Prometheus...
• The metric export is implemented at the node-level which is pushing to the backend on a separate process at intervals.

[haixuanTao]

Opening an Opentelemetry-rust-system-metrics repo

See: https://github.com/haixuanTao/opentelemetry-rust-system-metrics

Opentelemetry-rust-system-metrics allow process-level system metrics export into opentelemetry.

Observed metrics are:

• CPU usage.
• Memory and Virtual memory usage.
• disk usage (read and write).

It is based on sysinfo which is OS-agnostic.

I'm currently pushing the addition of network usage monitoring within sysinfo.
See: GuillaumeGomez/sysinfo#740

This is will monitor the cost/gain of serialization/deserialization, shared-memory, zenoh, ...

But it might not be easy, and to make it efficient a novel approach using ebpf and aya might be interesting.

[haixuanTao]

Monitoring Otel metrics via InfluxDB

• Connecting Opentelemetry metrics with InfluxDB using Telegraph.
• Test everything on Pylot, and integrating Pylot special InfluxDB node.

[haixuanTao]

Discussion:
Grafana?
-> Looking closer at the Licensing GPL. It is possible to integrate GPL, it should be investigated.

[haixuanTao]

https://opensource.google/documentation/reference/using/agpl-policy/

Avoid AGPL

[haixuanTao]

Rust Runtime

We created an initial prototype version of a dora runtime, which is able to run "light-weight" operators. It currently supports the following:

• link operators compiled as shared libraries (*.so files)
• provides input through a C-compatible ABI (implemented by the operator):

extern "C" fn dora_on_input(
    id_start: *const u8,
    id_len: usize,
    data_start: *const u8,
    data_len: usize,
    output_fn_raw: OutputFnRaw,
    output_context: *const c_void,
) -> isize

• Sending output is possible through the supplied OutputFnRaw (implemented in the runtime). It has the following API/ABI:

type OutputFnRaw = unsafe extern "C" fn(
    id_start: *const u8,
    id_len: usize,
    data_start: *const u8,
    data_len: usize,
    output_context: *const c_void,
) -> isize;

• Operators can communicate with custom nodes in both directions

Next Steps

There are still some things missing:

• keeping state between function calls
  • We plan to let operators define an init function that returns a context (fat?) pointer, which can point to operator-defined state
  • This context pointer is passed in dora_on_input, which allows the operator to access the state
  • A high-level API could map this context pointer to a struct instance
• creating an abstraction for typed I/O

[haixuanTao]

Meeting: Do Dora follow the Dynamic deadline design?

[haixuanTao]

• First step will be static deadline with a sound architecture
• Dynamic deadline will come with a policy, but we normally should have all the information necessary to implement it, without having to rearchitecture everything if we want to implement it.

[heyong4725]

This discussion about tokio async performance might be relevant to us in future, when we deal with performance issues.
tokio-rs/tokio#4703