Status Update: 2022-04-13

[phil-opp]

In this discussion, we describe our recent progress and the main topics that we would like to discuss in today's meeting. We use a separate post for each topic to keep the discussion focused. Feel free to create new posts for topics that are not mentioned yet.

[phil-opp]

Design Overview

Our current design draft looks like this:

The coordinator is responsible for reading the dataflow from a YAML file, verifying it, and deploying the operators to the specified or automatically determined machines. It monitors the operator's health and implements high level cluster management functionality. For example, we could implement automatic scaling for cloud nodes or operator replication and restarts. The coordinator can be controlled through a command line program (CLI).

Dora provides its main functionality through an API, ideally for multiple languages. Operators communicate through a shared communication layer, which will be zenoh in the first iteration. However, we will design dora in a way that makes it easy to use different communication layers in the future. All communication layers should be robust against disconnections, so operators should be able to keep running even if they lose the connection to the coordinator.

Light-weight and Heavy-weight Operators

Dora supports two types of operators: heavy-weight and light-weight operators. (The names are only placeholders for now.)

Each heavy-weight operator is a separate, isolated process that communicates with other operators through the dora library. This operator type is for example useful for security-critical operators (isolation is important), resource-heavy operators (resources can be pinned to processes), or operators which need maximal control (e.g. for complex input rules). However, running each operator has a separate executable also has some drawbacks. For example, the isolation between operators comes with some overhead and not all features of dora are usable (e.g. no real-time support). For this reason, dora also supports so-called light-weight operators, organized into operators groups.

Light-weight operators are cooperative, library-based components that are executed by a dora runtime process. Multiple formats are supported for light-weight operators, for example they can be implemented as a Python object, a WebAssembly module, or a traditional C-compatible shared library. The dora runtime links multiple light-weight operators (of the same operator group) together and executes them from a single, multi-threaded process. The full set of dora's features are available for light-weight operators, for example priority scheduling or native deadline support.

[heyong4725]

I think we can unify all long running processes/daemons, which include: dora-rs runtime (for light-weight operators), heavy-weight operators, control plane processes (e.g. YAML processor, cluster management, monitoring etc). In future, all of these daemons can be packaged into containers and managed by cloud native K8S.

[phil-opp]

I updated the design docs: https://github.com/futurewei-tech/dora-rs/blob/main/design/overview.md

We now use the term "node" instead of "heavy-weight operator" and "operator" instead of "light-weight operator". I also updated the design diagram to include dora operators and the corresponding runtime node, and rewrote the text to include more details.

[haixuanTao]

Thanks! I feel like for WASM and Python, executing them in a custom node or a Dora Runtime Node is not going to make that much of a difference. (in term of efficiency, excluding IO)
I feel that custom nodes are more geared toward Rust-written nodes or C-written nodes.

What do you think?

[phil-opp]

Agreed! I pushed 2bfdd68 to list them only as "maybe".

[phil-opp]

Dataflow Specification

Dataflows are specified through a YAML file. This section presents our current draft for the file format. It only includes basic functionality for now, we will extend it later when we introduce more advanced features.

Run Operation

For heavy-weight operators (see #22 (comment)), we plan to specify the executable name and arguments like a normal shell operation:

operators:
  - run: path/to/executable arg1 arg2
    env:
      - ENVIRONMENT_VARIABLE_1: true
    working-directory: some/path
    inputs: # ...
    outputs: # ...

This way, we stay close to popular continuous integration (CI) services such as GitHub Actions, so the format should already be familiar to many people. We haven't decided yet on a syntax for light-weight operators.

Dataflow Graph

Each operator has a separate namespace for its outputs. To refer to outputs, the <operator>/<output> syntax is used. This way, there are no name conflicts between operators.

Input operands are specified using the <name>: <operator>/<output> syntax, where <data> is the internal name that should be used for the operand. The main advantage of this name mapping is that the same operator executable can be reused multiple times on different input.

Example

operators:
  # sources (only outputs)
  - id: timer
    name: Clock timer    # optional, human-readable name
    description: Send the time.
    run: path/to/timer --interval 0.5
    outputs:
      - time
  - id: camera-uuid-1
    name: Front camera
    run: camera --id camera_id
    outputs:
      - image
      - metadata
  - id: camera-uuid-2
    name: Front camera
    run: camera --id camera_id
    outputs:
      - image
      - metadata
      
  # actions (inputs and outputs)
  - id: timestamp
    description: Add a watermark on the camera.
    run: path/to/timestamp
    inputs:
      timestamp: timer/time
      image: camera-uuid-1/image
    outputs:
      - image # with timestamp watermark

  # sinks (only inputs)
  - id: logger
    description: Sink the data into the logger.
    run: path/to/logger --target-dir /foo/bar
    inputs:
      image: timestamp/image
      camera: camera-uuid-2/metadata

Visualized

flowchart TB
  timer[\timer/]
  camera[\camera/]
  timestamp
  logger[/logger\]
  camera -- image as pic --> timestamp
  timer -- time as timestamp --> timestamp
  timestamp -- image as data --> logger
  timer -- time --> logger

Loading

Integration with ROS 1/2

To integrate dora-rs operators with ROS1 or ROS2 operators, we plan to provide special bridge operators. These operators act as a sink in one dataflow framework and push all messages to a different dataflow framework, where they act as source.

For example, we plan to provide a to_ros_2 operator, which takes a single data input, which is then published to a specified ROS 2 dataflow.

[heyong4725]

On YAML Spec

Regarding to the YAML specification, we can also study Nvidia's Isaac robotic platform. It uses JSON (which is equivalent to YAML) to specify the data graph: https://docs.nvidia.com/isaac/isaac/doc/engine/alice.html#
Here the graph has multiple nodes, and each node may contain multiple components, and components are connected thru edges:

• Isaac Graph --> dora dataflow (or graph)
• Isaac node --> dora heavy-weight operator, dora may call it dora node as well, with different kind:
  • dora-runtime: this type of dora node has dora runtime, which manages multiple dora operators
  • control node: this type of dora node is control plane node
  • dora-program: this type of dora node is heavy-weight operator, a long running user process
• Isaac component --> dora light-weight operator (or dora operators), with different kind:
  • source: only produce the data; single output
  • normal: compute data, with multiple inputs and outputs
  • sink: only consume the data; single input
• Isaac edge --> dora edge connecting dora operators

In Isaac, Components are the basic building blocks of a robotics application. Components are developed thru the code template called codelet, which in the case of Isaac is derived from C++ class that developers will need to provide class member functions, such as tick, start etc. More details at: https://docs.nvidia.com/isaac/isaac/doc/engine/components.html

[heyong4725]

Isaac Components (codelet template) run in one of the three following ways:

• Tick periodically: The tick function is executed regularly after a fixed time period. A typical example is a controller which ticks 100 times a second to send control commands to hardware.
• Tick on message: The tick function is executed whenever a new message is received. A typical example is an image processing algorithm which computes certain information on every new camera image which is captured.
• Tick blocking: The tick function is executed immediately again after it has finished. A typical example is a hardware driver which reads on a socket in blocking mode. Blocking is normally used for hardware interaction.

[phil-opp]

• Isaac node --> dora heavy-weight operator, dora may call it dora node as well, with different kind:
  • dora-runtime: this type of dora node has dora runtime, which manages multiple dora operators
  • control node: this type of dora node is control plane node
  • dora-program: this type of dora node is heavy-weight operator, a long running user process

Sounds good to me! I like the "node"/"operator" distinction and that the dora-runtime is just another node. I'm not not sure if we want to specify the control node(s?) explictly in the YAML file, or if we want to manage that somewhere else. But we can decide this later, when we build the first prototype of the control plane.

different kind:

• source: only produce the data; single output
• normal: compute data, with multiple inputs and outputs
• sink: only consume the data; single input

Do we want to set these kinds explicitly, or do we want to derive them from the inputs/outputs? I think the latter would be more flexible. Also, what is the reason for only allowing a single output per source and a single input per sink?

Isaac edge --> dora edge connecting dora operators

Looks like Isaac specifies the edges separately from the components. The format we proposed above specifies the connections directly on the operators, in an inputs table. Which format do you prefer? I personally like our inputs table more because it makes adding new operators easier and makes sure that we don't forget to link an input.

[phil-opp]

Pylot Advancement

Added

CARLA bridge

• Spawn other vehicles representing traffic. Necessary for evaluating planning
• Spawn pedestrians. Necessary for evaluating planning
• Attach a Depth Camera. Necessary for locating obstacles
• Send position information. Necessary for planning using a map

Detection

• Port perfect obstacle operator. for faster development and test
• Port obstacle location operator. Necessary for planning to avoid obstacle

Planning

• (In progress) Port route planning operator. Planning is the most sensitive element to latency and accuracy of pylot. It is going to be the key element of evaluation of DORA.

Visualisation

• Visualize waypoints from camera. to evaluate the planning

Deployment

• Packaged Demo into a Docker. many Python depencencies are hard to install

Those implementation are currently committed into #18.

Changed

• Replaced array messages with pickle messages format as Pylot embed lots of logic within its messages. This logic is hard to decouple at this stage.
• Replaced deep learning based object detection with perfect object detection from simulator for better developper experience and smoothing out testing.

Fixed

• Thread-unsafe deque operation of Pylot Planning using Mutex, meaning that for thread-unsafe Pylot operator, Pylot is almost running single-threaded.

[heyong4725]

State Management

During our status meeting, we have lots of discussion around state management, and also stateless and stateful operators.

Operator state basically is its local running context (e.g. local variables). For stateless operator (we can view operator as a function f ), state is irrelevant, y = f(x); for stateful operator, state needs to be part of operator function, y=f(x, s).

For stateless operator, y = f(x), the same input x always produce the same output y. Since there is no state, there is no state management. This removes lot of complexity around state management, which is also why majority of the commercial FaaS service is stateless serverless functions.

For stateful operator, y = f(x, s), the same input x may not produce the same output y, which also depends on operator state (its context). In dora-rs, we have a few cases around state management:

• For lightweigth operator, the dora-rs runtime will need to management the state:
  • between operator invokations, dora-rs runtime will need to pass the operator state (context) to the user code (e.g. operator run(context, inputs,...) )
  • for operator fault tolerance/recovery, dora-rs runtime will need to store the state in some storage (e.g. K/V store) such that when operator recovers, its state is recovered as well
• For heavyweight operator, dora-rs does not provide runtime for it, it is upto the user code to manage its state, for example, fault-tolerance and recovery

Another key concept for state management is state sharing, e.g. how to share the state (similar to global varibles) among operators, normally there are three alternatives:

1. Message passing. The state information is passed thru message only. This is normally used in share-nothing architecture, for example actor model.
2. Shared memory. The state information is stored in some kind of global storage that every operators have access to. In dora-rs, we may utilize anna-rs K/V store for this purpose.
3. Function chaining: y = g(f(x1),x2), operator f's state is also its output, which is part of operator g's inputs

Hopefully this short, high level description clarifies many confusion around state management.

[haixuanTao]

I'm wondering if we should have:
For stateful operator, (y, s') = f(x, s) as we will explicitly tell what is going to be saved within the state.

[heyong4725]

Yes, the s' should be output as well, as input x will cause state changes.