Talking to dora nodes on different machines

[Geoffrey-Hooton]

Hi!

Sorry if this question has been asked before; I'm still relatively new to using Dora.

I'm currently running three Dora nodes on my local machine, but I'm hoping to run another node (meant for plotting) on a separate machine. I saw issues #459 and #535 and am still a bit confused about whether there is currently a way to hardcode the address of a separate machine for communication between nodes.

My question is:

• Is there a way to hardcode communication to a different machine, or is that still in development?
• If there is a way, are there any examples (preferably in Rust) that you could point me to?

Thanks in advance for your help!

[phil-opp]

Dora currently works like this:

• There is one central coordinator instance. You can start it using dora coordinator. Choose any machine that you like.
• Each machine runs a separate daemon instance. You can start it using dora daemon --machine-id <id>.
  • The <id> must be a unique identifier of the daemon instance. The coordinator will complain if multiple deamons with the same ID try to connect to it.
  • (If you don't supply a --machine-id argument, the <id> defaults to the empty string.)
  • If the daemon does not run on the same machine as the coordinator, you need to supply a --coordinator-addr argument to dora daemon.
• In your dataflow.yml file, you can add a _unstable_deploy.machine key to each node to specify the ID of the machine that the node should run on. If not given, this ID defaults to the empty string.
  • Example:

      nodes:
        - id: first-node
          _unstable_deploy:
            machine: A
          path: /path/to/node
          inputs:
            tick: dora/timer/millis/10
          outputs:
            - random
        - id: second-sink
          _unstable_deploy:
            machine: B
          path: /path/to/second/node
          inputs:
            some-input: first-node/random
        ```

• Note that relative path fields are not allowed right now for nodes that run on remote machines (as there is not a clear base directory).

Does that make sense? Please let me know if something is unclear!

[Geoffrey-Hooton]

That's very helpful. Thanks for the quick response!

There are a few things that I would like to ask for clarification about:

1. Should I use the IP address of the machine running the coordinator as the --coordinator-addr argument? Can you provide an example of what I should use there?
2. How do I specify in the dataflow.yml file the path to the source file of the node running on the second machine if all the binaries are split up across different machines?
3. Would both machines have their own Dora projects and dataflow.yml files?
4. Do I need to run the dora start command on both machines, or only on the one with the coordinator?

Thanks again!

[phil-opp]

See dora daemon --help for details on the argument format. The --coordinator-addr argument expects an address in IP:PORT format. The default port number that the coordinator listens on is 53290 (this is printed by dora coordinator).

2. How do I specify in the dataflow.yml file the path to the source file of the node running on the second machine if all the binaries are split up across different machines?

You need to specify the absolute path of the file on the remote machine. We don't have a deploy feature yet, so you need to copy the executable to the remote machine yourself.

3. Would both machines have their own Dora projects and dataflow.yml files?

Both machines need to use the same dora version to ensure that they're compatible (i.e. dora --version needs to print the same version both machines). The easiest way to install dora is cargo install dora-cli.

The dataflow.yml file only needs to be available on the machine where you execute the dora start command.

4. Do I need to run the dora start command on both machines, or only on the one with the coordinator?

You can run it on any machine you like, even a completely unrelated one. You just need to tell it how to connect to the coordinator if you're running it on a different machine. You can do using dora start --coordinator-addr <IP> dataflow.yml. If no --coordinator-addr is given, it defaults to 127.0.0.1 (i.e. it expects that the coordinator runs on the same machine).

[Geoffrey-Hooton]

That's great! Thanks for the help.

[Geoffrey-Hooton]

Hi again @phil-opp,

I'm currently trying to implement this, and I've managed to get the two computers both connected to the same coordinator, however it seems that when I try to run the dora project, it says that the dataflow could not find the source path for my python file on the remote computer.

Here is how I've formatted the .yml file:

nodes:
  - id: local-node
    _unstable_deploy:
      machine: A
    custom:
      build: cargo build -p local-node
      source: target/debug/local-node
      inputs:
        tick: dora/timer/millis/1
      outputs:
        - data

  - id: remote-node
    _unstable_deploy:
      machine: B
    path: /home/<user>/<project_name>/remote-node/remote-node.py
    inputs:
        input: local-node/data

Am I missing something obvious about the formatting for the path? It's worth noting that I am running machine B on WSL2 so that may be causing some problems with the path. Any suggestions you might have would be greatly appreciated.

Thanks!

[phil-opp]

Dora does not do any deployment yet, so you need to copy the file to the remote machine manually.

If you did that already and still get the same error, there might be some invalid check in our CLI. Could you paste the full error here in that case to help us debugging?

[Geoffrey-Hooton]

I did have the file on the remote machine already (I copied the whole dora project over to the second machine at some point so it was already there). Here is the full error that I'm getting:

>> dora start dataflow.yml --name hobie-sensor-nodes

[ERROR]
Could not validate yaml: Dataflow could not be validated.: Could not find source path `/home/gmch/PRL/hobie-sensor-nodes/imu-plotter/imu-plotter.py`: Could not find source path /home/gmch/PRL/hobie-sensor-nodes/imu-plotter/imu-plotter.py

Would I have to include the IP address of the remote machine or something similar within the path?

Thanks again!