Task Server Protocol

[domenkozar]

Task Server Protocol

https://devenv.sh allows defining tasks in your favorite language using JSON-RPC
protocol and exposing them as an executable :

{
  task.serverProtocol = [ "myexecutable" ];
}

Listing tasks

Which would launch myexecutable /tmp/rando.sock and devenv would on startup immediately ask for a list of tasks:

{
  "jsonrpc": "2.0",
  "method": "initialize",
  "params": {},
  "id": 1
}

And the server responds:

{
  "jsonrpc": "2.0",
  "result": {
    "tasks": [{
      "name": "prefix:custom",
      "after": []
    }]
  },
  "id": 1
}

Running tasks

Then the client can ask the server to run a task:

{
  "jsonrpc": "2.0",
  "method": "run",
  "params": {
    "task": "prefix:name",
    "inputs": {},
    "outputs": {}
  },
  "id": 2
}

And server streams updates about stdout/stderr:

{
  "jsonrpc": "2.0",
  "method": "log",
  "params": {
    "task": "prefix:name",
    "line": "some text",
    "stderr": false,
    "time": "20240828T212611.11Z",
  }
}

Until the final response from the server with outputs and the final status of the task.

{
  "jsonrpc": "2.0",
  "result": {
    "task": "prefix:name",
    "outputs": {},
    "status": "succeeded"
  },
  "id": 2
}

Rust SDK

use task_server_sdk::{Task, TaskServerProtocl, cli::Args};
use clap::Parser;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let tasks = vec![
        Task {
            name: "myapp:task1".to_string(),
            after: vec![],
            exec: Box::new(async {
                println!("Executing custom task");
                Ok(())
            }) as Box<dyn Future<Output = anyhow::Result<()>> + Send + Unpin>),
        },
    ];

    let args = Args::parse();
    TaskServerProtocl::run(tasks, &args).await
}

Implementation

• Use reth-ipc for the Client using jsonrspee
• A test suite with a defined set of tasks that Server SDK should implement for the test suite to run against
• Task Server Protocol implementation

[bobvanderlinden]

I was a bit surprised by this. I thought the tasks were usually short-lived operations that just run one or more commands in a certain order and then just end.

The TSP seems to indicate this is a long lived process that can interactively start one or more tasks. It almost seems like the protocol could be used for the processes of devenv up?

That's pretty cool, I'm just wondering what the intention is.

Just to illustrate, the same could be done by myexecutable run task < json_file? Where myexecutable always lives as long as the tasks it's running.

That said, the methods/responses make sense to me. I just wasn't sure where the tsp would get its configuration from if that resides in devenv.nix?

Nitpick (feel free to ignore): The stderr being bool was a bit surprising, but the alternatives I thought up might not be better (io: stdout or even fileNo: 1).

[domenkozar]

@bobvanderlinden you're correct to spot the intention for tasks to be an implementation detail for processes. That way we get systemd-level dependency trees also for processes.

Just to illustrate, the same could be done by myexecutable run task < json_file? Where myexecutable always lives as long as the tasks it's running.

There are two issues here when I was thinking about the design:

• providing language specific interface via command line parsing is inflexible for future changes
• we need a way for myexecutable to report logging, etc at which point you're already introducing bidirectional channel

I just wasn't sure where the tsp would get its configuration from if that resides in devenv.nix?

it gets configuration from initialization phase that happens as soon as executable runs.

Nitpick (feel free to ignore): The stderr being bool was a bit surprising, but the alternatives I thought up might not be better (io: stdout or even fileNo: 1).

I had the same thought, the alternative would be line_stdout and line_stderr?

[domenkozar]

See #1471

[bennyandresen]

Very interested in this! I've built a PoC for Clojure (with a GraalVM target, for startup speed). Maybe this is a way to marry babashka tasks (common in the Clojure world) and making them available to devenv. So a "devenv up" would just work(tm) for Clojure devs.

[domenkozar]

@bennyandresen happy to code up the TSP if you'd like to take this forward and test!

[bennyandresen]

Yeah, very much so.

Just thinking out loud for Clojure/babashka integration, in case others are also interested in this.
We often use babashka as a task runner for Clojure projects, except the tasks or processes are written in Clojure.
But things like standing up a PostgresQL DB and filling it with initial data is done in other ways. That's where Just or in my case devenv comes in.

So in my mind, based on my PoC given this description one could use metadata inside a bb.edn (the babashka project file that makes babashka a task runner) so that they could be exported to devenv via the TSP.

As an example, a normal bb.edn file looks like this:

{:tasks
 {test
  {:doc "clojure -Srepro -M:poly test <args>"
   :task (apply poly "test" *command-line-args*)}

  test-all-verbose
  {:doc "bb test :all :verbose"
   :task (poly "test" :all :verbose)}

  outdated
  {:doc "clojure -Srepro -M:outdated"
   :task (clojure-repro "-M:outdated")}}}

you could then add ^:devenv or other metadata to a key like test-all-verbose (= ^:devenv test-all-verbose) which then makes it so it gets picked up by the TSP and it can be used further downstream.

My imagined use case would be to export the processes inside bb.edn that one needs in a devenv up, like the clojure backend process and other helps in combination with the devenv provided redis and postgres services. This could be achieved by merely setting a metadata inside bb.edn which doesn't affect babashka usage but is a pure value-add for devenv users.

Clojure metadata also allows to give parameters, so one could indicate whether or not one wants to export a task or a process to devenv.