Task pipeline

[davidolrik]

Is it possible to create a pipeline of tasks?

It would be really nice to be able to enqueue a group of tasks and have them depend on each other, and then have them executed in sequence.

Such pipelines should have support for both single and multiple parents, so that all the things than can be run in parallel will do so, and the sequences specified will be respected.

Something like this:

flowchart LR
A[Task A]
B[Task B]
C[Task C]
D[Task D]
E[Task E]
F[Task F]
G[Task G]
H[Task H]

A --> B
A --> C
A --> D

B-->E
C-->E

D-->F

E-->G
F-->H

Loading

• When Task A is done, B, C & D are available run in parallel
• When both B & C is done, E is available to run
• When E is done, G is available to run
• When D is done, F is available to run
• When F is done, H is available to run

It would be nice to be able to specify the dependencies upfront, before the "flow" is started,
so you can "see" the structure of the flow in the database.

[brandur]

@davidolrik The short answer is "no", this is not currently supported by River. However, we do intend to support this eventually, although it's one of the larger features that's on the backlog, so we don't have a definite timeline yet.

I know that @bgentry's thought a lot more about this one than me, so I'll let him weigh in with any other thoughts/details.

[bgentry]

@davidolrik thanks for opening this up and for the great mermaid diagram 😄 It's absolutely something we want to build and I have a pretty good idea of how to make that happen, but there are a few underlying pieces we need to work out first.

Are there any other similar systems you've used in the past that have this kind of functionality, or any particular features you're interested in beyond the dependency graph? Any specific use cases you have in mind?

[davidolrik]

I have used Minion on several projects with great success.

I have also used Celery, procrastinate and a few commercial offerings but most of them lack the simplicity and ease of use that Minion provides.

Right now I'm now looking for a similar "Task Queue" in Go, and it seems that River has most of what I need - including the simplicity and ease of use.

Minion also has a feature it calls "Guards" which is an advanced version of a task lock, like the one River has for making sure only one instance of a given task can run at a single time.
With Guards you can specify a higher concurrency limit, which makes it super useful for rate limiting expensive operations. E.g. only allow 5 instances of this task type to run at any given time.

My primary use case is offloading expensive operations from http APIs, and having them split up into "atomic" idempotent operations, so when something fails, it is easy to monitor, debug, fix and re-queue.

If you introduce pipelining, then the possible use cases will be virtually endless 😃 business workflows, Data Science flows like ETLs and any other kind of long running or fragile task.

I am of the opinion that every backend API, that wants to do anything that either takes a long time, or has the potential to fail needs a simple reliable Task Queue.

[davidolrik]

Just thought about another feature I really like in Minion: Task notes.

Task notes is just a jsonb for data the tasks generates during its run - Like number of refries, but for arbitrary data.

[bgentry]

Hi @davidolrik, just wanted to make you aware that River Pro was just launched. It includes a fast and reliable workflow engine integrated into River, as well as an interface built into River UI.

There's quite a bit more coming (I've been kicking around your task notes idea among others) but I hope what's already here is a great start. Definitely check out the demo too!