On-demand kubernetes workers

[fsaintjacques]

I would like to see the support for allowing tasks to be executed by on-demand transient workers. The typical use case are tasks that require specialized/costly resources, e.g. GPU, high mem, etc. Having a long running worker waiting for work is cost prohibitive and where elastic scale can do wonder here (kubernetes clusters with scale down to 0).

An example workflow/implementation:

• Enqueue a task in a dedicated queue
• Said queue is configured to spawn a worker (a Kubernetes' Jobs is probably the best bet)
• The binary is exec-ed with something like ./my-binary worker $task-id
• The binary is responsible for marking the task as done / heartbeat
• Kubernetes is responsible for retries (i.e. aligned in the CRD definition)

I can somewhat already hack this with with 2 queues, one that dispatches and creates the CRD, and the kubernetes-worker would drain from the second queue with custom filter to listen only for a give task-id. But all of this is tedious and filled with edge cases I don't want to worry about.

I'd want this to be supported natively, i.e. only one logical task in the UI but running in "detached" mode.

[brandur]

So are you suggesting something like River would plug into the Kubernetes API (or something like that) to start jobs?

[fsaintjacques]

Precisely, or at least having the support for "hand-off" workers that allows to build on top of Kubernetes Jobs API.

[brandur]

Okay, good to know.

Hm, this feels a little beyond project scope. I suppose something like a hand off hook isn't beyond the realm of possibility, but it also feels like that could easily just be user-specific code that goes right into a worker body. i.e. A single client with a very high concurrency setting could be operating somewhere, and as it starts working jobs it hands them off to Kubernetes with specific instructions on the work that should be done and waits for the Kubes node to start up, work, then finish before returning (and thereby completing the job).

[fsaintjacques]

I'm conveying one of our biggest pain point with gue, which will likely translate to river if we try to switch.

[fsaintjacques]

But yes, your solution is one manageable, especially that river does not hold transactions like gue.