Dynamic Job Parallelism and Resource Scaling Based on Backlog Metrics

[woehrl01]

What would you like to be added:

We would like to propose a new feature in Kueue that enables dynamic scaling of job parallelism and resource allocation (CPU, RAM, and pods) based on job backlog metrics and predefined formulas.

Idea: This feature would introduce a custom resource definition (CRD) that allows users to define scaling formulas and thresholds, which dynamically adjust the maximum parallelism and resource limits, similar to KEDA or HPA. A generic approach could be the exposing of the /scale subresource to have a generic interface.

Why is this needed:

Currently, we are processing around 4.5 million jobs per day, and managing resource usage and costs is critical. There is a need for a mechanism that can dynamically limit or expand the maximum parallelism of jobs based on real-time backlog conditions. This would help ensure that jobs are processed efficiently without overcommitting resources or incurring unnecessary costs.

By introducing a formula-based approach to flavor resources, we can achieve a more granular and responsive system. For example, the system could increase the max CPU or RAM allocation as the admission backlog grows, ensuring that delays are minimized during high-load periods while conserving resources during low-demand times. This functionality is crucial for maintaining both performance and cost-effectiveness in large-scale Kubernetes environments.

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kannon92]

Reading the ask, I’m not entirely sure Kueue is the right place for this.

It sounds like you want metrics to influence elastic job scaling. AFAIK Kueue would help admit jobs based on dynamic scaling but I think the controller that looks at metrics and patches elastic jobs would probably be a separate CRD. I would think that this CRD would be separate from Kueue as it seems you want HPA at the job level.

[kannon92]

I'll leave the final decision on scope to @tenzen-y or @alculquicondor.

[alculquicondor]

similar to KEDA or HPA

Why not just use KEDA or HPA?

I don't think Kueue is the right component to decide that.

Still there are two things that we need to do in Kueue to improve the experience:

• Support job resizing, which was started but not completed.
• Add any missing metrics about the length of the queues that can be fed into the external scaler.

And in Kubernetes:

• Support the /scale sub resource for jobs (if this is the API you are targeting).

[mimowo]

FYI the request to support dynamically scaled Jobs in Kueue: #77, it already has a KEP: https://github.com/kubernetes-sigs/kueue/tree/main/keps/77-dynamically-sized-jobs

[tenzen-y]

FYI the request to support dynamically scaled Jobs in Kueue: #77, it already has a KEP: https://github.com/kubernetes-sigs/kueue/tree/main/keps/77-dynamically-sized-jobs

Yes, that's right. After we implement the feature, we may be able to use DynamicJob + Keda.