Task Queues

Fenzo now supports task queues and introduces a scheduling service that iteratively loops over the queue to launch them, repeatedly trying to assign resources to pending tasks. The scheduling service can be customized using its builder.

Existing use of Fenzo's TaskScheduler is unaffected. That is, all existing users of Fenzo are unaffected. Using the new task queues will, however, require one to replace use of TaskScheduler for scheduling to using the service summarized below.

While we work on a more comprehensive documentation, here’s a brief overview.

True to its customizable behavior, Fenzo allows multiple implementations of a queue. At this time Fenzo provides a tiered queue implementation. It is possible to develop other implementations of a queue, such as a simple FIFO queue, by implementing the interface TaskQueue. See TieredQueue as an example.

A tiered queue divides the queues into multiple tiers, usually a handful of them. Each tier is then divided into multiple buckets. There can be a large number of buckets in a tier.

Tiers represent coarse grain priority. Tasks in buckets of a higher tier are considered for resource assignment before tasks in buckets of lower tiers. Tiers are numbered 0 to N-1 for a queue of N tiers, with 0 being the highest priority tier and N-1 being the lowest priority tier.

Within a tier, buckets are kept in a sorted order based on dominant resource usage from all of the bucket’s tasks. Effectively, the bucket with the lowest usage is considered for resource assignment before other buckets.

Within a bucket, tasks are added in a FIFO manner. A task is added to a bucket based on the property it provides in QueuableTask definition.

The following diagram shows a 2-tier queue with each tier having 4 buckets.

          +------------+-------+---+-----+
   tier 0 |  Bucket A  |  B    | C |  D  |
          +------------+-------+---+-----+

          +-----+----------+----+--------+
   tier 1 |  E  |     F    |  G |   H    |
          +-----+----------+----+--------+

The new scheduling service, TaskSchedulingService, simplifies the use of Fenzo. Frameworks can now give tasks to Fenzo once, instead of repeatedly calling TaskScheduler.scheduleOnce(). The service can be called concurrently - new tasks can be added to the queue and new mesos resource offers can be added to TaskSchedulingService at any time, concurrently with any ongoing scheduling iterations. Possible upcoming changes

We summarize a few possible upcoming changes here for the purposes of sharing the thought process on how we see these queues being used in the longer term, not necessarily as a confirmed list of enhancements at this time.

• Ordering of buckets in a tier to be based on configured weights for buckets. For example, some buckets may be given higher weights to support a higher amount of resources used in relation to other buckets.
• Finer grain entities within buckets to support hierarchical weights for sub-grouping of queues.
• Task preemptions in order to launch tasks in a higher tier, if the cluster is running at capacity.
• Task preemptions in order to balance the usage across buckets of a tier.