Scheduling Tasks

This page explains how to describe the tasks you want to schedule and the resource offers to match with those tasks, and then how to use the Fenzo task scheduler to assign tasks to hosts. Alternatively, you can use Fenzo task scheduling service to queue the tasks and receive callbacks for resource assignment. For details on this alternative, read the Task Queues section.

Contents:

1. Scheduling Tasks
   1. The Task Scheduling Loop
   2. Tasks and Offers
   3. The scheduleOnce Method
2. Managing Resource Offers

Scheduling Tasks

You schedule tasks by passing a list of tasks and a list of resource offers into your task scheduler (which may also take into account previously-unused resource offers that are still available).

The Task Scheduling Loop

This is the typical task scheduling loop that you perform with the Fenzo task scheduler:

1. You get the latest resource offers from Mesos.
2. You create the list of tasks you want to assign.
3. You call the Fenzo task scheduler to assign tasks to hosts.
   1. The task scheduler rejects any old, unused offers that have expired.
   2. The task scheduler combines new and old offers to determine total resource availability.
   3. The task scheduler assigns tasks to hosts.
   4. The task scheduler rejects any offers that remain unused and that have expired.
   5. The task scheduler (optionally) launches the autoscaling evaluator, asynchronously.
   6. The task scheduler returns a report on the task assignments.
4. You check this result for tasks that Fenzo was unable to assign and may need to be reattempted.
5. You tell Mesos to launch any tasks that Fenzo has successfully assigned to hosts.
6. You call Fenzo’s taskAssigner for each task that you launch.
7. If you choose to not launch the tasks that Fenzo assigned resources to from a host, you will have to either reject the resource offers in the assignment result for that host, or re-offer those resource offers to the task scheduler again. Otherwise, those resource offers would be "leaked out".
8. You return to the beginning and start this process again.

Tasks and Offers

Tasks are defined as TaskRequest objects, and offers as VirtualMachineLease objects. You need to create a set of each of these to pass into your task scheduler so that it can correctly assign tasks to offers. The following subsections show you how to create these objects.

The TaskRequest Object

You describe your task’s needs by creating an object that implements the TaskRequest interface. This interface has a variety of methods with which Fenzo can query your task to determine its resource needs, which constraints apply to it (see Constraints), and to which task group it belongs.

When you implement this interface for your task, you give Fenzo the insight into the task that it needs to schedule your task appropriately.

The VirtualMachineLease Object

You describe each Mesos resource offer by means of an object that implements the VirtualMachineLease interface. You can get such an object by passing the Offer object that represents that Mesos offer (and that you received from Mesos) into the VMLeaseObject constructor.

The scheduleOnce Method

The scheduleOnce() method accepts a list of those tasks you would like to assign (in the form of TaskRequest objects), and a list of new resource offers (in the form of VirtualMachineLease objects). It then attempts to assign those tasks to hosts, based on both those new offers it is passed, and any unused and unexpired offers from previous such calls.

The Fenzo scheduler keeps track of all of the resource offers it has been made aware of, so you should never pass this method an offer that you have previously passed to it (the method will fail with an IllegalStateException if you do). It also concatenates these offers when appropriate. For instance, if one offer says that a host has ports 9020–9280 available, and a second offer says that a host has ports 8000–9019 available, Fenzo will understand that the whole 8000–9280 range is available to a single task.

scheduleOnce() returns a SchedulingResult object. You can use this object to get a map of which tasks were assigned to which virtual machines, and then can use this assignment map to launch the assigned tasks via Mesos. Here is a simplified example of how you might do this (for a more complete example, see the sample framework):

/*
 * get the task assignments from the response returned from scheduleOnce():
 */
Map<String,VMAssignmentResult> resultMap = schedulingResult.getResultMap();
/*
 * loop through each assignment in this map of task assignments:
 */
for(VMAssignmentResult result: resultMap.values()) {
    /*
     * get the resource offers that have been assigned:
     */
    List<VirtualMachineLease> offersUsed = result.getLeasesUsed();
    /*
     * create a list of tasks to launch, suitable for Mesos:
     */
    List<Protos.TaskInfo> taskInfos = new ArrayList<>();
    /*
     * convert each task that scheduleOnce() assigned into a
     * Mesos-appropriate TaskInfo object and assign it to this list:
     */
    for(TaskAssignmentResult t: result.getTasksAssigned()) {
        taskInfos.add(convertToTaskInfo(offersUsed, t.getTaskId()));
        /*
         * don't forget to notify the Fenzo task assigner of each task you assign:
         */
        myTaskScheduler.getTaskAssigner().call(t.getRequest(), offersUsed.get(0).hostname());
    }
    /*
     * create a list of resource offers, suitable for Mesos
     */ 
    List<Protos.OfferID> offerIDs = new ArrayList<>();
    /*
     * add to this list each resource offer ID that scheduleOnce() assigned a task to
     */
    for(VirtualMachineLease l: offersUsed)
        offerIDs.add(l.getOffer().getId());
    /*
     * tell Mesos to launch the tasks that scheduleOnce() assigned
     */
    mesosSchedulerDriver.launchTasks(offerIDs, taskInfos);
}

You can also use the list of task assignment failures that is available through the SchedulingResult object to make an additional attempt to launch those tasks on your next scheduling loop.

getTaskAssigner() and getTaskUnassigner()

For each task you assign and launch, you must call your task scheduler’s getTaskAssigner().call() method in order to notify Fenzo that the task has actually been assigned to a host. Pass two arguments to this call method: the TaskRequest object for the task assigned and the hostname.

Note that you may not call the task assigner action concurrently with scheduleOnce(). If you do so, the task assigner action will throw an IllegalStateException.

In addition, in your framework’s task completion callback that you supply to Mesos, you must call your task scheduler’s getTaskUnAssigner().call() method to notify Fenzo that the task is no longer assigned. Pass two String arguments to this call method: the taskId and the hostname.

This method is safe to be called concurrently with other calls to TaskScheduler. The tasks to be unassigned are stored internally and actually unassigned at the beginning of the next scheduling iteration, that is, the next time scheduleOnce() is called.

Some scheduling optimizers need to know not only which tasks are waiting to be scheduled and which hosts have resource offers available, but also which tasks have previously been assigned and are currently running on hosts. These two methods help Fenzo provide this information to these scheduling optimizers.

Managing Resource Offers

The task scheduler maintains a list of resource offers that you have passed it by means of the scheduleOnce() method and that remain unused. It will automatically reject any offers that remain unused longer than the expiration time that you established when you built the task scheduler. By default, Fenzo limits the number of offers it rejects. This default and the limits can be controlled when building the scheduler.

method	description
withLeaseOfferExpirySecs(expirySecs)	Call this method to set the number of seconds after which a resource offer is considered expired.
withMaxOffersToReject(max)	By default, Fenzo rejects a limited (4) number of offers every time period of expiry seconds. Use this method to set the limit to a different number.
withRejectAllExpiredOffers()	Use this convenient method to reject all offers upon expiry instead of using a limit. Using this effectively cancels any limit set by the above method.

But you may also force certain of these offers (or all of them) to expire by calling one of the following TaskScheduler methods:

method	description
expireAllLeases()	Call this method to instruct the task scheduler to reject all of the unused offers it is currently holding. This is useful to call, for example, when you get a notification from Mesos driver that a connection to Mesos master has been established, or reestablished.
expireAllLeases(hostname) or expireAllLeasesByVMId(id)	Call one of these methods to instruct the task scheduler to reject all of the unused offers it is currently holding that concern resources offered by a particular VM.
expireLease(offerID)	Call this method to instruct the task scheduler to reject a particular offer.