AppTaskQueue

Executive Summary: Configures and deploys queues on top of RabbitMQ and Celery as needed for the Google App Engine TaskQueue API.

Why do we need it?

Google App Engine applications sometimes require doing background workers. The AppTaskQueue component allows for an application to enqueue a worker into RabbitMQ/Celery to be done asynchronously, either right away, or with a given ETA.

What does it do, and how does it do it?

It is a REST based service that receives requests from different application servers. Requests are sent in protocol buffer serialization. The primary request is "Add", where a new task is added to be run by the system. Queues can have certain parameters such as the number of tasks per minute, or a backoff period given a failure. An app's queue.yaml gives this information and it is then used to set up Celery workers with the correct configuration.

When does it do it?

When the application is first uploaded, the queue.yaml file is read and Celery workers are started for each queue. Whenever an application does a call to the TaskQueue API the service is called and enqueues the given tasks into the specified queue.

Where does it do it?

AppTaskQueue runs on any node which has a taskqueue role, whether it is a master or a slave.

Developer stuff

AppTaskQueue code is written in Python, although the start scripts for the REST server (taskqueue_server.py) and RabbitMQ are in Ruby. Celery processes are started for each new queue, and by default there is always at least one default queue for each application. The code can be found in appscale/AppTaskQueue. The high level web service is taskqueue_server.py and runs on port 64839. Flower is used to visualize the running of the tasks which can be accessed from the AppScale Dashboard.

Tests are written with test/unit and flexmock. They can be found in appscale/AppTaskQueue/test/unit. You can go into this directory and run "nosetests" to run just the tests for this component.