Logging in AppScale

Introduction

Logging is one of those things that programmers take for granted. We rely on logging to bring important debugging and error info to the forefront, and that's no different in web applications. One particularly nice thing about working at the platform-as-a-service layer is that it can control your runtime, allowing it to automatically collect your logs and display them to you. In AppScale, our upcoming 1.7.0 release takes this a step further, allowing you to retrieve both application-level logs and system-level logs via both a web or command-line interface. This post details how AppScale implements this support for arbitrary Google App Engine applications, what the interfaces look like that users interact with, and how other services can leverage this support.

Storing and Querying Logs via the AppDashboard

Let's start off by looking at how we can store and retrieve logs. Ideally, we need the following:

• A replicated database, with the ability to query based on the name of the service that stores logs or the hostname of the machine they came from.
• A caching layer in front of the database, to speed up repeated accesses to this data.
• A simple web programming framework, to enable users and applications to access the data.
• Authentication, to ensure that only the right users can see the data.

By no small coincidence, these are all things that the Google App Engine APIs offer! We can use NDB to store and retrieve our logs, which automatically leverages the the Memcache API to speed up accessing our log data. Programming Python 2.7 apps on Google App Engine is a breeze with webapp2, and we can use the Google App Engine Users API if we need authentication when viewing logs. We thus use webapp2 to expose a single route in our AppDashboard, at "/logs/upload", that accepts POST requests to store log data. That code performs the following task:

boo

Logging in Python App Engine apps

Logging in Java App Engine apps

Logging in the AppController

Future Work

Conclusion