mainevent

mainevent provides a suite of tools to gain insight into log files.

• Collect, parse and store updates from local and SSH-accessible log files.
• Easily write new parser modules for any file format.
• Searchable timeline with optional real-time updates.
• Dashboard analytics generated by MongoDB MapReduce, Redis and jqPlot.
• Write custom Pub/Sub listeners to real-time updates.
• more ...

Use Cases

• Search development/production environment logs which span a variety of formats.
• Produce dashboard data and graphs specific to your insight needs.
• Filter and replicate real-time updates into additional systems for specialized processing or alerting.

How To

Create a new parser for an unsupported log format

Each parser lives in a separate directory under app/parsers/ which holds its JS, CSS, templates and tests.

All parser classes extend a base and only need to implement a small number of interfaces.

• Required
  • parse(log): Accepts a log line string, returns an object of parsed fields.
• Optional
  • buildTemplateContext(template, log): Modify the context object sent to dust.js based on the type of template. Currently there are two types: preview and event. See the extension example to see them in action.
  • buildPreviewText(log): Build the preview string manually (rather than using on a preview template).
  • extractTime(log): The default implementation will detect millisecond/second timestamps and Date.parse()-able strings in log.time values. For incompatible formats, define this function to extract the millisecond timestamp manually.
• Utilities
  • namedCapture(subject, regex): Wrapper around XRegExp named capture expression handling.
  • candidateCapture(subject, candidates): Wrapper around namedCapture that lets you define multiple potential patterns and the first match wins.

See app/parsers/prototype.js for more interface details.

Extending the base class is a simple one-call process via a backbone.js-like extend() function. See the extension example for a working implementation and screenshots of the output content. Or browse any of the modules under app/parsers/.

Create a Pub/Sub listener for log updates

1. Create a module that exports an on(logs) function that receives an array of one or more log objects.

2. Perform any non-native tasks you need.

3. Find this config/app.js section and add the location of your listener module to the subscribers list:

{
  // ...
  mongodb: {
    // ...
    listeners: [
      {
        // Customize this event but do not remove.
        event: 'InsertLog',
        enabled: true,
        subscribers: [
          'app/modules/redis/InsertLogPubSub.js'
        ],
      }
    ],
    // ...
  },
  // ...
}

See app/modules/redis/InsertLogPubSub.js for a working example.

Display a real-time update with Web Notifications

• Enable Desktop Notifications via Timeline drop-down menu.
• Required event attribute(s):
  • WebNotifyApiBody
    • OR WebNotifyApiBodyAttr to specify the attribute to use instead.
• Optional event attribute(s):
  • WebNotifyApiTitle
    • OR WebNotifyApiTitleAttr to specify the attribute to use instead.
  • WebNotifyApiUrl (icon location)
    • OR WebNotifyApiUrlAttr to specify the attribute to use instead.

Configuration

$ cp config/app.js.dist config/app.js

Notes about the main properties:

• sources
  • path: Absolute path to the log file.
  • parser: Parser class/class-file name, e.g. Json or NginxAccess.
  • tags: (Optional) One or more tags to automatically attach to every event.
  • timeAttr: (Optional) By default, mainevent expects parsers to return a time property to represent the event's timestamp. Select a different property name here.
  • To specify a remote host (all are required):
    • sshKey: Ex. /path/to/my/ec2.pem
    • sshPort: Ex. 22
    • sshUser: Ex. ubuntu
    • sshHost: Ex. ec2-135-28-52-91.compute-1.amazonaws.com
  • previewAttr: (Optional) Allows parsers like Json, which do not have preview templates, to know which properties should be included in preview text.
• express
  • mainevent_server: Verify the default port/ip.
  • test_server: Verify the default port/ip.
• mongodb
  • Customize the collection name in collections.event.
  • Select a different pagination maximum in maxResultSize if needed.
  • Add additional indexes if needed. Future versions may automate that process based on metrics.
• redis
  • host/port/options: Passed to createClient() in node_redis.

Main Components

Frontend

mainevent_server.js

$ bin/mainevent_server.js

Has three responsibilities:

1. / serves the backbone.js MVC app via express.js.
2. /api serves JSON data including graph points, event objects and timeline pages via express.js.
3. /socket.io serves real-time timelime updates.

Triggers public/build.js on startup to build the static/ directory.

public/build.js

$ public/build.js [--prod]

• Combines and compresses (--prod) JS/CSS files located in public/.
• Relies on public/js/app.build.js for RequireJS configuration.
• Triggers public/js/templates.build.js to compile dust.js templates.

Outputs all files into static/.

public/js/templates.build.js

Compiles dust.js templates in app/views/ and app/parsers/*/templates/.

public/js/app.build.js

RequireJS configuration for client-side dependencies.

Background

bin/tail.js

$ bin/tail.js $ bin/tail.js --help

Spawns tail instances for each source described in config/app.js.

app/graphs/CountAllPartitioned.js

Required by the dashboard.

$ app/graphs/CountAllPartitioned.js --verbose --jobWait 2 --chunkWait 2 --limit 1000 $ app/graphs/CountAllPartitioned.js --help

Generates the cached data used by the graph on /dashboard visualizing total events. Runs continually and sleeps for 1 minute if no new events are read.

Utilities

bin/import.js

$ bin/import.js --parser json --path /var/log/myApp/prod.json --tags myApp,import $ bin/import.js --help

Like tail.js except it processes the entire file. (The file does not need to be described in config/app.js.)

File Layout Notes

• app/ : Holds most server-side modules and classes.
  • controllers/: Handlers for express.js routes defined in bin/mainevent_server.js.
  • graphs/: Background scripts which run at intervals to cache point data in Redis.
  • jobs/: Classes used by graphs scripts which define the MapReduce logic.
  • modules/: Covers MongoDB, Redis, static builds and global objects like mainevent.
  • parsers/: Self-contained parser modules, their prototype, and a test utility module.
  • sockets/: Like controllers except for socket messages rather than routes.
  • views/: All non-parser dust.js templates.
• bin/: All HTTP servers and background processes like tail.js.
• public/
  • js/
    • backbone/: Additions to backbone.js prototypes like Backbone.View.prototype.
    • collections/: backbone.js collections.
    • controllers/: Handlers for backbone.js routes.
    • helpers/: Ex. mainevent.helpers.Socket for creating new socket.io connections.
    • models/: backbone.js models.
    • observers/: Global listeners of custom events like ContentPreRender.
    • shared/: Modules/classes available server-side and client-side, ex. mainevent.shared.Date.
    • views/: backbone.js views.
• static/: JS/CSS/images from public/ processed by public/build.js.
• test/
  • browser/: Client-side tests processed by app/views/test.html.
  • modules/: Test helpers.

Testing

Server-side tests rely on nodeunit. Example:

$ nodeunit test/redis.js

Run all tests found under app/parsers/ and test/.

$ npm test

Remote Logs

$ cp test/fixtures/tail-config.js test/fixtures/tail-config-remote.js

Update ssh* configuration values in test/fixtures/tail-config-remote.js.

Events

Server-side

• InsertLog
  • Triggered in app/modules/mongodb.js in insertLog().
  • Callbacks receive the array of document objects.
  • Example listener.

Client-side

• LinkClick
  • Triggered in public/js/helpers/Event.js on any link with a relative href.
• CritFetchError
  • Triggered when a view cannot fetch data critical to its presentation, ex. the event view cannot retrieve the object describing the event.
  • Callbacks receive the response object from $.ajax.
• ContentPreRender
  • Triggered in public/js/index.js before the content.html template is rendered.

Bundled dependencies and their licenses

• backbone.js (MIT)
• clientsiiide (MIT)
• dust.js (MIT)
• Glyphicons Free (CC BY 3.0)
• jqPlot (MIT/GPLv2)
• jQuery (MIT/GPLv2)
• jQuery UI (MIT/GPLv2)
  • Core, Widget, Mouse, Datepicker, Slider
• jQuery-Timepicker-Addon (MIT/GPLv2)
• moment.js (MIT)
• RequireJS (MIT/New BSD)
• socket.io (MIT)
• Twitter Bootstrap (Apachev2)
• underscore.js (MIT)
• XRegExp (MIT)

Copyright and license (MIT)

Copyright (c) 2012 David Smith, [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.