redis-log-valve

A Tomcat Valve for sending access log events to a Redis server for easy integration with Logstash.

It also protects the JVM against OOM's from too many buffered events. And it exposes metrics about its throughput and data loss.

Configuration

Configuration for the Valve should be added to Tomcat's server.xml

<Server>
  ..
  <Service>
    ..
    <Host>
    ..
      <Engine>
      ..

        <Valve className="com.bol.tomcat.valve.RedisLogValve"
               alwaysBatch="false"
               batchSize="100"
               flushInterval="1000"
               key="logstash.pro.tomcat_access"
               queueSize="5000"
               redisHost="shd-logredis-pro-vip.bolcom.net"
               redisPort="6379"
               userFields="application:pie,role:pie-app,opex_team:team2a"
               />
       </Host>
     </Engine>
  </Service>
</Server>

Where:

• redisHost (optional, default: localhost) Hostname/IP of the Redis server to sends events to.
• redisPort (optional, default: 6379) Port number of Redis server.
• key (required) Redis key of the list to RPUSH events to.
• password (optional) Redis password, if required.
• alwaysBatch (optional, default: true) whether to wait for a full batch. If true, will only send once there are batchSize events enqueued.
• batchSize (optional, default: 100) the number of events to send in a single Redis RPUSH command.
• flushInterval (optional, default: 500) the period in milliseconds between flush attempts. If events are flushed depends on the 'alwaysBatch' setting and the number of events in the buffer.
• queueSize (optional, default: 5000) the maximum number of events the Valve holds in memory, awaiting flush. If flushing is not possible, or is too slow the queue will slowly fill up. When the queue is full, new events will be dropped to protect the JVM.
• purgeOnFailure (optional, default: true) whether to purge/drop events if Redis responds to a RPUSH with an OOM error. If 'false' the appender will attempt to send the events to Redis. If that keeps failing the queue will slowly fill up and new events will be dropped.
• userFields (optional, default: none) comma separated list of key-values to be added to the events. Each key-value is separated by a colon character.

Event layout

Example JSON document as generated by the Valve:

{
  "@fields": {
    "response": 200,
    "timestamp": 1444680772320,
    "unique_id": "zVfU54jcApepAi07CWHDWA**.ps1^VhwURApirxcAANMg1tkAAAGU",
    "httpversion": "HTTP/1.1",
    "application": "abc",
    "bytes": 24,
    "valve_version": "1.3.6",
    "verb": "POST",
    "client": "1.2.3.4",
    "time_in_sec": 0,
    "time_in_msec": 1,
    "role": "abc-app",
    "host_header": "my-app:8080",
    "mime": "application/octet-stream",
    "agent": "httpclient/1.3"
  },
  "@message": "/v1/shopping_contexts",
  "@source_host": "tst-abc-app-001"
}

The @message field contains the request URI with optional query string added, separated by a ? character.

The @source_host fields contains the local hostname, without the domain name.

The @fields fields are:

• response: the HTTP response code (e.g. 200 = OK).
• timestamp: request processing start time in milliseconds since epoch. Logstash needs to convert this fields into the default timestamp field.
• unique_id (optional): value for the X-RID request header.
• httpversion: value of the HTTP protocol used (e.g. HTTP/1.0 or 1.1).
• bytes: response byte size
• valve_version: version of the Valve, usefull in big infra to track JVMs using older versions.
• verb: HTTP method used (e.g. POST, GET).
• client: client IP address.
• time_in_sec: processing time in whole seconds.
• time_in_msec: processing time in milliseconds.
• host_header: value of the Host request header.
• mime: MIME type of response.
• agent: valve of the User-Agent request header.

The application and role fields in the example are custom fields. See the configuration setting userFields.

Metrics

Tomcat automagically exposes the Valve as an MBean. The following metrics are available under MBean "Catalina:host=localhost,name=RedisLogValve,type=Valve":

• eventCounter: (counter) number of events received by the Valve and put in the queue. Configuration queueSize controls the the maximum number of events this queue can hold.
• eventsDroppedInQueueing: (counter) number of events that got dropped, because the queue was full. You get a full queue if Redis is full or your application is processing requests faster than the resulting events can be pushed to Redis. You can increase queueSize.
• eventsDroppedInPush: (counter) number of events that got dropped, because the Redis' memory is full and responds to RPUSH commands with an OOM error.
• connectCounter: (counter)number of connects made to the Redis server(s).
• connectFailures: (counter) number of connect attempts that failed.
• batchPurges: (counter) number of times the purge queue got purged. This only happens if purgeOnFailure is set to true.
• eventsPushed: (counter) number of events succesfully pushed to Redis.
• eventQueueSize: (gauge) number of events in the queue. When reading this attribute you get a sampled value. Use this to get a feel about the average number of queued events. When this gets above 50% of queueSize you may want to investigate a slow Redis server or silly amounts of events being emitted by your application.

JCollectd configuration

Use Jcollectd in your JVM to periodically flush MBean metrics to collectd or Diamond's JCollectdCollector.

Jcollectd XML config for this appender:

<jcollectd-config>
  <mbeans name="tomcat">
    <mbean name="Catalina:host=localhost,name=RedisLogValve,type=Valve" alias="RedisValve">
      <attribute name="connectCounter" type="counter"/>
      <attribute name="connectFailures" type="counter"/>
      <attribute name="eventCounter" type="counter"/>
      <attribute name="eventsPushed" type="counter"/>
      <attribute name="eventsDroppedInPush" type="counter"/>
      <attribute name="eventsDroppedInQueueing" type="counter"/>
      <attribute name="eventQueueSize" />
      <attribute name="batchPurges" type="counter"/>
    </mbean>
  </mbeans>
</jcollectd-config>

Future plans

Some ideas for future improvements:

• add support for the Logstash V1 event format (we still use V0)
• add the X-Forwarded-For request header value to the event
• add sampling or a per-minute quota of events
• got ideas? let me know

Contributors

• Renzo Toma
• Roland Kool
• Luis Novais

Contribution

Feel free to create an issue or submit a pull request.

License

Published under Apache Software License 2.0, see LICENSE

Acknowledgement

This program was originally developed by bol.com and published as Open Source on github.