Feature: transformation handlers

[ghost]

Pull request #6 ("Generalized mail parsing") introduced mail handlers, which inspired extending the same idea to other contexts. Specifically, this pull request introduces transformation handlers that can be run as XMPP bots like any abusehelper.core.transformation, but can also be reused as command line tools and in automated tests.

This pull request also adds abusehelper.core.cymruwhois.Handler, a transformation handler that adds Cymru ASN lookup data to the input events. By default it watches all event keys for values that look like IP addresses but it can be configured to watch only a certain limited set of keys with e.g. {"type": "abusehelper.core.cymruwhois.Handler", "ip_keys": ["ip", "netblock"]}.

Writing a Transformation Handler

Writing a new transformation handler is very similar to writing a mail handler. Let's put this to myhandler.py:

import idiokit
from abusehelper.core import transformation


class MyHandler(transformation.Handler):
    def __init__(self, extras={"abc": "xyz"}, *args, **keys):
        transformation.Handler.__init__(self, *args, **keys)
        self.extras = extras

    @idiokit.stream
    def transform(self):
        self.log.info("Now transforming!")

        while True:
            event = yield idiokit.next()
            yield idiokit.send(event.union(self.extras))

Running from Command Line

abusehelper.core.transformation.tester allows us to run a handler as a command line tool:

$ python -m abusehelper.core.transformation.tester myhandler.MyHandler <<EOF
{"ip": "192.0.2.0"}
EOF

Outputs should look like this:

2016-05-14 01:23:06Z INFO Now transforming!
{"ip": ["192.0.2.0"], "abc": ["xyz"]}

Also configuring the handler works:

$ python -m abusehelper.core.transformation.tester '{"type": "myhandler.MyHandler", "extras": {"abc": "123"}}' <<EOF
{"ip": "192.0.2.0"}
EOF

Output:

2016-05-14 01:25:43Z INFO Now transforming!
{"ip": ["192.0.2.0"], "abc": ["123"]}

Running as a XMPP Bot

$ python -m abusehelper.core.transformation.bot [email protected] lobby.room myhandler.MyHandler

The launched bot behaves like any abusehelper.core.transformation bot. Session parameters src_room and dst_room control where the input events are read from and where the output events are delivered to.

Note that currently session parameters aren't routed to the handler, and the only way to configure a handler is to pass in startup parameters:

$ python -m abusehelper.core.transformation.bot [email protected] lobby.room '{"type": "myhandler.MyHandler", "extras": {"abc": "123"}}'

Creating Automated Tests

The following test_myhandler.py is an example how abusehelper.core.transformation.tester.handle takes in a handler and a list of input events and returns a list of output events:

import unittest
from abusehelper.core.transformation.tester import handle
from myhandler import MyHandler


class TestMyHandler(unittest.TestCase):
    def test_should_modify_events(self):
        self.assertEqual(
            handle(MyHandler, [{"ip": "192.0.2.0"}]),
            [{"ip": ["192.0.2.0"], "abc": ["xyz"]}]
        )