Skip to content
This repository has been archived by the owner on Aug 13, 2024. It is now read-only.

Webhooks

Jump to bottom
Josh edited this page Jun 26, 2018 · 5 revisions

Using webhooks with Factotum

As of version 0.3.0, Factotum supports streaming job information over HTTP. No changes to the Factfile are required, to stream job updates with your existing job, use the following command:

factotum run <your Factfile> --webhook "http://my-collector.com"

As of version 0.5.0, Factotum supports varying the size of the Webhook stdout and stderr fields from the default 10_000 characters.

factotum run <your Factfile> --webhook "http://my-collector.com" --max-stdouterr-size 20000

When updates are sent to the webhook

Updates are sent when the state of the job changes:

  1. Information that a new run has started is sent (all tasks are in the state WAITING)
  2. As each task is started it's state changes from WAITING to RUNNING and a new update is sent to the endpoint
  3. As each task is completed, it's state changes from RUNNING to SUCCESS (or FAILED) and a new update is sent
  4. When Factotum has no more tasks to execute, a COMPLETED update is sent, with the final states of each task.

Update format

Factotum updates are a single self describing event:

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "self": {
    "vendor": "com.snowplowanalytics.factotum",
    "name": "job_update",
    "version": "1-0-0",
    "format": "jsonschema"
  },
  "type": "object",
  "properties": {
    "jobName": {
      "type": "string"
    },
    "runReference": {
      "type": "string"
    },
    "factfile": {
      "type": "string"
    },
    "applicationContext": {
      "type": "object",
      "properties": {
        "version": {
          "type": "string",
          "pattern": "\\d+\\.\\d+\\.\\d+-?.*"
        }
      },
      "required": [
        "version"
      ],
      "additionalProperties": false
    },
    "jobReference": {
      "type": "string"
    },
    "runState": {
      "enum": [
        "RUNNING",
        "WAITING",
        "COMPLETED",
        "FAILED"
      ]
    },
    "startTime": {
      "type": "string",
      "format": "date-time"
    },
    "runDuration": {
      "type": "string"
    },
    "taskStates": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "taskName": {
            "type": "string"
          },
          "state": {
            "enum": [
              "RUNNING",
              "WAITING",
              "COMPLETED",
              "FAILED",
              "SKIPPED"
            ]
          },
          "started": {
            "type": "string",
            "format": "date-time"
          },
          "duration": {
            "type": "string"
          },
          "stdout": {
            "type": "string"
          },
          "stderr": {
            "type": "string"
          },
          "returnCode": {
            "type": "integer",
            "maximum": 32767,
            "minimum": -32767
          },
          "error_message": {
            "type": "string"
          }
        },
        "required": [
          "taskName",
          "state"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [
    "jobName",
    "jobReference",
    "runReference",
    "runState",
    "factfile",
    "applicationContext",
    "startTime",
    "runDuration",
    "taskStates"
  ],
  "additionalProperties": false
}

Here's a sample of the JSON Factotum will POST to your webhook.

{
  "schema": "iglu:com.snowplowanalytics.factotum/job_update/jsonschema/1-0-0",
  "data": {
    "jobName": "echo order demo",
    "jobReference": "89116d60c592d62068862ef88aa5cd400e195e5d2311ddc564ab5d00b617624e",
    "runReference": "e77eb3e0377eb3462b3ecf90e3afd07034cf80da981ede9a3b8d50a2c28ec8f0",
    "factfile": "ewogICAgInNjaGVtYSI6ICJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5mYWN0b3R1bS9mYWN0ZmlsZS9qc29uc2NoZW1hLzEtMC0wIiwKICAgICJkYXRhIjogewogICAgICAgICJuYW1lIjogImVjaG8gb3JkZXIgZGVtbyIsCiAgICAgICAgInRhc2tzIjogWwogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIGFscGhhIiwKICAgICAgICAgICAgICAgICJleGVjdXRvciI6ICJzaGVsbCIsCiAgICAgICAgICAgICAgICAiY29tbWFuZCI6ICJlY2hvIiwKICAgICAgICAgICAgICAgICJhcmd1bWVudHMiOiBbICJhbHBoYSIgXSwKICAgICAgICAgICAgICAgICJkZXBlbmRzT24iOiBbXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9LAogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIGJldGEiLAogICAgICAgICAgICAgICAgImV4ZWN1dG9yIjogInNoZWxsIiwKICAgICAgICAgICAgICAgICJjb21tYW5kIjogImVjaG8iLAogICAgICAgICAgICAgICAgImFyZ3VtZW50cyI6IFsgImJldGEiIF0sCiAgICAgICAgICAgICAgICAiZGVwZW5kc09uIjogWyAiZWNobyBhbHBoYSIgXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9LAogICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAibmFtZSI6ICJlY2hvIG9tZWdhIiwKICAgICAgICAgICAgICAgICJleGVjdXRvciI6ICJzaGVsbCIsCiAgICAgICAgICAgICAgICAiY29tbWFuZCI6ICJlY2hvIiwKICAgICAgICAgICAgICAgICJhcmd1bWVudHMiOiBbICJhbmQgb21lZ2EhIiBdLAogICAgICAgICAgICAgICAgImRlcGVuZHNPbiI6IFsgImVjaG8gYmV0YSIgXSwKICAgICAgICAgICAgICAgICJvblJlc3VsdCI6IHsKICAgICAgICAgICAgICAgICAgICAidGVybWluYXRlSm9iV2l0aFN1Y2Nlc3MiOiBbIDMgXSwKICAgICAgICAgICAgICAgICAgICAiY29udGludWVKb2IiOiBbIDAgXQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9CiAgICAgICAgXQogICAgfQp9Cg==",
    "applicationContext": {
      "version": "0.3.0"
    },
    "runState": "COMPLETED",
    "startTime": "2016-10-13T13:46:49.001109575+00:00",
    "runDuration": "PT1.847529443S",
    "taskStates": [
      {
        "duration": "PT0.002371592S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.001385064+00:00",
        "state": "COMPLETED",
        "stdout": "alpha",
        "taskName": "echo alpha"
      },
      {
        "duration": "PT0.002340387S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.004452530+00:00",
        "state": "COMPLETED",
        "stdout": "beta",
        "taskName": "echo beta"
      },
      {
        "duration": "PT0.001891810S",
        "returnCode": 0,
        "started": "2016-10-13T13:46:49.007247325+00:00",
        "state": "COMPLETED",
        "stdout": "and omega!",
        "taskName": "echo omega"
      }
    ]
  }
}

Field glossary

Field Notes
schema Self describing event wrapper
data Self describing event wrapper
data.jobName The name of the job, as it appears in the Factfile
data.jobReference An ID unique to the Factfile for this job
data.runReference A globally unique ID for this run
data.factfile A base64 encoded copy of the Factfile that's running
data.applicationContext.version The version of Factotum that's executing the job
data.runState The current state of the job. This can be WAITING,RUNNING,COMPLETED or FAILED
data.startTime The time the job started, in RFC 3339 format
data.runDuration The running time of the job so far in RFC 3339 duration format
data.taskStates An array of information on the state of each task
data.taskStates[_].taskName The name of the task, as it appears in the Factfile
data.taskStates[_].state The current state of the task. This can be WAITING,RUNNING,COMPLETED,FAILED or SKIPPED
data.taskStates[_].started Optional. The RFC 3339 start time of the task
data.taskStates[_].duration Optional. The RFC 3339 duration of the task
data.taskStates[_].stdout Optional. The output of the task to stdout
data.taskStates[_].stderr Optional. The output of the task to stderr
data.taskStates[_].returnCode Optional. The return code of the task
data.taskStates[_].errorMessage Optional. The reason the task failed, or was skipped

Retry behaviour

If your endpoint is unavailable or doesn't return 200, Factotum will attempt to send each update three times with a random back-off period of up to one minute between.

Updates are not cached to disk, and if the endpoint cannot be reached during a run, it will not be re-sent in subsequent runs.

Clone this wiki locally