Proposal: Develop Loggly logging plugin

[bisakhmondal]

Issue description

Initial phase:

reference : here
A logging plugin that works and does its job : )
Supported Protocol: SYSLOG-UDP

schema:

local schema = {
    type = "object",
    properties = {
        customer_token = {type = "string"},
        severity = {
            type = "string",
            default = "INFO",
            enum = {"DEBUG", "INFO", "NOTICE", "WARNING", "ERR", "CRIT", "ALERT", "EMEGR",
                    "debug", "info", "notice", "warning", "err", "crit", "alert", "emegr"},
            description = "base severity log level",
        },
        include_req_body = {type = "boolean", default = false},
        include_resp_body = {type = "boolean", default = false},
        include_resp_body_expr = {
            type = "array",
            minItems = 1,
            items = {
                type = "array"
            }
        },
        tags = {
            type = "array",
            minItems = 1,
            items = {
                type = "string"
            }
        },
        prefer_name = {type = "boolean", default = true}
    },
    required = {"customer_token"}
}

Metadata schema

local defaults = {
    host = "logs-01.loggly.com",
    port = 514,
    protocol = "syslog",
    timeout = 5000
}

local metadata_schema = {
    type = "object",
    properties = {
        host = {
            type = "string",
            default = defaults.host
        },
        port = {
            type = "integer",
            default = defaults.port
        },
        protocol = {
            type = "string",
            default = defaults.protocol,
            -- more methods coming soon
            enum = {"syslog"}
        },
        timeout = {
            type = "integer",
            minimum = 1,
            default= defaults.timeout
        },
        log_format = {
            type = "object",
        }
    }
}

Phase-2

• Supported Protocol: HTTP[S]
• Support Changing Log Severity level based on HTTP response code.

This issue is for the tracking purpose of the plugin development. If you have any useful insight, feel free to let us know : )

[spacewander]

I think we can remove include_req_body & tags?
Instead, we can use log_util.get_full_log & log_util.get_custom_format_log. The tags field can be implemented via get_custom_format_log.

[bisakhmondal]

I think we can remove include_req_body & tags? Instead, we can use log_util.get_full_log & log_util.get_custom_format_log. The tags field can be implemented via get_custom_format_log.

Hey, thanks for the nice suggestions. I looked into the http_logger implementation of custom_log_format definition in the plugin metadata. The default log format looks very minimal, I don't think it's suitable for logging. I think we can do one thing:

• Make this a field inside the plugin schema because tags must be configured route-specific instead of making it apisix specific.
• If log_format == empty ; dump get_full_log else generate get_custom_format_log

But one thing, even with tags shifted to the log format, we are going to expect the log format object something like
{
"host": "$host",
....
"tags": ["route-critical"]
}
Because we have to extract it from the generated log_format body and use it inside Syslog header generation.
So I think we are just making things convoluted, isn't it? WDYT. Just share some thoughts here. Thank you.

[spacewander]

The custom log format is very powerful. Note that you can use a custom variable to generate a custom log format. So we can register a variable which just returns different tags in different routes.

If log_format == empty ; dump get_full_log else generate get_custom_format_log

Do we already do it?

use it inside Syslog header generation

So the tag is not part of the log body? Could you explain it in more detail?

[bisakhmondal]

So the tag is not part of the log body? Could you explain it in more detail?

So in Loggly tags can be a crucial part of filtering. In syslog packet structure they are added to the syslog header part. In the next PR when we add support for emitting logs to loggly HTTP endpoints, these tags will be a part of HTTP headers. In both cases, the body contains the typical apisix payload.
see this:

Extra reference -> https://documentation.solarwinds.com/en/success_center/loggly/content/admin/tags.htm

[spacewander]

I see. So we need to use a separate configuration.