Fallback values for templated fields

[lukesteensen]

When interpolating event fields, it would be helpful to specify a fallback value for when the specified field is not present.

Some questions to answer before implementing:

• Should we support falling back to a different field?
• Should we support a chain of fallbacks?
• How to determine when to fall back? Only when the field is not present? What about empty values?

[binarylogic]

Yep, I think this could potentially fall into a larger "field interpolation macros" project. Ex: a liquid style syntax with support for defaults, truncation, etc.

I definitely do not want to explode scope on this issue, but it's worth a quick discussion around that before we implement our own templating syntax.

[benzaita]

Should this support falling back to null values?

For example, I'm trying to use:

[sinks.s3]
  type = "aws_s3"
  endpoint = "${AWS_S3_ENDPOINT_URL}"
  region = "${AWS_REGION}"

But when AWS_S3_ENDPOINT_URL is not defined, this falls back to an empty string. The s3 sink then tries to use "" as the endpoint.

This could be a corner case. Or maybe this should be handled in the s3 sink.

[jszwedko]

We think VRL could be a good fit here if there isn't significant performance overhead. Ref: #17501

[pgassmann]

We saw a lot of errors in vectors because of some labels that are not present on certain containers.

ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}: vector::internal_events::template: Failed to render template for "label_value "{{ label."com.docker.compose.container-number" }}" with label_key "compose_container_number"". error=Missing fields on event: ["label.\"com.docker.compose.container-number\""] error_type="template_failed" stage="processing" internal_log_rate_limit=true
ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}: vector::internal_events::template: Internal log [Failed to render template for "label_value "{{ label."com.docker.compose.container-number" }}" with label_key "compose_container_number"".] is being suppressed to avoid flooding.
ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}: vector::internal_events::template: Internal log [Failed to render template for "label_value "{{ label."com.docker.compose.container-number" }}" with label_key "compose_container_number"".] has been suppressed 14 times.

I implemented the workaround with transforms and VRL. But I would prefer a simpler way to just drop the label if it cannot find the key.

# Read logs from Docker API and Send to loki sink

[sources.docker-local]
  type = "docker_logs"
  docker_host = "/var/run/docker.sock"
  exclude_containers = []

  # Identify zero-width space as first line of a multiline block.
  multiline.condition_pattern = '^\x{200B}' # required
  multiline.mode = "halt_before" # required
  multiline.start_pattern = '^\x{200B}' # required
  multiline.timeout_ms = 1000 # required, milliseconds

[transforms.docker-local_with_defaults]
  inputs = ["docker-local"]
  type   = "remap"
  source = '''
      # workaround for missing fallback option.
      if exists(.label."com.docker.compose.container-number") {
        .compose_container_number = .label."com.docker.compose.container-number"
      } else {
        .compose_container_number = ""
      }
      if exists(.label."com.docker.compose.project") {
        .compose_project = .label."com.docker.compose.project"
      } else {
        .compose_project = ""
      }
      if exists(.label."com.docker.compose.project.working_dir") {
        .compose_project_working_dir = .label."com.docker.compose.project.working_dir"
      } else {
        .compose_project_working_dir = ""
      }
      if exists(.label."com.docker.compose.oneoff") {
        .compose_oneoff = .label."com.docker.compose.oneoff"
      } else {
        .compose_oneoff = ""
      }
      if exists(.label."com.docker.compose.service") {
        .compose_service = .label."com.docker.compose.service"
      } else {
        .compose_service = ""
      }

  '''

[sinks.loki]
  # General
  type = "loki" # required
  inputs = ["docker-local_with_defaults"] # required
  endpoint = "https://loki.example.com:443" # required


  # Auth
  auth.strategy = "basic" # required
  auth.user = "asdfasfd" # required
  auth.password = "asdfasdf" # required

  # Encoding
  # encoding.codec = "json" # Content = all info as json structure
  encoding.codec = "text"   # Content = only the actual log line, message field from the event.

  # Healthcheck
  healthcheck.enabled = false # optional, default true

  buffer.max_events = 5000 # default: 500

  batch.max_bytes = 1024000 # default 102400 (0.1mb)
  batch.max_events = 50000
  batch.timeout_secs =  5 # default 1
  request.retry_max_duration_secs = 600 # default 3600, 1h

  compression = "gzip" # default none, promtail uses snappy (not available in vector)

  out_of_order_action = "accept"

  # Labels

  labels.instance = 'dev01'

  labels.forwarder = 'vector'
  labels.category = 'dockerlogs'
  labels.host = '{{ host }}'
  labels.source_type =  '{{ source_type }}'

  labels.source = '{{ stream }}'
  labels.stream = '{{ stream }}'

  labels.container_created_at = '{{ container_created_at }}'
  labels.container_id = '{{ container_id }}'
  labels.container_name = '{{ container_name }}'
  labels.container_image = '{{ image }}'

  ## add all labels as json, see ansible vector role Readme.
  labels.container_labels = '{{ label }}'

  ## add labels useful for log filtering
  labels.compose_container_number = '{{ compose_container_number }}'
  labels.compose_oneoff = '{{ compose_oneoff }}'
  labels.compose_project = '{{ compose_project }}'
  labels.compose_project_working_dir = '{{ compose_project_working_dir }}'
  labels.compose_service = '{{ compose_service }}'

  ## container labels not useful for log filtering. info can be extracted from container_labels
  # labels.compose_config_hash = '{{ label."com.docker.compose.config-hash" }}'
  # labels.compose_project_config_files = '{{ label."com.docker.compose.project.config_files" }}'
  # labels.compose_version = '{{ label."com.docker.compose.version" }}'

  # labels.container_image_created = '{{ label."org.opencontainers.image.created" }}'
  # labels.container_image_revision = '{{ label."org.opencontainers.image.revision" }}'
  # labels.container_image_source = '{{ label."org.opencontainers.image.source" }}'
  # labels.container_image_url = '{{ label."org.opencontainers.image.url" }}'

[jszwedko]

Related #17487