syslog-ng-4.8.1

4.8.1

Highlights

• elasticsearch-datastream() destinations can be used to feed Elasticsearch data streams.

  Example config:

  elasticsearch-datastream(
    url("https://elastic-endpoint:9200/my-data-stream/_bulk")
    user("elastic")
    password("ba3DI8u5qX61We7EP748V8RZ")
  );
  

  (#5069)

• building: thanks to Sergey Fedorov and Marius Schamschula, macOS builds now support gcc again. They also updated the MacPort version of syslog-ng (develop). Great work, and thank you so much for your contribution!
  (#5108)

Features

• tls(): expose the key fingerprint of the peer in ${.tls.x509_fp} if
  trusted-keys() is used to retain the actual peer identity in the received
  messages.
  (#5068)

• syslog-parser: Added the no-piggyback-errors and the piggyback-errors flags to control whether the message retains the original message or not on parse error(s). By default the old behaviour/piggyback-errors flag is active.

  • no-piggyback-errors: On failure, the original message will be left as it was before parsing, the value of $MSGFORMAT will be set to syslog-error, and a tag will be placed on the message corresponding to the parser's failure.
  • piggyback-errors: On failure, the old behaviour is used (clearing the entire message then syslog-ng will generate a new message in place of the old one describing the parser's error).

  The following new tags can be added by the syslog-parser to the message when the parsing failed:

  • syslog.rfc5424_missing_hostname
  • syslog.rfc5424_missing_app_name
  • syslog.rfc5424_missing_procid
  • syslog.rfc5424_missing_msgid
  • syslog.rfc5424_missing_sdata
  • syslog.rfc5424_invalid_sdata
  • syslog.rfc5424_missing_message
    (#5063)

Bugfixes

• syslog-ng-ctl: fix escaping of stats prometheus

  Metric labels (for example, the ones produced by metrics-probe()) may contain control characters, invalid UTF-8 or \
  characters. In those specific rare cases, the escaping of the stats prometheus output was incorrect.
  (#5046)

• wildcard-file(): fix crashes can occure if the same wildcard file is used in multiple sources

  Because of some persistent name construction and validation bugs the following config crashed syslog-ng
  (if there were more than one log file is in the /path folder)

  @version: current
  
  @include "scl.conf"
  
  source s_files1 {
      file("/path/*.log"
          persist-name("p1")
      );
  };
  
  source s_files2 {
      file("/path/*.log"
          persist-name("p2")
      );
  };
  
  destination s_stdout {
      stdout();
  };
  
  log {
      source(s_files1);
      destination(s_stdout);
  };
  
  log {
      source(s_files2);
      destination(s_stdout);
  };
  

  NOTE:

  • The issue occurred regardless of the presence of the persist-name() option.
  • It affected not only the simplified example of the legacy wildcard file() but also the new wildcard-file() source.
    (#5091)

• syslog-ng-ctl: fix crash of syslog-ng service in g_hash_table lookup function after syslog-ng-ctl reload
  (#5087)

• file(), stdout(): fix log sources getting stuck

  Due to an acknowledgment bug in the file() and stdout() destinations,
  sources routed to those destinations may have gotten stuck as they were
  flow-controlled incorrectly.

  This issue occured only in extremely rare cases with regular files, but it
  occured frequently with /dev/stderr and other slow pseudo-devices.
  (#5134)

• directory-monitor: fixed a main thread assertion crash that might have occurred during syslog-ng stop or restart
  (#5086)

• Config @version: fixed compat-mode inconsistencies when @version was not specified at the top of the configuration
  file or was not specified at all
  (#5145)

• grpc: Fix potential memoryleak when the grpc module is loaded but not used.
  (#5062)

• s3(): Eliminated indefinite memory usage increase for each reload.

  The increased memory usage is caused by the botocore library, which
  caches the session information. We only need the Session object, if
  role() is set. The increased memory usage still happens with that set,
  currently we only fixed the unset case.
  (#5149)

• opentelemetry() sources: fix crash when workers() is set to > 1
  (#5138)

• opentelemetry() sources: fix source hang-up on flow-controlled paths
  (#5148)

• metrics-probe(): fix disappearing metrics from stats prometheus output

  metrics-probe() metrics became orphaned and disappeared from the syslog-ng-ctl stats prometheus output
  whenever an ivykis worker stopped (after 10 seconds of inactivity).
  (#5075)

• affile: Fix an invalid lseek call mainly on the pipe() source, but also possible if using affile on pipe like files (pipe, socket and FIFO).
  (#5058)

Other changes

• format-json: spaces around = in $(format-json) template function could cause a
  crash.
  The fix of the issue also introduced an enhancement, from now on spaces are allowed
  around the = operator, so the following $(format-json) template function calls
  are all valid:

  $(format-json foo =alma)
  $(format-json foo= alma)
  $(format-json foo = alma)
  $(format-json foo=\" alma \")
  $(format-json foo= \" alma \")
  $(format-json foo1= alma foo2 =korte foo3 = szilva foo4 = \" meggy \" foo5=\"\")
  

  Please note the usage of the escaped strings like \" meggy \", and the (escaped and) quoted form
  that used for an empty value \"\", the latter is a breaking change as earlier an expression like
  key= led to a json key-value pair with an empty value {"key":""} that will not work anymore.
  (#5080)

• building: fixed multiple potentional FreeBSD build errors
  (#5099)

• docker: Changed the container image's base to debian:bookworm.
  (#5056)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Attila Szakacs, Balazs Scheidler, Hofi,
Kovács Gergő Ferenc, László Várady, Mate Ory,
Peter Czanik (CzP), Sergey Fedorov, Marius Schamschula, Szilard Parrag,
Tamas Pal, shifter

syslog-ng-4.8.0

4.8.0

We have new documentation

You can find our new up-to-date documentation in the new Administration Guide at syslog-ng.github.io.

Highlights

Default config version in configuration files

cfg: allow usage of current in config @version by default if it is not presented

This change allows syslog-ng to start even if the @version information is not present in the configuration file and treats the version as the latest in that case.

NOTE: syslog-ng will still raise a warning if @version is not present. Please use @version: current to confirm the intention of always using the latest version and silence the warning. (#5030)

BSD directory monitoring with kqueue

directory-monitor: Added a kqueue based directory monitor implementation.

wildcard-file() sources are using a directory monitor as well to aid detection of changes in the folders of the followed files. The new kqueue-based directory monitor uses far fewer resources than the poll based version on BSD-based systems.
(#5022)

See more at the new syslog-ng documentation.

Wildcard file source fine-tuning

wildcard-file(): Added a dedicated monitor_freq option to control the poll frequency of the change detection in the directories separately when the poll method is selected via the monitor-method() option.

The monitor-method() option controls only the change detection method in the directories, not the following of the file changes, and if poll is the selected method the frequency must not necessarily be the same, e.g. if the (earlier) commonly used follow-freq() is set to 0 for switching to the poll_fd_events method for file content change detection, that also might be meant a directory change poll with zero delays (if monitor-method() was set to poll as well), and that could cause a heavy CPU load unnecessarily.
(#4998)

See more at the new syslog-ng documentation.

Features

• s3(): Introduced server side encryption related options

  server-side-encryption() and kms-key() can be used to configure encryption.

  Currently only server-side-encryption("aws:kms") is supported.
  The kms-key() should be:

  • an ID of a key
  • an alias of a key, but in that case you have to add the alias/prefix
  • an ARN of a key

  To be able to use the aws:kms encryption the AWS Role or User has to have the following
  permissions on the given key:

  • kms:Decrypt
  • kms:Encrypt
  • kms:GenerateDataKey

  Check this page on why the kms:Decrypt is mandatory.

  Example config:

  destination d_s3 {
    s3(
      bucket("log-archive-bucket")
      object-key("logs/syslog")
      server-side-encryption("aws:kms")
      kms-key("alias/log-archive")
    );
  };
  

  See the S3 documentation for more details.
  (#4993)

• filter: Added numerical severity settings.

  The level filter option now accepts numerical values similar to facility.

  Example config:

  filter f_severity {
    level(4)
  };
  

  This is equivalent to

  filter f_severity {
    level("warning")
  };
  

  For more information, consult the documentation.
  (#5016)

• opentelemetry(), loki(), bigquery(): Added headers() option

  Enables adding headers to RPC calls.

  Example config:

  opentelemetry(
    ...
    headers(
      "my_header" = "my_value"
    )
  );
  

  (#5012)

• Added new proxy options to the syslog() and network() source drivers

  The transport(proxied-tcp), transport(proxied-tls), and transport(proxied-tls-passthrough) options are now available when configuring syslog() and network() sources.
  (#4544)

Bugfixes

• disk-buffer(): fix crash when pipeline initialization fails

  log_queue_disk_free_method: assertion failed: (!qdisk_started(self->qdisk))
  (#4994)

• rate-limit(): Fixed a crash which occured on a config parse failure.
  (#5033)

• Fixed potential null pointer deref issues
  (#5035)

• wildcard-file(): fix a crash and detection of file delete/move when using ivykis poll events

  Two issues were fixed

  • Fixed a crash in log pipe queue during file deletion and EOF detection (#4989)

    The crash was caused by a concurrency issue in the EOF and file deletion detection when using a wildcard-file() source.

    If a file is written after being deleted (e.g. with an application keeping the file open), or if these events happen concurrently, the file state change poller mechanism might schedule another read cycle even though the file has already been marked as fully read and deleted.

    To prevent this re-scheduling between these two checks, the following changes have been made:
    Instead of maintaining an internal EOF state in the WildcardFileReader, when a file deletion notification is received, the poller will be signaled to stop after reaching the next EOF. Only after both conditions are set the reader instance will be deleted.

  • Fixed the file deletion and removal detection when the file-reader uses poll_fd_events to follow file changes, which were mishandled. For example, files that were moved or deleted (such as those rolled by a log-rotator) were read to the end but never read again if they were not touched anymore, therefore switching to the new file never happened.
    (#4998)

• syslog-ng-ctl query: fix showing Prometheus metrics as unnamed values

  none.value=726685
  (#4995)

• macros: Fixed a bug which always set certain macros to string type

  The affected macros are $PROGRAM, $HOST and $MESSAGE.
  (#5024)

• syslog-ng-ctl query: show timestamps and fix g_pattern_spec_match_string assert
  (#4995)

• csv-parser(): fix escape-backslash-with-sequences dialect on ARM

  csv-parser() produced invalid output on platforms where char is an unsigned type.
  (#4947)

Other changes

• bigquery(), loki(), opentelemetry(), cloud-auth(): C++ modules can be compiled with clang

  Compiling and using these C++ modules are now easier on FreeBSD and macOS.
  (#4933)

• syslog-ng-ctl: do not show orphan metrics for stats prometheus

  As the stats prometheus command is intended to be used to forward metrics
  to Prometheus or any other time-series database, displaying orphaned metrics
  should be avoided in order not to insert new data points when a given metric
  is no longer alive.

  In case you are interested in the last known value of orphaned counters, use
  the stats or query subcommands.
  (#4921)

• s3(): new metric syslogng_output_event_bytes_total
  (#4958)

• multiline-options: Allow multi_line_timeout to be set to a non-integer value.

  Since multi_line_timeout is suggested to be set as a multiple of follow-freq, and follow-freq can be much smaller than one second, it makes sense to allow this value to be a non-integer as well.
  (#5002)

• packages/dbld: add support for Ubuntu 24.04 (Noble Numbat)
  (#4925)

• packages/dbld: add support for AlmaLinux 9
  (#5009)

• packages/dbld: added support for Fedora Rawhide and CentOS Stream 9 as testing platforms
  (#5009)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alex Becker, Andras Mitzki, Arpad Kunszt, Attila Szakacs,
Balazs Scheidler, Bálint Horváth, Dmitry Levin, Hofi, Ilya Kheifets,
joohoonmaeng, ktzsolt, László Várady, Mate Ory, Natanael Copa,
Peter Czanik, qsunchiu, Robert Fekete, shifter, Szilárd Parrag,
Tamas Pal, Wolfram Joost

syslog-ng-4.7.1

4.7.1

This is the combination of the news entries of 4.7.0 and 4.7.1.
4.7.1 hotfixed two crashes related to configuration reload.

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Collecting Jellyfin logs

The new jellyfin() source, reads Jellyfin logs from its log file output.

Example minimal config:

source s_jellyfin {
  jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
  );
};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on a wildcard-file() source, all of the
wildcard-file() source options are applicable, too.
(#4802)

Collecting *arr logs

Use the newly added *arr() sources to read various *arr logs:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
  radarr(
    dir("/path/to/my/radarr/log/dir")
  );
};

The logging module is stored in the <prefix><module> name-value pair,
for example: .radarr.module => ImportListSyncService.
The prefix can be modified with the prefix() option.
(#4803)

Features

• opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option.

  This option configures the maximal number of in-flight gRPC requests per worker.
  Setting this value to the range of 10s or 100s is recommended when there are a
  high number of clients sending simultaneously.

  Ideally, workers() * concurrent-requests() should be greater or equal to
  the number of clients, but this can increase the memory usage.
  (#4827)

• loki(): Support multi-tenancy with the new tenant-id() option
  (#4812)

• s3(): Added support for authentication from environment.

  The access-key() and secret-key() options are now optional,
  which makes it possible to use authentication methods originated
  from the environment, e.g. AWS_... environment variables or
  credentials files from the ~/.aws/ directory.

  For more info, see:
  https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
  (#4881)

• gRPC based drivers: Added channel-args() option.

  Affected drivers are:

  • bigquery() destination
  • loki() destination
  • opentelemetry() source and destination
  • syslog-ng-otlp() source and destination

  The channel-args() option accepts name-value pairs and sets channel arguments
  defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html

  Example config:

    opentelemetry(
      channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
      )
    );
  

  (#4827)

• ${TRANSPORT} macro: Added support for locally created logs.

  New values are:

  • "local+unix-stream"
  • "local+unix-dgram"
  • "local+file"
  • "local+pipe"
  • "local+program"
  • "local+devkmsg"
  • "local+journal"
  • "local+afstreams"
  • "local+openbsd"
    (#4777)

• tags: Added new built-in tags that help identifying parse errors.

  New tags are:

  • "message.utf8_sanitized"
  • "message.parse_error"
  • "syslog.missing_pri"
  • "syslog.missing_timestamp"
  • "syslog.invalid_hostname"
  • "syslog.unexpected_framing"
  • "syslog.rfc3164_missing_header"
  • "syslog.rfc5424_unquoted_sdata_value"
    (#4804)

• mqtt() source: Added ${MQTT_TOPIC} name-value pair.

  It is useful for the cases where topic() contains wildcards.

  Example config:

  log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
  };
  

  (#4824)

• template(): Added a new template function: $(tags-head)

  This template function accepts multiple tag names, and returns the
  first one that is set.

  Example config:

  # resolves to "bar" if "bar" tag is set, but "foo" is not
  template("$(tags-head foo bar baz)")
  

  (#4804)

• s3(): Use default AWS URL if url() is not set.
  (#4813)

• opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option.

  This option can be used to fine tune the performance. To minimize locking while
  moving messages between source and destination side queues, syslog-ng can move
  messages in batches. The log-fetch-limit() option sets the maximal size of
  the batch moved by a worker. By default it is equal to log-iw-size() / workers().
  (#4827)

• dqtool: add option for truncating (compacting) abandoned disk-buffers
  (#4875)

Bugfixes

• opentelemetry(): fix crash when an invalid configuration needs to be reverted
  (#4910)

• gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded
  (#4909)

• opentelemetry(), syslog-ng-otlp() source: Fixed a crash.

  It occurred with multiple workers() during high load.
  (#4827)

• rename(): Fixed a bug, which always converted the renamed NV pair to string type.
  (#4847)

• With IPv6 disabled, there were linking errors
  (#4880)

Metrics

• http(): Added a new counter for HTTP requests.

  It is activated on stats(level(1));.

  Example metrics:

  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
  syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
  

  (#4805)

• gRPC based destination drivers: Added gRPC request related metrics.

  Affected drivers:

  • opentelemetry()
  • syslog-ng-otlp()
  • bigquery()
  • loki()

  Example metrics:

  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
  

  (#4811)

• New metric to monitor destination reachability

  syslogng_output_unreachable is a bool-like metric, which shows whether a
  destination is reachable or not.

  sum() can be used to count all unreachable outputs, hence the negated name.

  It is currently available for the network(), syslog(), unix-*()
  destinations, and threaded destinations (http(), opentelemetry(), redis(),
  mongodb(), python(), etc.).
  (#4876)

• destinations: Added "syslogng_output_event_retries_total" counter.

  This counter is available for the following destination drivers:

  • amqp()
  • bigquery()
  • http() and all http based drivers
  • java()
  • kafka()
  • loki()
  • mongodb()
  • mqtt()
  • opentelemetry()
  • python() and all python based drivers
  • redis()
  • riemann()
  • smtp()
  • snmp()
  • sql()
  • stomp()
  • syslog-ng-otlp()

  Example metrics:

  syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
  

  (#4807)

• syslogng_memory_queue_capacity

  Shows the capacity (maximum possible size) of each queue.
  Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages.
  Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding
  source log-iw-size() is the upper limit.
  (#4831)

Other changes

• opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior.

  syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward
  the received message because of backpressure. Instead, syslog-ng will block until the
  destination can accept more messages.
  ([#4827](https://github.com/syslog-ng/syslog-ng/pull/...

syslog-ng-4.7.0

4.7.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Collecting Jellyfin logs

The new jellyfin() source, reads Jellyfin logs from its log file output.

Example minimal config:

source s_jellyfin {
  jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
  );
};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on a wildcard-file() source, all of the
wildcard-file() source options are applicable, too.
(#4802)

Collecting *arr logs

Use the newly added *arr() sources to read various *arr logs:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
  radarr(
    dir("/path/to/my/radarr/log/dir")
  );
};

The logging module is stored in the <prefix><module> name-value pair,
for example: .radarr.module => ImportListSyncService.
The prefix can be modified with the prefix() option.
(#4803)

Features

• opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option.

  This option configures the maximal number of in-flight gRPC requests per worker.
  Setting this value to the range of 10s or 100s is recommended when there are a
  high number of clients sending simultaneously.

  Ideally, workers() * concurrent-requests() should be greater or equal to
  the number of clients, but this can increase the memory usage.
  (#4827)

• loki(): Support multi-tenancy with the new tenant-id() option
  (#4812)

• s3(): Added support for authentication from environment.

  The access-key() and secret-key() options are now optional,
  which makes it possible to use authentication methods originated
  from the environment, e.g. AWS_... environment variables or
  credentials files from the ~/.aws/ directory.

  For more info, see:
  https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
  (#4881)

• gRPC based drivers: Added channel-args() option.

  Affected drivers are:

  • bigquery() destination
  • loki() destination
  • opentelemetry() source and destination
  • syslog-ng-otlp() source and destination

  The channel-args() option accepts name-value pairs and sets channel arguments
  defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html

  Example config:

    opentelemetry(
      channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
      )
    );
  

  (#4827)

• ${TRANSPORT} macro: Added support for locally created logs.

  New values are:

  • "local+unix-stream"
  • "local+unix-dgram"
  • "local+file"
  • "local+pipe"
  • "local+program"
  • "local+devkmsg"
  • "local+journal"
  • "local+afstreams"
  • "local+openbsd"
    (#4777)

• tags: Added new built-in tags that help identifying parse errors.

  New tags are:

  • "message.utf8_sanitized"
  • "message.parse_error"
  • "syslog.missing_pri"
  • "syslog.missing_timestamp"
  • "syslog.invalid_hostname"
  • "syslog.unexpected_framing"
  • "syslog.rfc3164_missing_header"
  • "syslog.rfc5424_unquoted_sdata_value"
    (#4804)

• mqtt() source: Added ${MQTT_TOPIC} name-value pair.

  It is useful for the cases where topic() contains wildcards.

  Example config:

  log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
  };
  

  (#4824)

• template(): Added a new template function: $(tags-head)

  This template function accepts multiple tag names, and returns the
  first one that is set.

  Example config:

  # resolves to "bar" if "bar" tag is set, but "foo" is not
  template("$(tags-head foo bar baz)")
  

  (#4804)

• s3(): Use default AWS URL if url() is not set.
  (#4813)

• opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option.

  This option can be used to fine tune the performance. To minimize locking while
  moving messages between source and destination side queues, syslog-ng can move
  messages in batches. The log-fetch-limit() option sets the maximal size of
  the batch moved by a worker. By default it is equal to log-iw-size() / workers().
  (#4827)

• dqtool: add option for truncating (compacting) abandoned disk-buffers
  (#4875)

Bugfixes

• opentelemetry(), syslog-ng-otlp() source: Fixed a crash.

  It occurred with multiple workers() during high load.
  (#4827)

• rename(): Fixed a bug, which always converted the renamed NV pair to string type.
  (#4847)

• With IPv6 disabled, there were linking errors
  (#4880)

Metrics

• http(): Added a new counter for HTTP requests.

  It is activated on stats(level(1));.

  Example metrics:

  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
  syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
  

  (#4805)

• gRPC based destination drivers: Added gRPC request related metrics.

  Affected drivers:

  • opentelemetry()
  • syslog-ng-otlp()
  • bigquery()
  • loki()

  Example metrics:

  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
  

  (#4811)

• New metric to monitor destination reachability

  syslogng_output_unreachable is a bool-like metric, which shows whether a
  destination is reachable or not.

  sum() can be used to count all unreachable outputs, hence the negated name.

  It is currently available for the network(), syslog(), unix-*()
  destinations, and threaded destinations (http(), opentelemetry(), redis(),
  mongodb(), python(), etc.).
  (#4876)

• destinations: Added "syslogng_output_event_retries_total" counter.

  This counter is available for the following destination drivers:

  • amqp()
  • bigquery()
  • http() and all http based drivers
  • java()
  • kafka()
  • loki()
  • mongodb()
  • mqtt()
  • opentelemetry()
  • python() and all python based drivers
  • redis()
  • riemann()
  • smtp()
  • snmp()
  • sql()
  • stomp()
  • syslog-ng-otlp()

  Example metrics:

  syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
  

  (#4807)

• syslogng_memory_queue_capacity

  Shows the capacity (maximum possible size) of each queue.
  Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages.
  Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding
  source log-iw-size() is the upper limit.
  (#4831)

Other changes

• opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior.

  syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward
  the received message because of backpressure. Instead, syslog-ng will block until the
  destination can accept more messages.
  (#4827)

• opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers.
  (#4827)

• APT packages: Dropped Debian Buster support.

  Old packages are still available, but new syslog-ng versions will not
  be available on Debian Buster
  (#4840)

• dbld: AlmaLinux 8 support
  ([#49...

syslog-ng-4.6.0

4.6.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Forwarding logs to Google BigQuery

The bigquery() destination inserts logs to a Google BigQuery table via the
high-performance gRPC API.

Authentication is done via Application Default Credentials.

You can locate your BigQuery table with the project() dataset() and table()
options.

There are two ways to configure your table's schema.

• You can set the columns and their respective type and template with the
  schema() option. The available types are: STRING, BYTES, INTEGER,
  FLOAT, BOOLEAN, TIMESTAMP, DATE, TIME, DATETIME, JSON,
  NUMERIC, BIGNUMERIC, GEOGRAPHY, RECORD, INTERVAL.
• Alternatively you can import a .proto file with the protobuf-schema() option,
  and map the templates for each column.

The performance can be further improved with the workers(), batch-lines(),
batch-bytes(), batch-timeout() and compression() options. By default the
messages are sent with one worker, one message per batch and without compression.

Keepalive can be configured with the keep-alive() block and its time(),
timeout() and max-pings-without-data() options.

Example config:

bigquery(
    project("test-project")
    dataset("test-dataset")
    table("test-table")
    workers(8)

    schema(
        "message" => "$MESSAGE"
        "app" STRING => "$PROGRAM"
        "host" STRING => "$HOST"
        "pid" INTEGER => int("$PID")
    )

    on-error("drop-property")

    # or alternatively instead of schema():
    # protobuf-schema("/tmp/test.proto"
    #                 => "$MESSAGE", "$PROGRAM", "$HOST", "$PID")

    # keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
);

Example .proto schema:

syntax = "proto2";
​
message CustomRecord {
  optional string message = 1;
  optional string app = 2;
  optional string host = 3;
  optional int64 pid = 4;
}

(#4733)
(#4770)
(#4756)

Collecting native macOS system logs

Two new sources have been added on macOS: darwin-oslog(), darwin-oslog-stream().
darwin-oslog() replaced the earlier file source based solution with a native OSLog
framework based one, and is automatically used in the system() source on darwin
platform if the darwinosl plugin is presented.

This plugin is available only on macOS 10.15 Catalina and above, the first version
that has the OSLog API.

darwin-oslog()

This is a native OSLog Framework based source to read logs from the local store of
the unified logging system on darwin OSes.
For more info, see https://developer.apple.com/documentation/oslog?language=objc

The following parameters can be used for customization:

• filter-predicate()
  • string value, which can be used to filter the log messages natively
  • default value: (eventType == 'logEvent' || eventType == 'lossEvent' || eventType == 'stateEvent' || eventType == 'userActionEvent') && (logType != 'debug')
  • for more details, see
    • man log
    • https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Predicates/Articles/pSyntax.html
• go-reverse()
  • boolean value, setting to yes will provide a reverse-ordered log list
    (from latest to oldest)
  • default value: no
• do-not-use-bookmark()
  • boolean value, setting to yes will prevent syslog-ng from continuing to
    feed the logs from the last remembered position after a (re-)start, which means,
    depending on the other settings, the feed will always start from the end/beginning
    of the available log list
  • default value: no, which means syslog-ng will attempt to continue feeding from
    the last remembered log position after a (re-)start
• max-bookmark-distance()
  • integer value, maximum distance in seconds that far an earlier bookmark can point
    backward, e.g. if syslog-ng was stopped for 10 minutes and max-bookmark-distance
    is set to 60 then syslog-ng will start feeding the logs only from the last 60
    seconds at startup, 9 minutes of logs 'will be lost'
  • default value: 0, which means no limit
• read-old-records()
  • boolean value, controls if syslog-ng should start reading logs from the oldest
    available at first start (or if no bookmark can be found)
  • default value: no
• fetch-delay()
  • integer value, controls how much time syslog-ng should wait between reading/sending
    log messages, this is a fraction of a second, where wait_time = 1 second / n, so,
    e.g. n=1 means that only about 1 log will be read and sent in each second,
    and n=1 000 000 means only 1 microsecond (the allowed minimum value now!)
    will be the delay between read/write attempts
  • Use with care, though lower delay time can increase log feed performance, at the
    same time could lead to a heavy system load!
  • default value: 10 000
• fetch-retry-delay()
  • integer value, controls how many seconds syslog-ng will wait before a repeated
    attempt to read/send once it's out of available logs
  • default value: 1
• log-fetch-limit()
  • Warning: This option is now disabled due to an OSLog API bug
    (https://openradar.appspot.com/radar?id=5597032077066240), once it's fixed it
    will be enabled again
  • integer value, that limits the number of logs syslog-ng will send in one run
  • default value: 0, which means no limit

NOTE: the persistent OSLog store is not infinite, depending on your system setting usually,
it keeps about 7 days of logs on disk, so it could happen that the above options cannot
operate the way you expect, e.g. if syslog-ng was stopped for about more then a week it
could happen that will not be able to restart from the last saved bookmark position
(as that might not be presented in the persistent log anymore)

darwin-oslog-stream()

This is a wrapper around the OS command line "log stream" command that can provide a live
log stream feed. Unlike in the case of darwin-oslog() the live stream can contain
non-persistent log events too, so take care, there might be a huge number of log events
every second that could put an unusual load on the device running syslog-ng with this source.
Unfortunately, there's no public API to get the same programmatically, so this one is
implemented using a program() source.

Possible parameters:

• params()
  • a string that can contain all the possible params the macOS log tool can accept
  • see log --help stream for full reference, and man log for more details
  • IMPORTANT: the parameter --style is used internally (defaults to ndjson), so it
    cannot be overridden, please use other sysylog-ng features (templates, rewrite rules, etc.)
    for final output formatting
  • default value: --type log --type trace --level info --level debug,
    you can use `def-osl-stream-params` for referencing it if you wish to keep the
    defaults when you add your own

(#4423)

Collecting qBittorrent logs

The new qbittorrent() source, reads qBittorrent logs from its log file output.

Example minimal config:

source s_qbittorrent {
  qbittorrent(
    dir("/path/to/my/qbittorrent/root/log/dir")
  );
};

The root dir of the qBittorrent logs can be found in the
"Tools" / "Preferences" / "Behavior" / "Log file" / "Save path" field.

As the qbittorrent() source is based on a file() source, all of the file()
source options are applicable, too.

(#4760)

Collecting pihole FTL logs

The new pihole-ftl() source reads pihole FTL (Faster Than Light) logs, which
are usually accessible in the "Tools" / "Pi-hole diagnosis" menu.

Example minimal config:

source s_pihole_ftl {
  pihole-ftl();
};

By default it reads the /var/log/pihole/FTL.log file.
You can change the root dir of Pi-hole's logs with the dir() option,
where the FTL.log file can be found.

As the pihole-ftl() source is based on a file() source, all of the
file() source options are applicable, too.

(#4760)

Parsing Windows Eventlog XMLs

The new windows-eventlog-xml-parser() introduces parsing support for Windows Eventlog XMLs.

Its parameters are the same as the xml() parser.

Example config:

parser p_win {
    windows-eventlog-xml-parser(prefix(".winlog."));
};

(#4793)

Features

• cloud-auth(): Added support for user-managed-service-account() gcp() auth method.

  This authentication method can be used on VMs in GCP to use the linked service.

  Example minimal config, which tries to use the "default" service account:

  cloud-auth(
    gcp(
      user-managed-service-account()
    )
  )
  

  Full config:

  cloud-auth(
    gcp(
      user-managed-service-account(
        name("[email protected]")
        metadata-url("my-custom-metadata-server:8080")
      )
    )
  )
  

  This authentication method is extremely useful with syslog-ng's `google-pubs...

syslog-ng-4.5.0

4.5.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending log messages to OpenObserve

The openobserve-log() destination feeds OpenObserve via the JSON API.

Example config:

openobserve-log(
    url("http://openobserve-endpoint")
    port(5080)
    stream("default")
    user("[email protected]")
    password("V2tsn88GhdNTKxaS")
);

(#4698)

Sending messages to Google Pub/Sub

The google-pubsub() destination feeds Google Pub/Sub via the HTTP REST API.

Example config:

google-pubsub(
  project("syslog-ng-project")
  topic("syslog-ng-topic")
  auth(
    service-account(
      key("/path/to/service-account-key.json")
    )
  )
);

See the Google Pub/Sub documentation to learn more about configuring a service account.
(#4651)

Parsing PostgreSQL logs

The postgresql-csvlog-parser(): add a new parser to process CSV log formatted by
PostgreSQL (https://www.postgresql.org/docs/current/runtime-config-logging.html).
The CSV format is extracted into a set of name-value pairs.
(#4586)

Features

• http(): Added support for using templates in the url() option.

  In syslog-ng a template can only be resolved on a single message, as the same
  template might have different resolutions on different messages. A http batch
  consists of multiple messages, so it is not trivial to decide which message should
  be used for the resolution.

  When batching is enabled and multiple workers are configured it is important to
  only batch messages which generate identical URLs. In this scenario one must set
  the worker-partition-key() option with a template that contains all the templates
  used in the url() option, otherwise messages will be mixed.

  For security reasons, all the templated contents in the url() option are getting
  URL encoded automatically. Also the following parts of the url cannot be templated:

  • scheme
  • host
  • port
  • user
  • password
    (#4663)

• $TRANSPORT: this is a new name-value pair that syslog-ng populates
  automatically. It indicates the "transport" mechanism used to
  retrieve/receive the message. It is up to the source driver to determine
  the value. Currently the following values were implemented:

  BSD syslog drivers: tcp(), udp() & network()

  • rfc3164+tls
  • rfc3164+tcp
  • rfc3164+udp
  • rfc3164+proxied-tls
  • rfc3164+<custom logproto like altp>

  UNIX domain drivers: unix-dgram(), unix-stream()

  • unix-stream
  • unix-dgram

  RFC5424 style syslog: syslog():

  • rfc5426: syslog over udp
  • rfc5425: syslog over tls
  • rfc6587: syslog over tcp
  • rfc5424+<custom logproto like altp>: syslog over a logproto plugin

  Other drivers:

  • otlp: otel() driver
  • mqtt: mqtt() driver
  • hypr-api: hypr-audit-source() driver

  $IP_PROTO: indicate the IP protocol version used to retrieve/receive the
  message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6.
  (#4673)

• network() and syslog() drivers: Added ignore-validity-period as a new flag to ssl-options().

  By specifying ignore-validity-period, you can ignore the validity periods
  of certificates during the certificate validation process.
  (#4642)

• tls() in udp()/tcp()/network() and syslog() drivers: add support
  for a new http() compatible ssl-version() option. This makes the TLS
  related options for http() and other syslog-like drivers more similar. This
  requires OpenSSL 1.1.0.
  (#4682)

• cloud-auth(): Added a new plugin for drivers, which implements different cloud related authentications.

  Currently the only supported authentication is GCP's Service Account for the http() destination.

  Example config:

  http(
    cloud-auth(
      gcp(
        service-account(
          key("/path/to/service-account-key.json")
          audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher")
        )
      )
    )
  );
  

  (#4651)

• csv-parser(): allow parsing the extracted values into matches ($1, $2, $3 ...)
  by omitting the columns() parameter, which normally specifies the column
  names.
  (#4678)

• --check-startup: a new command line option for syslog-ng along with the
  existing --syntax-only. This new option will do a complete configuration
  initialization and then exit with exit code indicating the result. Since
  this also initializes things like network listeners, it will probably not
  work when there is another syslog-ng instance running in the background. The
  recommended use of this option is a dedicated config check container, as
  explained in #4592.
  (#4646)

Bugfixes

• s3: Fixed an ImportError.

  ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object'
  (#4700)

• loki(): fixed mixing non-related label values
  (#4713)

• type hinting: Parsing and casting fractions are now done locale independently.
  (#4702)

• metrics-probe(): Fixed a crash.

  This crash occurred when a metrics-probe() instance was used in multiple source threads,
  like a network() source with multiple connections.
  (#4685)

• flags() argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used.
  No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file.
  (#4689)

• Fix $PROTO value for transport(tls) connections, previously it was set
  to "0" while in reality these are tcp connections (e.g. "6").

  Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source
  drivers (e.g. udp6()/tcp6() or when using ip-protocol(6) for tcp()/udp()).
  Previously V4-mapped addresses would be represented as
  "::ffff:<ipv4 address>". This is not wrong per-se, but would potentially
  cause the same host to be represented in multiple ways. With the fix,
  syslog-ng would just use "<ipv4 address>" in these cases.
  (#4673)

• db-parser(): support nested match characters in @QSTRING@ pattern parser
  (#4717)

Other changes

• LogSource and LogFetcher: additional documentation was added to these
  Python classes to cover explicit source-side batching functionalities (e.g.
  the auto_close_batch attribute and the close_batch() method).
  (#4673)

• rate-limit(): Renamed the template() option to key(), which better communicates the intention.
  (#4679)

• templates: The template-escape() option now only escapes the top-level template function.

  Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting
  escaped the output of each template function, so the parent template function received an
  already escaped string. This was never the intention of the template-escape() option.

  Although this is a breaking change, we do not except anyone having a config that is affected.
  If you have such a config, make sure to follow-up this change. If you need help with it, feel
  free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server.
  (#4666)

• loki(): The timestamp() option now supports quoted strings.

  The valid values are the following, with or without quotes, case insensitive:

  • "current"
  • "received"
  • "msg"
    (#4688)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Cedric Ar...

syslog-ng-4.4.0

4.4.0

Read Axoflow's blog post for more details.
You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending messages between syslog-ng instances via OTLP/gRPC

The syslog-ng-otlp() source and destination helps to transfer the internal representation
of a log message between syslog-ng instances. In contrary to the syslog-ng() (ewmm())
drivers, syslog-ng-otlp() does not transfer the messages on simple TCP connections, but uses
the OpenTelemetry protocol to do so.

It is easily scalable (workers() option), uses built-in application layer acknowledgement,
out of the box supports google service authentication (ADC or ALTS), and gives the possibility
of better load balancing.

The performance is currently similar to ewmm() (OTLP is ~30% quicker) but there is a source
side limitation, which will be optimized. We measured 200-300% performance improvement with a
PoC optimized code using multiple threads, so stay tuned.

Note: The syslog-ng-otlp() source is only an alias to the opentelemetry() source.
This is useful for not needing to open different ports for the syslog-ng messages and other
OpenTelemetry messages. The syslog-ng messages are marked with a @syslog-ng scope name and
the current syslog-ng version as the scope version. Both sources will handle the incoming
syslog-ng messages as syslog-ng messages, and all other messages as simple OpenTelemetry
messages.
(#4564)

Grafana Loki destination

The loki() destination sends messages to Grafana Loki using gRPC.
The message format conforms to the documented HTTP endpoint:
https://grafana.com/docs/loki/latest/reference/api/#push-log-entries-to-loki

Example config:

loki(
    url("localhost:9096")
    labels(
        "app" => "$PROGRAM",
        "host" => "$HOST",
    )

    workers(16)
    batch-timeout(10000)
    batch-lines(1000)
);

Loki requires monotonic timestamps within the same label-set, which makes
it difficult to use the original message timestamp without the possibility
of message loss. In case the monotonic property is violated, Loki discards
the problematic messages with an error. The source of the timestamps can be
configured with the timestamp() option (current, received, msg).

(#4631)

S3 destination

The s3() destination stores log messages in S3 objects.

Minimal config:

s3(
    url("http://localhost:9000")
    bucket("syslog-ng")
    access-key("my-access-key")
    secret-key("my-secret-key")
    object-key("${HOST}/my-logs")
    template("${MESSAGE}\n")
);

Compression

Setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the
created object's key. Use the compresslevel() options to set the level of compression (0-9).

Rotation based on object size

The max-object-size() option configures syslog-ng to finish an object if it reaches a certain
size. syslog-ng will append an index ("-1", "-2", ...) to the end of the object key when
starting a new object after rotation.

Rotation based on timestamp

The object-key-timestamp() option can be used to set a datetime related template, which gets
appended to the end of the object (e.g. "${R_MONTH_ABBREV}${R_DAY}" => "-Sep25"). When a log
message arrives with a newer timestamp template resolution, the previous timestamped object gets
finised and a new one is started with the new timestamp. Backfill messages do not reopen and append
the old object, but starts a new object with the key having an index appended to the old object.

Rotation based on timeout

The flush-grace-period() option sets the number of minutes to wait for new messages to arrive to
objects, if the timeout expires the object is finished, and a new message will start a new with
an index appended.

Upload options

The objects are uploaded with the multipart upload API. Chunks are composed locally. When a chunk
reaches a certain size (by default 5 MiB), the chunk is uploaded. When an object is finished, the
multipart upload gets completed and the chunks are merged by S3.

Upload parameters can be configured with the chunk-size(), upload-threads() and
max-pending-uploads() options.

Additional options

Additional options include region(), storage-class() and canned-acl().

(#4624)

Features

• http(): Added compression ability for use with metered egress/ingress

  The new features can be accessed with the following options:

  • accept-encoding() for requesting the compression of HTTP responses form the server.
    (These are currently not used by syslog-ng, but they still contribute to network traffic.)
    The available options are identity (for no compression), gzip or deflate.
    If you want the driver to accept multiple compression types, you can list them separated by
    commas inside the quotation mark, or write all, if you want to enable all available compression types.
  • content-compression() for compressing messages sent by syslog-ng. The available options are
    identity for no compression, gzip, or deflate.

  Below you can see a configuration example:

  destination d_http_compressed{
    http(url("127.0.0.1:80"), content-compression("deflate"), accept-encoding("all"));
  };
  

  (#4137)

• opensearch: Added a new destination.

  It is similar to elasticsearch-http(), with the difference that it does not have the type()
  option, which is deprecated and advised not to use.
  (#4560)

• Added metrics for message delays: a new metric is introduced that measures the
  delay the messages accumulate while waiting to be delivered by syslog-ng.
  The measurement is sampled, e.g. syslog-ng would take the very first message
  in every second and expose its delay as a value of the new metric.

  There are two new metrics:

  • syslogng_output_event_delay_sample_seconds -- contains the latency of
    outgoing messages
  • syslogng_output_event_delay_sample_age_seconds -- contains the age of the last
    measurement, relative to the current time.
    (#4565)

• metrics-probe: Added dynamic labelling support via name-value pairs

  You can use all value-pairs options, like key(), rekey(), pair() or scope(), etc...

  Example:

  metrics-probe(
    key("foo")
    labels(
      "static-label" => "bar"
      key(".my_prefix.*" rekey(shift-levels(1)))
    )
  );
  

  syslogng_foo{static_label="bar",my_prefix_baz="almafa",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4
  

  (#4610)

• systemd-journal(): Added support for enabling multiple systemd-journal() sources

  Using multiple systemd-journal() sources are now possible as long as each source uses a unique
  systemd namespace. The namespace can be configured with the namespace() option, which has a
  default value of "*".
  (#4553)

• stdout(): added a new destination that allows you to write messages easily
  to syslog-ng's stdout.
  (#4620)

• network(): Added ignore-hostname-mismatch as a new flag to ssl-options().

  By specifying ignore-hostname-mismatch, you can ignore the subject name of a
  certificate during the validation process. This means that syslog-ng will
  only check if the certificate itself is trusted by the current set of trust
  anchors (e.g. trusted CAs) ignoring the mismatch between the targeted
  hostname and the certificate subject.
  (#4628)

Bugfixes

• syslog-ng: fix runtime undefined symbol: random_choice_generator_parser' when executing syslog-ng -V or
  using an example plugin
  (#4615)

• Fix threaded destination crash during a configuration revert

  Threaded destinations that do not support the workers() option crashed while
  syslog-ng was trying to revert to an old configuration.
  (#4588)

• redis(): fix incrementing seq_num
  (#4588)

• python(): fix crash when using Persist or LogTemplate without global python{} code block in configuration
  (#4572)

• mqtt() destination: fix template option initialization
  (#4605)

• opentelemetry: Fixed error handling in case of insert failure.
  (#4583)

• pdbtool: add validation for types of <value> tags

  In patterndb, you can add extra name-value pairs following a match with the tags.
  But the actual value of these name-value pairs were never validated against their types,
  meaning that an incorrect value could be set using this construct.
  (#4621)

• grouping-by(), group-lines(): Fixed a persist name generating error.
  (#4478)

Packaging

• debian: Added tzdata-legacy to Buil...

syslog-ng-4.3.1

4.3.1

This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed
a python-parser() related crash and a metrics related memory leak. It also
added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()
destination.

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    **N...

syslog-ng-4.3.0

4.3.0

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    NOTE: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick)
  • dbi_driver_dir to define an optional DBI driver location for DBD initializati...

syslog-ng-4.2.0

4.2.0

Read Axoflow's blog post for more details.

Highlights

Sending messages to Splunk HEC

The splunk-hec-event() destination feeds Splunk via the HEC events API.

Minimal config:

destination d_splunk_hec_event {
  splunk-hec-event(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
  );
};

Additional options include:

• event()
• index()
• source()
• sourcetype()
• host()
• time()
• default-index()
• default-source()
• default-sourcetype()
• fields()
• extra-headers()
• extra-queries()
• content-type()

The splunk-hec-raw() destination feeds Splunk via the HEC raw API.

Minimal config:

destination d_splunk_hec_raw {
  splunk-hec-raw(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
    channel("05ed4617-f186-4ccd-b4e7-08847094c8fd")
  );
};

(#4462)

Smart multi-line for recognizing backtraces

multi-line-mode(smart):
With this multi-line mode, the inherently multi-line data backtrace format is
recognized even if they span multiple lines in the input and are converted
to a single log message for easier analysis. Backtraces for the following
programming languages are recognized : Python, Java, JavaScript, PHP, Go,
Ruby and Dart.

The regular expressions to recognize these programming languages are
specified by an external file called
/usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on
configure arguments), in a format that is described in that file.

group-lines() parser: this new parser correlates multi-line messages
received as separate, but subsequent lines into a single log message.
Received messages are first collected into streams related messages (using
key()), then collected into correlation contexts up to timeout() seconds.
The identification of multi-line messages are then performed on these
message contexts within the time period.

  group-lines(key("$FILE_NAME")
              multi-line-mode("smart")
        template("$MESSAGE")
        timeout(10)
        line-separator("\n")
  );

(#4225)

HYPR Audit Trail source

hypr-audit-trail() & hypr-app-audit-trail() source drivers are now
available to monitor the audit trails for HYPR applications.

See the README.md file in the driver's directory to see usage information.

(#4175)

ebpf() plugin and reuseport packet randomizer

A new ebpf() plugin was added as a framework to leverage the kernel's eBPF
infrastructure to improve performance and scalability of syslog-ng.

Example:

source s_udp {
        udp(so-reuseport(yes) port(2000) persist-name("udp1")
                ebpf(reuseport(sockets(4)))
        );
        udp(so-reuseport(yes) port(2000) persist-name("udp2"));
        udp(so-reuseport(yes) port(2000) persist-name("udp3"));
        udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};

NOTE: The ebpf() plugin is considered advanced usage so its compilation is
disabled by default. Please don't use it unless all other avenues of
configuration solutions are already tried. You will need a special
toolchain and a recent kernel version to compile and run eBPF programs.

(#4365)

Features

• network source: During a TLS handshake, syslog-ng now automatically sets the
  certificate_authorities field of the certificate request based on the ca-file()
  and ca-dir() options. The pkcs12-file() option already had this feature.
  (#4412)

• metrics-probe(): Added level() option to set the stats level of the generated metrics.
  (#4453)

• metrics-probe(): Added increment() option.

  Users can now set a template, which resolves to a number that modifies
  the increment of the counter. If not set, the increment is 1.
  (#4447)

• python: Added support for typed custom options.

  This applies for python source, python-fetcher source, python destination,
  python parser and python-http-header inner destination.

  Example config:

  python(
    class("TestClass")
    options(
      "string_option" => "example_string"
      "bool_option" => True  # supported values are: True, False, yes, no
      "integer_option" => 123456789
      "double_option" => 123.456789
      "string_list_option" => ["string1", "string2", "string3"]
      "template_option" => LogTemplate("${example_template}")
    )
  );
  

  Breaking change! Previously values were converted to strings if possible, now they are passed
  to the python class with their real type. Make sure to follow up these changes
  in your python code!
  (#4354)

• mongodb destination: Added support for list, JSON and null types.
  (#4437)

• add-contextual-data(): significantly reduce memory usage for large CSV
  files.
  (#4444)

• python(): new LogMessage methods for querying as string and with default values

  • get(key[, default])
    Return the value for key if key exists, else default. If default is
    not given, it defaults to None, so that this method never raises a
    KeyError.

  • get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal'):
    Return the string value for key if key exists, else default.
    If default is not given, it defaults to None, so that this method never
    raises a KeyError.

    The string value is decoded using the codec registered for encoding.
    errors may be given to set the desired error handling scheme.

    Note that currently repr='internal' is the only available representation.
    We may implement another more Pythonic representation in the future, so please
    specify the repr argument explicitly if you want to avoid future
    representation changes in your code.
    (#4410)

• kubernetes() source: Added support for json-file logging driver format.
  (#4419)

• The new $RAWMSG_SIZE hard macro can be used to query the original size of the
  incoming message in bytes.

  This information may not be available for all source drivers.
  (#4440)

• syslog-ng configuration identifier

  A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example:

  @config-id: cfg-20230404-13-g02b0850fc
  

  This keyword can be used for config identification in managed environments, where syslog-ng instances and their
  configuration are deployed/generated automatically.

  syslog-ng-ctl config --id can be used to query the active configuration ID and the SHA256 hash of the full
  "preprocessed" syslog-ng configuration. For example:

  $ syslog-ng-ctl config --id
  cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83)
  

  (#4420)

• syslog-ng: add --config-id command line option

  Similarly to --syntax-only, this command line option parses the configuration
  and then prints its ID before exiting.

  It can be used to query the ID of the current configuration persisted on
  disk.
  (#4435)

• Health metrics and syslog-ng-ctl healthcheck

  A new syslog-ng-ctl command has been introduced, which can be used to query a healthcheck status from syslog-ng.
  Currently, only 2 basic health values are reported.

  syslog-ng-ctl healthcheck --timeout <seconds> can be specified to use it as a boolean healthy/unhealthy check.

  Health checks are also published as periodically updated metrics.
  The frequency of these checks can be configured with the stats(healthcheck-freq()) option.
  The default is 5 minutes.
  (#4362)

• $(format-json) and template functions which support value-pairs
  expressions: new key transformations upper() and lower() have been added to
  translate the caps of keys while formatting the output template. For
  example:

  template("$(format-json test.* --upper)\n")
  

  Would convert all keys to uppercase. Only supports US ASCII.
  (#4452)

• python(), python-fetcher() sources: Added a mapping for the flags() option.

  The state of the flags() option is mapped to the self.flags variable, which is
  a Dict[str, bool], for example:

  {
      'parse': True,
      'check-hostname': False,
      'syslog-protocol': True,
      'assume-utf8': False,
      'validate-utf8': False,
      'sanitize-utf8': False,
      'multi-line': True,
      'store-legacy-msghdr': True,
      'store...