RESOLVED: free(): invalid pointer (segmentation fault with systemd plugin and jemalloc)

[kenhys]

Describe the bug

There is a known issue that combination with systemd plugin and jemalloc causes a
crash bug.

There are some feedback from users.

• Fluentd encountered a memory error (free(): invalid pointer)  #1484
• fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-elasticsearch8-2.1 systemd plugin segementation fault causing log not send to the destination #1482
• free(): invalid pointer with latest fluent/fluentd-kubernetes-daemonset:v1-debian-forward-arm64 image #1478

Related issue:

• fluent-plugin-systemd fails with SIGABORT on Ubuntu 21.04 fluent-package-builder#369

• Segmentation fault on Ubuntu 22.04 fluent-plugins-nursery/fluent-plugin-systemd#110

• Segmentation fault after updating from version 1.16.3 to 1.16.4 fluentd-docker-image#378

• fluentd-kubernetes-daemonset segmentation fault #1406

  • fluentd-kubernetes-daemonset segmentation fault #1406 may be different case.

Important

As a workaround, you can avoid a segmentation fault to disable jemalloc memory allocator by specifying empty LD_PRELOAD for each container.

env:
  ...
  LD_PRELOAD: ""

Caution

UPDATE: 2024 Sep 11
fluent-plugin-systemd 1.1.0 (systemd-journal 2.0.0) was shipped to fix this issue.
As v1.17.1-1.1 image bundles fluent-plugin-systemd 1.1.0 or later, please use newer image.

To Reproduce

See above mentioned issues and related bugs.
(Especially fluent/fluentd-docker-image#378 is the simplified to reproduce without k8s)

Expected behavior

Even though jemalloc is enabled, it does not cause a segmentation fault)

Your Environment

- all of v1.16.x or later

Your Configuration

See already mentioned issues.

Your Error Log

See already mentioned issues.

Additional context

To avoid this issue, jemalloc memory allocator will be disabled by default for v1.17.0-1.3 or later and v1.16.5-1.3 or later.

If you don't use systemd plugin at all, you can enable jemalloc memory allocator explicitly via env: parameter.

env:
  ...
  LD_PRELOAD="/usr/lib/libjemalloc.so.2"

[ashie]

ledbettj/systemd-journal#96 will fix this issue.

[kenhys]

This issue was fixed via fluent-plugin-systemd 1.1.0 (which uses systemd-journal 2.0.0)

Now no need to disable jemalloc explicitly.

Please use v1.17.1-1.1 or later images which uses fluent-plugin-systemd 1.1.0 or later.

ref.
ledbettj/systemd-journal#96
ledbettj/systemd-journal#97
fluent/fluentd-docker-image#378

Note

Keep this issue open for a while to inform you.