Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RESOLVED: Workaround: disable jemalloc when image crashed (segmentation fault) with systemd plugin #385

Open
kenhys opened this issue Aug 8, 2024 · 2 comments
Labels

Comments

@kenhys
Copy link
Contributor

kenhys commented Aug 8, 2024

Describe the bug

There is a known issue that combination with systemd plugin and jemalloc causes a
crash bug.

There are some feedback from users.

Related issue:

Important

As a workaround, you can avoid a segmentation fault to disable jemalloc memory allocator by specifying empty LD_PRELOAD for each container.

docker run --rm -it -e LD_PRELOAD="" ....

Caution

UPDATE: 2024 Sep 11
fluent-plugin-systemd 1.1.0 (systemd-journal 2.0.0) was shipped to fix this issue.
Please use fluent-plugin-systemd 1.1.0 or later.

To Reproduce

See above mentioned issues and related bugs.
(Especially #378 is the simplified to reproduce without k8s)

Expected behavior

Even though jemalloc is enabled, it does not cause a segmentation fault)

Your Environment

- all of v1.16.x or later

Your Configuration

See already mentioned issues.

Your Error Log

See already mentioned issues.

Additional context

To avoid this issue, jemalloc memory allocator will be disabled by default for v1.17.0-1.3 or later and v1.16.5-1.3 or later for fluent-kubernetes-daemonset images.

If you don't use systemd plugin at all, no need to disable jemalloc memory allocator explicitly.

NOTE: recent version of jemalloc is not solution for us. checked with:

@kenhys kenhys pinned this issue Aug 8, 2024
@kenhys kenhys added bug help wanted Need contributors labels Aug 8, 2024
@ashie
Copy link
Member

ashie commented Aug 30, 2024

ledbettj/systemd-journal#96 will fix the root cause.

@ashie ashie removed the help wanted Need contributors label Aug 30, 2024
kenhys added a commit to kenhys/fluentd-kubernetes-daemonset that referenced this issue Sep 11, 2024
systemd-journal 2.0.0 fixes segmentation fault with jemalloc memory
allocator.
fluent-plugin-systemd 1.1.0 adopts systemd-journal 2.0.0 or later.

fluent/fluentd-docker-image#378
fluent/fluentd-docker-image#385
fluent#1517

Signed-off-by: Kentaro Hayashi <[email protected]>
@kenhys
Copy link
Contributor Author

kenhys commented Sep 11, 2024

This issue was fixed via fluent-plugin-systemd 1.1.0 (which uses systemd-journal 2.0.0)

Now no need to disable jemalloc explicitly.

Please use fluent-plugin-systemd 1.1.0 or later.

ref.
ledbettj/systemd-journal#96
ledbettj/systemd-journal#97
#378

Note

Keep this issue open for a while to inform you.

@kenhys kenhys changed the title Workaround: disable jemalloc when image crashed (segmentation fault) with systemd plugin RESOLVED: Workaround: disable jemalloc when image crashed (segmentation fault) with systemd plugin Sep 11, 2024
kenhys added a commit to fluent/fluentd-kubernetes-daemonset that referenced this issue Sep 11, 2024
systemd-journal 2.0.0 fixes segmentation fault with jemalloc memory
allocator.
fluent-plugin-systemd 1.1.0 adopts systemd-journal 2.0.0 or later.

fluent/fluentd-docker-image#378
fluent/fluentd-docker-image#385
#1517

Signed-off-by: Kentaro Hayashi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants