Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manually watch and parse journal log files #24

Open
dead-claudia opened this issue Aug 15, 2024 · 0 comments
Open

Manually watch and parse journal log files #24

dead-claudia opened this issue Aug 15, 2024 · 0 comments

Comments

@dead-claudia
Copy link
Owner

This will bring a nice perf boost as I can decode a lot less, and it'll kick to the curb one of two reasons I interface with libsystemd at all. (The other, sd_notify, is very easy to reimplement.)

I can then make a few new extremely-useful changes:

  • I can count all fields without needing to even inspect any of them. (The reason why I'm not listing fields right now is because I don't want the allocating and copying nonsense that sd_journal_* field iterators currently do.)
  • I can completely drop the libsystemd dependency. This simplifies building by a lot, and I don't have to worry as hard about linking.
    • It also lets me use musl libc and make binaries entirely self-contained.
  • I can report generic per-field lengths and not just message lengths and such.
  • I can do streaming decompression of large entries.
    • I can reuse a shared thread-local buffer for everything, making it zero-copy at runtime.
    • Message filters (Add message filter support #19) can use streaming regexps, though that'll unfortunately require me to roll my own regexp implementation (the POSIX API offers no incremental processing capability).
    • Does mean I have to take on three dependencies: lz4, xz, and zstd.
  • I can report the true size of all ingested journal data, in both disk size and actual processed size.
  • I can track all message fields and not just the first. (Q: multiline support? #23 is an example of how this could matter.)

Useful links:

Note: This requires inotify and file reading. Be sure to only perform one inotify read at a time. Also, inotify only detects FS modifications - epoll is still required for detecting readability, and so I'll have to use both in concert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant