slogfiles grow without bound, contrary to devops norms a la logrotate

[dckc]

Describe the bug

By setting $SLOGFILE or $SOLO_SLOGFILE, operators can produce detailed diagnostics about swingset events. Unfortunately, these files grow without bound, which is awkward for the node operators and for data analysis.

Expected behavior

logrotate is a norm for splitting logfiles into manageable pieces.

Ideally, these files would be split on block boundaries.

Additional context

originally noted in #3630

cc @rowgraus @warner @FUDCo

[dckc]

@michaelfig I'm inclined to take the SwingSet label off this one, leaving it on the cosmic-swingset agenda.

[dckc]

adding @dtribble in a customer role assignment

[dckc]

In order to investigate what seems like unnecessary round-trips in the governance demo (#3869), I looked into the options here and found a couple that seem workable plus others that I would rather avoid:

named pipe + multilog / svlogd

This option appeals to me:

Have each worker log to stdout, but connect each stdout to a utility that automatically spools and rotates logs based on size or time. multilog and svlogd are examples of such. -- pilcrow Feb '13

add log rotation to swingset slogger with logrotate-stream npm package

logrotate-stream - npm seems to be designed just for this case. It would only need to go in the controller, not the kernel, right?

ad-hoc script with named pipes: let's not

I suppose this is possible...

• linux - Rotate logs of a dumb non-interactive application - Server Fault

... but the docs for multilog make it clear that there are lots of corner cases...

file descriptor + systemd: unlikely

I went to flesh out the idea o f using SLOGFILE=&7 to mean: log to file descriptor 7, with the expectation that systemd / journald would listen there and provided log rotation. But I don't see any support for anything like that in systemd.unit docs nor systemd-journald.service.

• daemontools - Can systemd manage a pipeline? - Server Fault
• logging - How can I send a message to the systemd journal from the command line? - Server Fault

[Tartuffo]

Solved by @michaelfig (Flight Recorder #3742)