Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debugging Spontaneous Shutdown #14

Open
dm17 opened this issue Oct 29, 2020 · 7 comments
Open

Debugging Spontaneous Shutdown #14

dm17 opened this issue Oct 29, 2020 · 7 comments

Comments

@dm17
Copy link

dm17 commented Oct 29, 2020

Facts:

  1. I had started dhcpcd too many times because my device names changed, and I was trying to get it working (yes, I should have killed or restarted the old dhcpcd instances instead of doing 'sudo dhcpcd')
  2. I ran exactly: "sudo pkill -f dhcpcd"
  3. The system immediately shut down (it did not reboot; it acted exactly as if I did "sudo shutdown -h now")
  4. I saved everything in /var/log/socklog & faillog for later debugging

Any idea how this could have happened? Any idea where to look for clues?
Thanks!

@Duncaen
Copy link
Member

Duncaen commented Oct 29, 2020

pkill probably matched the runsvdir process because you used -f and stdout and stderr of dhcpcd end in the command line of the runsvdir process because it has no logging service.

should have used sv to send signals to services, first for not accidentially matching the wrong process, second to avoid race conditions with reused pids.

@dm17
Copy link
Author

dm17 commented Oct 29, 2020

pkill probably matched the runsvdir process because you used -f and stdout and stderr of dhcpcd end in the command line of the runsvdir process because it has no logging service.

should have used sv to send signals to services, first for not accidentially matching the wrong process, second to avoid race conditions with reused pids.

You sound like you know what you're talking about :) So I guess you would not mark this as a potential bug? I'm happy to consider is user error for now and not use "pkill -f" again on any Void system :)

@Duncaen
Copy link
Member

Duncaen commented Oct 29, 2020

Its the only possible explanation with the given information, if runsvdir receives the TERM signal, it will exit with 0 and runit-init will transition into stage 3 and do a clean shutdown.

runsvdir has this annoying mechanism where it changes the cmdline to what child processes write to stderr, so services without a log service that write to stderr (and in this case dhcpcd's run file redirects stdout to stderr) will end up in the command line
as a last resort destination for logs.
The space for the log is limited to the size of the command when runsvdir is executed, see the dots in the end of /etc/runit/2 and it acts like a ring buffer.

Here is the output from my system, it doesn't contain the "dhcpcd" string, but its very likely that dhcpcd might end up there when it starts or logs something different:

$ ps -C runsvdir -o command=
runsvdir -P /run/runit/runsvdir/current log: ed br0: IAID 61:a3:0f:48 br0: adding address xxx br0: soliciting an IPv6 router br0: soliciting a DHCP lease br0: offered 192.168.0.120 from 192.168.0.1 br0: probing address 192.168.0.120/24 br0: leased 192.168.0.120 for 3600 seconds br0: adding route to 192.168.0.0/24 br0: adding default route via 192.168.0.1 br0: no IPv6 Routers available ..............................

@ericonr
Copy link
Member

ericonr commented Oct 29, 2020

runsvdir has this annoying mechanism where it changes the cmdline to what child processes write to stderr, so services without a log service that write to stderr (and in this case dhcpcd's run file redirects stdout to stderr) will end up in the command line
as a last resort destination for logs.

We could change void-runit to make the buffer passed in /etc/runit/2 optional.

@heliocat
Copy link

runsvdir has this annoying mechanism where it changes the cmdline to what child processes write to stderr, so services without a log service that write to stderr (and in this case dhcpcd's run file redirects stdout to stderr) will end up in the command line
as a last resort destination for logs.

We could change void-runit to make the buffer passed in /etc/runit/2 optional.

That would break catch-all logging without some kind of alternative.

@Duncaen
Copy link
Member

Duncaen commented Sep 11, 2021

/dev/console output is as good if not better than the command line buffer. (they are both shit)

@heliocat
Copy link

I would agree with you if Void was better about shipping appendant loggers. As it stands though making that change will make an absolute mess out of things unless there's a suitable console reader set up somewhere else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants