Increase fd limits (and warn as we approach them) #7237
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cdecker
approved these changes
Apr 21, 2024
rustyrussell
force-pushed
the
increase-fd-limits
branch
from
4640693
to
f4c76a4
Compare
endothermicdev
force-pushed
the
increase-fd-limits
branch
from
f4c76a4
to
f858d34
Compare
|
Added the connecting map to the memleak detection callback in connectd. Also included a driveby fix for tracepoints memory initialization. |
|
Fixing up test assumption that hard limit > soft limit (on CI it seems to be 65536 for both!). |
rustyrussell
force-pushed
the
increase-fd-limits
branch
from
f858d34
to
e663658
Compare
|
This works fine for me locally, but the CI seems unable to raise the fd limit after pytest constrains it. I noticed if I run pytest as root, it exhibits the same behavior. |
|
would be nice to test this in docker, kubernetes, and the like to make sure it doesn't have the same constraints |
Agreeed. Meanwhile, this doesn't make it worse if we can't increase fd limits, and at least now we'll get a warning if it seems limiting. |
So we can log when we hit fd limits on accept/recvmsg. Signed-off-by: Rusty Russell <[email protected]>
This can happen if we're totally out of fds, but previously we gave no log message indicating this! Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
I thought I was going to want to have a convenient way of counting these, but it turns out unnecessary. Still, this is slightly more efficient and simple, so I am including it. Signed-off-by: Rusty Russell <[email protected]>
We use a crude heuristic: if we were trying to contact them, it's a "deliberate" connection, and should be preserved. Changelog-Changed: connectd: prioritize peers with channels (and log!) if we run low on file descriptors. Signed-off-by: Rusty Russell <[email protected]>
… channels. 1024 is a common limit, and people are starting to hit that many channels, so we should increase it: twice the number of channels seems reasonable, though we only do this at restart time. Changelog-Changed: lightningd: we now try to increase the number of file descriptors, if it's less than twice the number of channels at startup (and log if we cannot!). Signed-off-by: Rusty Russell <[email protected]>
They're cheap. The 2x channels heuristic is nice, but does assume they restart every so often. If someone hits 64k connections I would like to know anyway! Signed-off-by: Rusty Russell <[email protected]>
rustyrussell
force-pushed
the
increase-fd-limits
branch
from
e663658
to
995fefd
Compare
We expect: UNUSUAL.*WARNING: we have 1 channels but can file descriptors limited to 65536 We get: lightningd: WARNING: we have 1 channels but can file descriptors limited to 32768! This is strange, since the first line is from Python's hard limit. Presumably something is restricting the fd limit of children Signed-off-by: Rusty Russell <[email protected]>
rustyrussell
force-pushed
the
increase-fd-limits
branch
from
995fefd
to
71c039e
Compare
vincenzopalazzo
approved these changes
May 9, 2024
There was a problem hiding this comment.
ACK 71c039e
Passing the github CI should also give us some clue about the limits in VMs?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On many systems, the default fd limit is 1024, and we've got users with over 900 channels. But that's a soft limit, and you can simply ask for an increase!
Also, we should log when we are running low, so people don't have to ask me what to do!