How to limit libp2p memory usage?

[Wiezzel]

I am running a network of libp2p nodes, which use the following protocols:

    identify: identify::Behaviour,
    kademlia: Kademlia<MemoryStore>,
    autonat: Toggle<autonat::Behaviour>,
    relay: RelayClient,
    dcutr: dcutr::Behaviour,
    request: request_response::Behaviour<MessageCodec<T>>,
    gossipsub: gossipsub::Behaviour,
    ping: ping::Behaviour,

Notes:

• identify is using interval 60s and pushing listen addr updates,
• kademlia is bootstrapped every 5 minutes, and used for querying peers when addr is not known,
• relay and dcutr are not actively used at the moment,
• gossipsub is used for most communication - there is a single topic that every peer is subscribed to and everyone pushes one message every 10s,
• request_response is used for additional messages, but I have noticed the problem even for nodes which don't send or receive any requests, so it can probably be ignored.

The problem which I have is that the process seems to be using too much memory. The usage grows ~15MB/sec seemingly without any limit (process gets killed by OOM at 10 GB). I've tried adding libp2p_memory_connection_limits::Behaviour, but it doesn't seem to help. Here is a flamegraph which I have generated using jemalloc. If I read it correctly (high chance I am not though), it's mostly libp2p_kad that's allocating memory.

How can I limit the memory usage of my node to some reasonable value (e.g. 4 GB)?

libp2p version: 0.52.3

[mxinden]

Thank you for the report! Much appreciated, especially the framegraph. I will take a more in-depth look tomorrow.

If you can, in the meantime, are you able to share more code? Also what network topology do you have? Are you connecting to a known network, e.g. IPFS or Ethereum?

[Wiezzel]

That's a private repo, so I'm not sure if I can share more, but I will check. We're not connecting to any known network, such as IPFS or Ethereum, it's an isolated network. The network looks like this:

• Two bootnodes with publicly known peer IDs and addresses which just run, not doing anything special.
• One special node called scheduler which listens on gossipsub messages and also send and receives some requests.
• Many worker nodes which connect to the bootnodes, regularly (every 10s) publish gossipsub messages and also exchange some requests with the scheduler and clients. They all have public addresses.
• Many client nodes which connect to the bootnodes, listen for gossipsub messages and exchanges some requests with the scheduler and workers. The are not required to have public addresses and most of them probably don't.
  All the nodes perform kademlia bootstrap every 5 minutes and start kademlia queries if they don't know the address of a node they want to send a request to.

[Wiezzel]

@mxinden here you can view interactive version of the flame graph (seems like GitHub butchered this one when uploading).

[thomaseizinger]

Can you do some investigation with heaptrack to see where the allocations are coming from? Do you have a custom MemoryStore config?

[Wiezzel]

@mxinden Only rust-libp2p. Current nodes have v0.52, but it is possible that there are some old ones which are still running v0.50 (we don't have control over them, it's an open network).

[thomaseizinger]

@mxinden So this is a backpressure issue, correct? We shouldn't overload a particular ConnectionHandler if it is struggleing to process our messages.

[Wiezzel]

@thomaseizinger Yes, that seems to be the problem. The send_queue in gossipsub's EnabledHandler is unbounded and when it receives behaviour events at a higher rate than it's able to send, this queue grows indefinitely.

What's still to find out is why in my case the messages were flowing in so fast, but couldn't be sent.

[mxinden]

I see two solutions:

1. Front or tail drop in EnabledHandler once send_queue reaches a certain size.
2. Implement backpressure between Gossipsubs NetworkBehaviour and ConnectionHandler and only forward from NetworkBehaviour to ConnectionHandler when the latter can handle another message. I.e. drop messages for a specific peer in NetworkBehaviour instead of as suggested in (1) in ConnectionHandler.

[mxinden]

@Wiezzel after our call last Friday, any updates? Where you able to find the source for the intermittent bug, i.e. the old messages?

[Wiezzel]

@mxinden Sorry, I prefer not to work on weekends. 😛 But I will investigate this further.

[Wiezzel]

@mxinden I've tried a couple of times to recreate this with a very small network where some nodes run libp2p 0.50 and some run 0.52, but without any luck. Maybe I would have to add some nodes with very slow connection to fully reflect the original circumstances, but I don't see any easy way to do this.