Problems with high throughput in request_response

[n-lebel]

I have a use case which requires sending messages around with very high throughput. With a small number of messages this causes no problems, but as soon as this number exceeds around 200, I start to have consistent OutboundFailure errors. Even if I attempt to keep track of pending messages and attempt resends on OutboundFailure, this doesn't seem to fix the issue. Everything I run is on localhost.

The actual instantiation of the swarm is the following:

 let mut swarm = SwarmBuilder::with_existing_identity(network_config.identity.clone())
            .with_tokio()
            .with_quic()
            .with_behaviour(|key| {
                let local_peer_id = PeerId::from(key.clone().public());
                info!("LocalPeerID: {local_peer_id}");

                let mut kad_config = KadConfig::default();
                kad_config.set_protocol_names(vec![StreamProtocol::new("/agent/connection/1.0.0")]);

                let kad_memory = KadInMemory::new(local_peer_id);
                let kad = KadBehavior::with_config(local_peer_id, kad_memory, kad_config);

                let identity_config = IdentifyConfig::new(
                    "/agent/connection/1.0.0".to_string(),
                    key.clone().public(),
                )
                .with_push_listen_addr_updates(true)
                .with_interval(Duration::from_secs(30));

                let rr_config = RequestResponseConfig::default();
                let rr_protocol = StreamProtocol::new("/agent/message/1.0.0");
                let rr_behavior = RequestResponseBehavior::<RouterData, ()>::new(
                    [(rr_protocol, ProtocolSupport::Full)],
                    rr_config,
                );

                let identify = IdentifyBehaviour::new(identity_config);
                RouterBehaviour::new(rr_behavior, kad, identify)
            })?
            .with_swarm_config(|cfg| cfg.with_idle_connection_timeout(Duration::from_secs(u64::MAX)))
            .build();

        swarm.behaviour_mut().set_server_mode();
        swarm.listen_on(format!("/ip4/0.0.0.0/udp/{}/quic-v1", network_config.port).parse()?)?;

        // Connect to bootstrap node if provided
        if let Some(ref remote) = network_config.bootstrap_node {
            swarm.dial(remote.clone())?;
            info!("Dialed to: {remote:?}");
        }

And a sample error may look something like this: OutboundFailure: Peer: PeerId("12D3KooWHnHFfSNd9V1egU5hJKbJtAeaNgMmDWDxFkiQcjXyMqER") | RequestId: OutboundRequestId(106) | Error: Io(Custom { kind: ConnectionReset, error: Stopped(0) }).

Even when these errors don't appear, the swarm only seems to be able to process a small number of messages before it pauses, seemingly doing nothing. I've played around with some of the buffer sizes in the swarm config but no luck.

The repo is unfortunately private, but I've been able to reproduce a similar behavior on a fork of a public repo here: https://github.com/n-lebel/poc-rust-libp2p-kademlia

Any pointers appreciated!

[jxs]

Hi, and thanks for your interest! @oblique have you experienced anything similar?

[dariusc93]

Although may not be relevant, but have you tried configuring other portions of libp2p::swarm::Config besides the idle connection timeout? Could you also provide the full logs if possible?