http3::SendMessage::send_data can stall

[mxinden]

Problem

The QUIC layer (neqo-transport) notifies the http3 layer (neqo-http3) when new data can be written to a stream via ConnectionEvent::SendStreamWritable. For example when the maximum stream data is increased by the remote.

neqo/neqo-transport/src/send_stream.rs

Lines 1206 to 1216 in c004359

	pub fn set_max_stream_data(&mut self, limit: u64) {
	if let SendStreamState::Ready { fc, .. } | SendStreamState::Send { fc, .. } =
	&mut self.state
	{
	let stream_was_blocked = fc.available() == 0;
	fc.update(limit);
	if stream_was_blocked && self.avail() > 0 {
	self.conn_events.send_stream_writable(self.stream_id);
	}
	}
	}

Note that the event is only emitted in case the stream was previously blocked on the limit, not each time the limit increases.

This event is surfaced through the http3 layer to the user as a Http3ServerEvent::DataWritable or Http3ClientEvent::DataWritable. The user can then write data to the stream, ultimately calling neqo_http3::SendMessage::send_data.

neqo/neqo-http3/src/send_message.rs

Line 168 in c004359

fn send_data(&mut self, conn: &mut Connection, buf: &[u8]) -> Res<usize> {

send_data will check with the underlying neqo_transport::SendStream how much bytes are available to send. To make sure it can send at least a header & a 1 byte payload, it checks that the available bytes are not <=2.

neqo/neqo-http3/src/send_message.rs

Lines 177 to 182 in c004359

	let available = conn
	.stream_avail_send_space(self.stream_id())
	.map_err(|e| Error::map_stream_send_errors(&e.into()))?;
	if available <= 2 {
	return Ok(0);
	}

If it is <=2 it returns with Ok(0), effectively wasting the DataWritable event.

In this case:

• The neqo_transport::SendStream has available bytes, even though small (0< available <=2), and thus is waiting for input from neqo_http3::SendMessage. It will not emit more DataWritable events e.g. on maximum stream data increase, given that it has data available (even though small 0< available <=2).
• The neqo_http3::SendMessage is waiting for neqo_transport::SendStream to have more bytes available, i.e. it is waiting for a DataWritable event.

Thus sending of data on the stream stalls indefinitely.

Potential solutions

1. Emit a DataWritable event on each increase of neqo_transport::SendStream::avail() (connection limit, stream limit, tx_buf limit).
   • This can lead to a lot of events and thus a lot of resulting work.
2. Always use up the available bytes of neqo_transport::SendStream in neqo_http3::SendMessage::send_data. If it is <=2, buffer the remaining in the already existing SendMessage::stream BufferedStream. Given that the neqo_transport::SendStream buffer is always used up, new DataWritable events will be emitted by neqo_transport::SendStream once more bytes are available to be sent.
3. Add API to neqo_transport::SendStream for neqo_http3::SendMessage to specify StreamWritable event low-watermark (@martinthomson).
4. ... other suggestions?

Related

• While this issue can arise in general, it does so more often with fix(http3): only add to stream_has_pending_data on has_data_to_send #1793. Before fix(http3): only add to stream_has_pending_data on has_data_to_send #1793 superfluous DataWritable events would be emitted, which, when timed correctly, would prevent the above.
• Other users of SendStream::send might stall as well. Does Firefox' WebTransport implementation?