Multiple connections on the same port

[MikeBishop]

@mikkelfj has raised questions about whether we might be able to maintain many connections on the same port. @ianswett has pointed out that other connection-ID-oriented protocols operate all connections on the same port to minimize the use of ephemeral ports. However, if any of those connections are to the same back end, there will be duplicate 5-tuples, and QUIC's current model doesn't support simultaneous handshakes with the same 5-tuple.

If we want to enable this, we would need to have the server's first flight continue using the client-chosen Connection ID, but contain the new Connection ID.

[janaiyengar]

Summarizing editors' discussion: We can use this new connection ID for only 1-RTT encrypted packets. We can further simplify the transport mechanism for server-chosen connection ID, by simply making it part of transport_params.

Note that #713 handles the case of server-chosen connection ID for Stateless Retries.

[igorlord]

That said, we should make sure to discourage the use a single client port for all of its QUIC traffic. Routers do EQMP based a on 5-tuple, and most of them will not be QUIC-aware for years.

[igorlord]

@janaiyengar, could you clarify what you mean by: "We can further simplify the transport mechanism for server-chosen connection ID, by simply making it part of transport_params"?

We are starting our load balancer implementation for IETF QUIC, so details like this are very pertinent.

[MikeBishop]

Basically, the reasoning in the meeting was this: If you want multiple handshakes sharing the same 5-tuple in parallel, then you have to be able to identify which handshake a given packet belongs to. That means the first packet from the server that gives you the server-chosen ID also has to have the client-chosen ID so it can be correlated. The client can start using the server-chosen ID after that, as currently defined.

If the server's choice of connection ID has to be carried in its first packet and that packet has to have the client-chosen connection ID, then it makes sense to just carry the new connection ID in the transport parameters, which are in that packet anyway for the normal case. (Of course, we also need a way to carry it in a HelloRetryRequest, but that's #713.)

[janaiyengar]

What Mike said. Thanks for speaking up about your implementation effort, Igor, that's useful to know. I think at this point, #713 and this issue are the only ones that we need to resolve... and actually for your load balancer implementation, you only really need this issue to be resolved. I'll spin up a PR soon.

[igorlord]

So the server's first 1-RTT packet would have client's connection ID and carry server's connection ID in transport params. But all subsequent server's packets will have server's connection ID? (Or will all subsequent server's packets still have client's connection ID, while client's 1-RTT packets will have Server's connection ID?)

In the former case (only the server's first 1-RTT packet has client's connection ID), what happens if that first packet is lost/reordered? Will the client respond with a Stateless Reset, when it receives server's second 1-RTT packet with server's connection ID?

[janaiyengar]

So, if we carry the server-chosen connection ID in transport params, we can be assured that the client will have this ID by the time it is trying to decrypt 1-RTT-encrypted packets.

So the rules then become:

• All 1-RTT encrypted packets use server-chosen connection ID.
• All other packets carry client-chosen connection ID (*)

(*) The exception to this rule is the Client Initial following a Stateless Retry Packet, but that does not matter from a load balancer's perspective, since the connection ID in Client Initial packets are supposed to be treated by load balancers as client-chosen.

Does that make sense?

[igorlord]

Ok, that makes sense.

One observation that such a multiplexing client would need to be especially careful with QUIC packets featuring unknown Connection IDs.

if we carry the server-chosen connection ID in transport params, we can be assured that the client will have this ID by the time it is trying to decrypt 1-RTT-encrypted packets

If the client receives 1-RTT-encrypted packet from the server before it received a packet with QUIC TLS transport params (server-chosen connection ID), quic-tls ("9.3. Receiving Out-of-Order Protected Frames") says that the client may store those packets and later decrypt them. If the client is in the process of negotiating multiple connections using the same 5-tuple, it will not be able to rely on a 5-tuple to identify a specific QUIC connection it is trying to negotiate and will need to do something more clever. That client would also need to be clever enough to know when it can and cannot send a stateless reset in response to a packet with a connection id it is not aware of.

[janaiyengar]

Good point. In this case, the client would have to maintain a shared buffer for undecrypted packets that arrive on the socket but cannot be assigned to a connection yet, or some such logic. That seems fine as the responsibility of a client that is choosing to multiplex connections over the same 5-tuple.

As for Stateless Reset, that is only sent by the server, and the connection ID is clear in that case.

[mikkelfj]

One observation that such a multiplexing client would need to be especially careful with QUIC packets featuring unknown Connection IDs.

Due to packet spoofing and port guessing this is/should also be a concern with 5-tuples during initial handshake - relying on connection IDs ought to signficantly reduce this concern because it is difficult to guess off path. But I agree that this must be considered regardless.

However, is it strictly relevant to buffer unknown packets. It is much simpler to drop them when there isn't yet an established context to hook them up to and it does protect against random noise/attacks. It will only be relevant in the rearly connection setup. I'm just guessing here, but for flow control it might not be a bad thing to register a dropped packet because clearly there is some performance issue unless it is trivial reordering - and multi-channel reordering typically happens in larger bulks I believe. So, optimizing performance under adverse conditions might not be the best option. A typical attacker could force dropped packets by overflowing network and router queues and here end-point buffering would add some reselience, but then, most end-point buffering would likely be from adversary.

[martinthomson]

With the recent changes to packet protection, packets for different connections are cryptographically distinct. However, that means a bunch of trial decryption for those packets from the server that are reordered. I think that we really want to make sure that we want this feature. It might be something we can defer.

[mikkelfj]

Possible solution:

• The Clear Text packets could carry the client connection ID unencrypted in both directions and expose the server chosen ID as a separate field to inform the client about future 1-RTT transmissions. It would make the server clear text header larger, but that is likely an acceptable cost.
• 1-RTT encrypted packets would carry the server chosen ID in both directions.
• If there is no connection ID in the packets, multiple connections on same port would not be supported.
• Some implementations or configurations may require the connection ID to always be present.
• 0-RTT must also be considered, which I have not done here.

This would simplify connection lookup when the connection ID is always present because there is no need for separate 5-tuple mappings, which require extra work for high speed interfaces such as netmap and might not be possible without on some non-UDP links without extra pre-header.

[martinthomson]

Another solution: don't change anything and have at most one outstanding connection setup running at a time on the same 5-tuple.

[mikkelfj]

That would hurt special cases such as links multiplexing many application clients where there is only one port open. For example, it may be convenient to only have one port for netmap driven server to server, or for non-UDP setups.

[nibanks]

Another possible solution:

Take 1 bit from each header's Type field and designate it the Client Connection ID flag (shown as I here). When set, it indicates the presence of the Client Connection ID field in the header.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|1|I|  Type (6) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                       Connection ID (64)                      +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Packet Number (32)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Version (32)                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                 [Client Connection ID (64)]                   +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Payload (*)                        ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|0|C|I|K|Type(4)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                     [Connection ID (64)]                      +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                 [Client Connection ID (64)]                   +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Packet Number (8/16/32)                ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Protected Payload (*)                   ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Then, a server that changes the Connection ID is required include the Client Connection ID in all packets (long and short) it sends until it receives a packet from the client that uses the new Connection ID.

Also, I am not positive, but I believe these new flags and fields would have constant across versions.

If a client doesn't intent to share the port locally, then they should include the omit_connection_id transport parameter. When that is set, the server isn't required to send the Client Connection ID.

[mikkelfj]

@nibanks I have suggested something similar earlier, except I would make unconditional. The overhead of always having the client ID in early long headers would be small and avoids affecting the crowded first byte. Connection migration also have to carry the old connection ID until peer uses the new ID.

[nibanks]

I agree that for Long Headers it shouldn't be a big deal to have it unconditionally present, but it would need to be optional for Short Headers. And I do feel the Client Connection ID is needed for Short Headers in the 0-RTT scenario, in case the handshake packet in the Long Header is lost or reordered.

If it is optional in one header, I figured it would be good to keep the design uniform and make it optional in both.

[mikkelfj]

I'm not up to speed on 0-RTT, but it seems that in this case a long header could be used until the connection settles on a single ID.

[nibanks]

The server replies to 0-RTT packets (Long Header) with 1-RTT packets (Short Header). If we require the client to acknowledge the new Connection ID before the server sends Short Header packets, then we effectively lose the 0-RTT feature if/when the server changes Connection ID.

[mikkelfj]

@nibanks Right.

Another concern is AEAD encryption based on the connection ID. If the client ID is encrypted, the client cannot use the ID to map to the new server ID. However, if the server continues to reply with client connection ID and carries the new server ID in the packet, or perhaps in some TLS parameter as suggestion by @ianswett, then the client can easily make the link between the connections and connection ID based client to server routing would still work since the client changes ID. Still not sure about the 0-RTT impact.

[nibanks]

@mikkelfj I'm unsure what you mean in your comment above. The client that receives the packet needs to know the previous Connection ID before it can decrypt it. That means it must be unencrypted.

I don't think we should rely on a model where the client is just expected to figure it out by trying to decrypt the packet for all outstanding Connections. I'm not sure how you would ever support encryption offload efficiently in a model like that.

As far as using a TLS parameter, that goes back to the problem of needing the previous Connection ID before decryption. Since all TLS data is inside the encryption, that won't work.

[mikkelfj]

@nibanks clear text packets are encrypted and tagged with the connection ID (cleat text AEAD). Thus you need the connection ID to decrypt and verify. But it was a brain fart because regardless of which connection ID is being used to derive a key for cleartext AEAD, the connection ID is avaiable in the packet as clear text at a fixed location.

With TLS, it could work if the server uses the client connection ID in responses and stores the new connection ID in TLS, whereas the opposite would not work. Or did I miss something?

[nibanks]

Maybe we are crossing paths and misunderstanding each other. Here is the problem I am trying to solve:

1. A client has N new Connections (in the handshake state) to the same remote IP/Port on the same local IP/Port.
2. The client receives a packet (long header or short header) on that local IP/Port with a Connection ID it doesn't know about.

How can the client determine which Connection to deliver that packet to for processing?

Since the packet is encrypted with the Client Connection ID and not the Server Connection ID, it cannot decrypt the packet until it has determined what the Client Connection ID was. Therefore, it cannot rely on any encrypted information to calculate the Client Connection ID.

[mikkelfj]

@nibanks I think we see and intend the same thing, largely.

Speaking only of clear text packets with no TLS magic: a clear text packet is encrypted with a key derived from the connection ID listed in the same packets header. It does not matter what type of connection ID it is because the key can be derived regardless. There is no need to make a connection assocation with internal context before decryption. Once decrypted the new proposed client-ID can be retrieved and the necessary association created. (Because encryption contexts may be cashed, it may matter for efficiency whether client-ID or server-ID is being used, but it does not for being able to decrypt or associate contexts, assuming the extra proposed ID field).

As to carrying the missing ID in TLS: (I'm not fully up to speed here), but it would be necessary to have the client ID visible such that the client can efficiently associate server packets with client context and retrieve content, but I'll skip the details. I believe Ians proposal was to keep using the client ID until a server ID update was received and for all practical purposes start a new connection with that ID, which might add som roundtrip overhead, though I'm not sure.

[nibanks]

@mikkelfj This statement is incorrect:

a clear text packet is encrypted with a key derived from the connection ID listed in the same packets header.

Handshake packets are encrypted with the client's connection ID, which the server doesn't include in the packet currently. See this quote from the TLS section 5.2.1:

Packets that carry the TLS handshake (Initial, Retry, and Handshake) are protected with secrets derived from the connection ID used in the client’s Initial packet.

[mikkelfj]

@nibanks ahh I see. Then some tweaking is needed such as keeping the client ID in return path and optionally list a new server ID as an optional packet field. Isn't this then largely Ians proposal in #713?

[ianswett]

I think we should try: 'Put a new connection ID in a TLS extension or QUIC transport params' and then always have the client be the one to change the routing connection ID first and the server replies once it sees it. I think that works both for stateless reset and for changing connection ID on the SHLO, and it seems a lot more explicit than just allowing the server to change the connection ID at certain points in the handshake without knowing what the new one will be.

@mikkelfj Yes, that's what I was hoping to use to close #713 as well.

[nibanks]

@ianswett That sounds like an acceptable solution to me.

[mikkelfj]

Just being curious, not attempting to shoot anything down, but some things to consider:

• How does Ian's proposal deal with connection migration?
• And are there any issues with 0-RTT and ekstra roundtrips?
• How does client update to server CID work with short packet headers when CID is not present in the header (normally multiple connections per source port would have a CID present, but still).

Presumably a new server CID is to move connection routing to a new server endpoint and 0-RTT ought to already hit the desired endpoint in first go, or it isn't 0-RTT, so is server changed CID a thing for 0-RTT?

[ianswett]

My proposal is only aimed at the handshake, so I don't think it should effect connection migration.

This shouldn't cost any extra round trips, since the TLS messages already include transport params and/or extensions.

For short headers, we're talking post-handshake, in which case I think the NEW_CONNECTION_ID frame is the right way to go.

[mikkelfj]

makes sense

[martinduke]

Update: Ian pointed out to me that this is not necessary. If the client NAT rebinds after sending the CHLO, it will perceive its 5tuple as unchanged and can process the Handshake message with new connection ID. The server will get its expected new connection ID and everything is OK.

If the further corner case where the Server flight is lost, the client initial retransmission will not be mapped correctly, but we'll recover when the Server retransmits the flight.

Original comment follows:

An additional benefit of addressing this issue is to allow handshakes to succeed during a NAT rebinding. This is not a requirement but would be a nice feature.

[martinthomson]

When we discussed this in the context of other things at Melbourne we decided not to address this use case in this version of the protocol. Many of the solutions proposed do not require changes to protocol invariants, so we could fix this in a future version.

[mikkelfj]

Even if the protocol does not consider this in v1, it would be highly useful to ensure that there is linkage between CID SID in all packages, so an implementation can define its own semantics to resolve connection conflicts.