[QUIC] Byte mixing and/or native access violation failures #52047
Comments
|
Tagging subscribers to this area: @dotnet/ncl Issue DetailsRepro: [Fact]
public async Task ByteMixingOrNativeAVE_MinimalFailingTest()
{
const int writeSize = 64 * 1024;
const int NumberOfWrites = 512;
byte[] data1 = new byte[writeSize * NumberOfWrites];
byte[] data2 = new byte[writeSize * NumberOfWrites];
Array.Fill(data1, (byte)1);
Array.Fill(data2, (byte)2);
Task t1 = RunTest(data1);
Task t2 = RunTest(data2);
async Task RunTest(byte[] data)
{
await RunClientServer(
iterations: 20,
serverFunction: async connection =>
{
await using QuicStream stream = await connection.AcceptStreamAsync();
byte[] buffer = new byte[data.Length];
int bytesRead = await ReadAll(stream, buffer);
Assert.Equal(data.Length, bytesRead);
AssertArrayEqual(data, buffer);
for (int pos = 0; pos < data.Length; pos += writeSize)
{
await stream.WriteAsync(data[pos..(pos + writeSize)]);
}
await stream.WriteAsync(Memory<byte>.Empty, endStream: true);
await stream.ShutdownCompleted();
},
clientFunction: async connection =>
{
await using QuicStream stream = connection.OpenBidirectionalStream();
for (int pos = 0; pos < data.Length; pos += writeSize)
{
await stream.WriteAsync(data[pos..(pos + writeSize)]);
}
await stream.WriteAsync(Memory<byte>.Empty, endStream: true);
byte[] buffer = new byte[data.Length];
int bytesRead = await ReadAll(stream, buffer);
Assert.Equal(data.Length, bytesRead);
AssertArrayEqual(data, buffer);
await stream.ShutdownCompleted();
}
);
}
await (new[] { t1, t2 }).WhenAllOrAnyFailed(millisecondsTimeout: 1000000);
}For me it now fails on every run with either dotnet.exe crashing with exit code -1073741819 = 0xc0000005 (Access Violation), or Assert failing on receiving wrong bytes -- either random bytes or bytes from parallel connection For native Access Violation, crash dump shows it happening on memcpy on sending data I would suspect that we treat our send buffers in some wrong way. We do pin them, but maybe we don't pin them long enough. From msquic docs: "As long as there is room to buffer the data, MsQuic will copy the data locally and then immediately complete the send back to the app, via the QUIC_STREAM_EVENT_SEND_COMPLETE event. If there is no room to copy the data, then MsQuic will hold onto the buffer until there is room."
|
dotnet-issue-labeler
bot
added
the
untriaged
CarnaViire
removed
the
untriaged
|
@nibanks Have you ever encountered an error like this? Is there something that we should be aware of when passing buffers to msquic? Will QUIC_STREAM_EVENT_SEND_COMPLETE always mean that msquic is done with our buffer? |
|
When do you unpin the data? QUIC_STREAM_EVENT_SEND_COMPLETE is the last time MsQuic uses the buffer, and its safe to unpin only after that is received for a specific buffer. |
|
It would be nice IMHO if the send function tells you directly if the buffer was consumed. Always waiting for event and going through the wait state seems to add lot of processing for normal case. |
|
@ThadHouse from what I see, now we always wait for QUIC_STREAM_EVENT_SEND_COMPLETE before unpinning |
|
Then it's likely we'll need logs to make any more progress. |
|
I think #52368 should address this. can you please check @CarnaViire and close this if everything looks ok? |
|
I haven't encountered these problems anymore on Windows, so it seems fixed. @ManickaP could you please check that everything is ok on Linux side too? |
|
Confirmation on Linux would be good. However, I think we should keep it open to track the suggestions from #52368. (unless we want to open new issue for cases when Stream is disposed in middle of IO) |
|
Triage: we haven't seen this in a very long time, it should be fixed, closing |
ghost
locked as resolved and limited conversation to collaborators
Repro:
For me it now fails on every run with either dotnet.exe crashing with exit code -1073741819 = 0xc0000005 (Access Violation), or Assert failing on receiving wrong bytes -- either random bytes
or bytes from parallel connection
For native Access Violation, crash dump shows it happening on memcpy on sending data
I would suspect that we treat our send buffers in some wrong way. We do pin them, but maybe we don't pin them long enough. From msquic docs: "As long as there is room to buffer the data, MsQuic will copy the data locally and then immediately complete the send back to the app, via the QUIC_STREAM_EVENT_SEND_COMPLETE event. If there is no room to copy the data, then MsQuic will hold onto the buffer until there is room."
Affected tests:
The text was updated successfully, but these errors were encountered: