-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Concurrent uploading files may cause System.Net.Http.Headers.HttpHeaders.ReadStoreValues IndexOutOfRange exception #27018
Comments
Thank you for your feedback. This has been routed to the support team for assistance. |
See: dotnet/runtime#65379 . This might be a dotnet 6 issue. |
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage. Issue DetailsLibrary name and versionAzure.Storage.Blobs 12.10.0 Describe the bugI'm having a random issue with uploading many files in parallel. It seems to be totally random, and from the related issues I find its a lock/thread safety issue. See dotnet/runtime#61798 , where I have posted a duplicate of this report.
It happens in a dotnet 6 console app running from I'm using Source code (slightly modified): (Inspired by this official example: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-scalable-app-upload-files?tabs=dotnet ) var blobContainerClient = new BlobContainerClient(new Uri(assetUploadToken.UploadDestination));
BlobUploadOptions uploadOptions = new BlobUploadOptions()
{
TransferOptions = new StorageTransferOptions()
{
MaximumConcurrency = 8
}
};
var tasks = new Queue<Task<Response<BlobContentInfo>>>();
foreach (string sourceFilePath in Directory.GetFiles(fileDirectoryPath, "*", searchOption)
.Where(x => regexFileExtensions.Match(x).Success))
{
var fileToUpload = new FileInfo(sourceFilePath);
var blobTargetPath = $"my-files";
var blobClient = blobContainerClient.GetBlobClient(blobTargetPath);
var uploadTask = blobClient.UploadAsync(fileToUpload.FullName, uploadOptions);
tasks.Enqueue(uploadTask);
}
// Run all the tasks asynchronously. The `uploadOptions` will handle concurrency.
await Task.WhenAll(tasks);
Console.WriteLine($"Uploaded {tasks.Count} files."); Stack Trace:
Expected behaviorUploading succeeds without issue. Actual behaviorUploading randomly fails (maybe in 1/10 uploading sessions) with the stack trace given in the description. Full stack trace
Reproduction StepsSee the code in the description. I upload around 1000 tiny files using this script, and it happens just rarely enough to having troubles recreating it. Environmentdotnet --info:Hosting information: Runtime Environment: Host (useful for support): Other infoWindows Server 2016
|
@jaschrep-msft James can you please look into this. |
I believe we're hitting the same issue, but with downloads instead of uploads. Is Our error looks like this:
|
@jaschrep-msft Could you take a look at this and see if this is a Core issue? |
@jaschrep-msft / @amnguye any updates? We've stopped using the same
|
Sorry for the late response. It's probably been acknowledged that the fix was made already on the dotnet side for the concurrency issue regarding HttpHeaders however the fix is in .NET 7.0 and it has been communicated that it will not be backported to .NET 6.0 Another mitigation that can be taken (aside from upgrading to .NET 7.0) is to provide a custom HttpMessageHandler that will prevent the race condition, which can be provided through the
|
Library name and version
Azure.Storage.Blobs 12.10.0
Describe the bug
I'm having a random issue with uploading many files in parallel. It seems to be totally random, and from the related issues I find its a lock/thread safety issue. See dotnet/runtime#61798 , where I have posted a duplicate of this report.
It happens in a dotnet 6 console app running from
dotnet run
on a TeamCity Agent. It seems very random and triggers in about 1/10 sessions when I upload around 1000 tiny files to the blob container. I do NOT haveNew Relic Monitoring
installed, as has been an identified issue in the other similar report ( #25501 ) .I'm using
Azure.Storage.Blobs v12.10.0
(Latest at the time of writing), and no other NuGet packages.Source code (slightly modified): (Inspired by this official example: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-scalable-app-upload-files?tabs=dotnet )
Stack Trace:
Expected behavior
Uploading succeeds without issue.
Actual behavior
Uploading randomly fails (maybe in 1/10 uploading sessions) with the stack trace given in the description.
Full stack trace
Reproduction Steps
See the code in the description. I upload around 1000 tiny files using this script, and it happens just rarely enough to having troubles recreating it.
Environment
dotnet --info:
Hosting information:
.NET SDK (reflecting any global.json):
Version: 6.0.101
Commit: ef49f6213a
Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\6.0.101\
Host (useful for support):
Version: 6.0.1
Commit: 3a25a7f1cc
Other info
Windows Server 2016
TeamCity 2021.2.2
The text was updated successfully, but these errors were encountered: