[BUG] BlockBlobsClient.StageBlockAsync failing.

[wade-dp]

Describe the bug
We are using the BlockBlobClient to store WAV files.
The data for the WAV file is Staged in blocks of < 4MB using calls to 'StageBlockAsync'.
Once complete we stage another block containing the WAV header.
Finally we commit the blocks in the correct order with the header first using 'CommitBlockListAsync'.

When we come to commit the block list, it is failing due to an invalid block list.
Investigation showed that after staging 4 blocks, a call to 'GetBlockList' only returned 2 blocks.
The responses to the 'StageBlockAsync' seemed to indicate success. (Status = 201, Reason = Created)

Expected behavior
The call to 'CommitBlockListAsync' to complete successfully, without throwing an exception.

Actual behavior (include Exception or Stack Trace)
Exception thrown when trying to commit the blocks:

The specified block list is invalid.
RequestId:02bc8058-101e-0060-80da-a09b76000000
Time:2021-09-03T15:41:33.3588351Z
Status: 400 (The specified block list is invalid.)
ErrorCode: InvalidBlockList

Content:

InvalidBlockListThe specified block list is invalid.

RequestId:02bc8058-101e-0060-80da-a09b76000000
Time:2021-09-03T15:41:33.3588351Z

Headers:
x-ms-request-id: 02bc8058-101e-0060-80da-a09b76000000
x-ms-client-request-id: d33f9865-baae-4c50-898d-d13f3b4b8469
x-ms-version: 2020-08-04
x-ms-error-code: InvalidBlockList
Content-Length: 221
Content-Type: application/xml
Date: Fri, 03 Sep 2021 15:41:32 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

1. Stage 4 Blocks Each with approximate size 3984589 bytes, Block ID's are base 64 encoded and of equal length.

using (var ms = new MemoryStream(File.ReadAllBytes(state.BufferFileFullName())))
{
var blockName = state.CreateBlockListEntry();
_blocklist.Add(blockName);

try
{
    var result = await state.BlobClient.StageBlockAsync(blockName, ms, cancellationToken).ConfigureAwait(false)					
    _logs.StageBlock(blockName, result.GetRawResponse().ToString());
}
catch (Exception e)
{
    _logs.ErrorStagingBlock(blockName, e);
}

}

2. Call 'CommitBlockListAsync'

await state.BlobClient.CommitBlockListAsync(_blockList, new BlobHttpHeaders() { ContentType = "audio/wav" }, cancellationToken: CancellationToken.None).ConfigureAwait(false);

Environment:

• Azure.Storage.Blobs - v12.9.1
• Azure CloudService - OS Family: Windows Server 2019, .Net Framework 4.8
• Visual Studio 16.9.3

[jsquire]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details

---

Describe the bug
We are using the BlockBlobClient to store WAV files.
The data for the WAV file is Staged in blocks of < 4MB using calls to 'StageBlockAsync'.
Once complete we stage another block containing the WAV header.
Finally we commit the blocks in the correct order with the header first using 'CommitBlockListAsync'.

When we come to commit the block list, it is failing due to an invalid block list.
Investigation showed that after staging 4 blocks, a call to 'GetBlockList' only returned 2 blocks.
The responses to the 'StageBlockAsync' seemed to indicate success. (Status = 201, Reason = Created)

Expected behavior
The call to 'CommitBlockListAsync' to complete successfully, without throwing an exception.

Actual behavior (include Exception or Stack Trace)
Exception thrown when trying to commit the blocks:

The specified block list is invalid.
RequestId:02bc8058-101e-0060-80da-a09b76000000
Time:2021-09-03T15:41:33.3588351Z
Status: 400 (The specified block list is invalid.)
ErrorCode: InvalidBlockList

Content:

InvalidBlockListThe specified block list is invalid.

RequestId:02bc8058-101e-0060-80da-a09b76000000
Time:2021-09-03T15:41:33.3588351Z

Headers:
x-ms-request-id: 02bc8058-101e-0060-80da-a09b76000000
x-ms-client-request-id: d33f9865-baae-4c50-898d-d13f3b4b8469
x-ms-version: 2020-08-04
x-ms-error-code: InvalidBlockList
Content-Length: 221
Content-Type: application/xml
Date: Fri, 03 Sep 2021 15:41:32 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

1. Stage 4 Blocks Each with approximate size 3984589 bytes, Block ID's are base 64 encoded and of equal length.

using (var ms = new MemoryStream(File.ReadAllBytes(state.BufferFileFullName())))
{
var blockName = state.CreateBlockListEntry();
_blocklist.Add(blockName);

try
{
    var result = await state.BlobClient.StageBlockAsync(blockName, ms, cancellationToken).ConfigureAwait(false)					
    _logs.StageBlock(blockName, result.GetRawResponse().ToString());
}
catch (Exception e)
{
    _logs.ErrorStagingBlock(blockName, e);
}

}

2. Call 'CommitBlockListAsync'

await state.BlobClient.CommitBlockListAsync(_blockList, new BlobHttpHeaders() { ContentType = "audio/wav" }, cancellationToken: CancellationToken.None).ConfigureAwait(false);

Environment:

• Azure.Storage.Blobs - v12.9.1
• Azure CloudService - OS Family: Windows Server 2019, .Net Framework 4.8
• Visual Studio 16.9.3

Author:	ppekrol
Assignees:	-
Labels:	Storage, Service Attention, Client, needs-triage, customer-reported, question, needs-team-attention
Milestone:	-

[amishra-dev]

@wade-dp there are 2 questions here

1. GetBlockList not showing all the staged blocks. For this can you paste the snippet that you used? You need to pass https://docs.microsoft.com/en-us/dotnet/api/azure.storage.blobs.models.blocklisttypes?view=azure-dotnet (All<== in the enum) to see blocks that have not been committed
2. About invalidblocklist: The first parameter that you are passing in your code snippet is actually blockId, the storage service expects this ID to be of the same length across all blocks in the blob. If you dump the output of getblocklist you will be able to check the length. If the length is the same across all your blocks, please do another repro and send me the x-ms-request-id and i can check the service logs for you.

Thanks

[wade-dp]

Thanks @amishra-dev.

1. I actually used two calls to 'GetBlockList' once for committed block and again for uncommitted blocks.

2. I have verified that the block id's all have the same length, and have reproduced the problem.
   The exception dump is below:

The specified block list is invalid.
RequestId:ae141122-e01e-005b-1389-a4ded2000000
Time:2021-09-08T08:11:26.0440797Z
Status: 400 (The specified block list is invalid.)
ErrorCode: InvalidBlockList

Content:

InvalidBlockListThe specified block list is invalid.

RequestId:ae141122-e01e-005b-1389-a4ded2000000
Time:2021-09-08T08:11:26.0440797Z

Headers:
x-ms-request-id: ae141122-e01e-005b-1389-a4ded2000000
x-ms-client-request-id: ed4ecf4b-5bc5-4c7e-8e8a-83908acd3727
x-ms-version: 2020-08-04
x-ms-error-code: InvalidBlockList
Content-Length: 221
Content-Type: application/xml
Date: Wed, 08 Sep 2021 08:11:25 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

[amishra-dev]

I checked the logs for this blob. I think i understand what went wrong.
You are trying to commit 3 blocks in a blob.
But before your putblocklist started failing you issues a PutBlob that succeeded. PutBlob call blows away all the blocks, hence beyond this point none of the putblocklist succeeded as you were referencing blocks that has been removed. Let me know if this does not match your understanding.

[ghost]

Hi @wade-dp. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[wade-dp]

/unresolve

Hi @amishra-dev. I have looked for where we perform a PutBlob, and it is happening as a result of a call to the 'UploadAsync' method on the .Net SDK.

From what I can see in the code these calls are only made after all the blocks are committed using the 'CommitBlocksAsync' call.

In the scenario I used to reproduce the problem, we committed blocks to 3 separate blobs, (2 worked) and (1 failed). The output above is for the failed case.

Is it possible that the 'PutBlob' calls that you saw were for the other blobs ?

[amishra-dev]

Hey @wade-dp sorry i missed that you reactivated this. No its not possible that the SDK changes a stageblock call to putblob REST API (and i did filter the service logs for the same blob). I saw a PutBlob REST API call, which means one of the clients called an upload or similar routine triggering the other block ids to be removed.
Let me know if you need further help/clarification here.