[BUG] Padding is invalid and cannot be removed while doing client side encryption for storage blob data using keyvault key #23082
Comments
ghost
added
needs-triage
jsquire
added
Azure.Identity
Client
ghost
removed
the
needs-triage
|
Hi @PPrabinKumarPatro. Thank you for your feedback and we regret that you're experiencing difficulties. I'm not sure the source of your error - specifically if you're seeing this when acquiring the token as you were with #16298 or if this is being surfaced by the KeyVault or Storage libraries. If you're able to share a stack trace and any other context that you have, it would be helpful in finding the right people to assist. For now, I've looped in our |
jsquire
added
the
needs-author-feedback
ghost
removed
the
needs-team-attention
|
updated with call stack |
ghost
added
needs-team-attention
|
Thank you! The call stack appears to imply that this is occurring in the Storage SDK. Rerouting to the storage team for assistance. |
jsquire
added
Service Attention
|
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage. Issue DetailsDescribe the bug Expected behavior Actual behavior (include Exception or Stack Trace) To Reproduce code snippet is exactly explained here except we are using SNI Environment:
Call stack:
|
|
HI Storage team, can you help here as we are having issues in most of prod regions including EUS, It would be challenging if it became bigger issue. Thanks. |
|
Can you provide a more detailed code snippet where the key is being passed to the I wonder what's wrong with the key being passed to the |
|
"Padding is invalid" is the hardest issue to track down. It essentially means that the last 16 byte encryption block was an unexpected value according to the AES specification and so there was some sort of error. Something could have gone wrong during encryption, decryption, key wrapping, key unwrapping, upload, download, or even blob metadata management; some piece of data is incorrect in the whole process. In the past, this has come up when users are working with their own IKeyEncryptionKey and IKeyEncryptionKeyResolver implementations and they made an error developing them. This doesn't appear to be the case here, though, since you're using KeyVault SDK. Additionally, you point us to your other issue for code snippets, but there isn't any code in there that uses the storage SDK. That will likely be necessary to figure out what is happening. |
|
Hi, what information do you need here to debug? Also can you ping me over teams for quick fix. We are still having the issue but in different region. The issue auto resolves in some regions by itself. |
|
below is the code and we use storage SDK ( BlobClient blobClient = containerClient.GetBlobClient("test").WithClientSideEncryptionOptions(clientSideOptions);) //Create CryptographyClient using Key Vault Key |
|
Any update here? |
|
Taken offline. Will update here when conclusions are reached. |
|
Any updates, please I am getting the same error. |
|
@PPrabinKumarPatro Did you find any quick fix for this issue? |
|
Any updates on this please??? |
|
If someone has a consistent reproduction of this error they can provide, that would help immensely. As stated earlier in this thread, this is a difficult issue to track because the actual issue has come and gone by the time this exception is raised. |
|
@jaschrep-msft I do, let me copy some stuff in for you and let me know what else if anything you need. If you need the actual blobs/data, we will need to go through Microsoft Support so NDA is in place. |
|
So I am basically running an Azure function, where it goes gets a list of blobs and pushes that into a queue, then the processing of that blob is to download it, decrypt it and encrypted it with a new Key Vault Key, or a new version of the Key and re-upload it to the container. We are doing this for a few reasons, one key comprised (not likely but have to have the process), changing key or the main reason wanting to re-encrypt all the data that was using standard RSA key to RSA-HSM key. The errors I am getting are as follows: code is: Environment: Name and version of the Library package used: Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): IDE and version : [e.g. Visual Studio 16.3] |
|
@wonderphil thank you! I will look into this with your sample when I have the time. However, at a glance, it looks like you've provided me with code that only does the download, decrypt, reencrypt, and reupload. The initial upload isn't present. Depending on what root cause is, that part may be relevant. If you could convert (or wrap) this sample to cut the queue out of the way and just upload a new blob that will still produce this issue, I can look into this with my own creds and account. If you can't manage a reproduction that way, or if you don't get a consistent reproduction, that's important! Let me know here. But since you actively have an instance of this issue and I have been unable to reproduce one, you're likely to have the code somewhere that can set this situation up. Otherwise, if this code succeeds in decryption with some blob I upload myself, I won't have an actual reproduction. |
|
So out of my testing, I haven't been able to upload a new files and then get the error. I have only been able to reproduce on files that have been in our environment already. I am currently trying to tell if it happens to the same blob every time or different blobs each run, our environment has 300+k blobs so I am going to move into another environment with 10+ blobs for testing clarity |
|
@wonderphil wondering if you've managed to narrow things down in your tests. In the meantime I've noticed something in your sample: await blobEncryptionClient.UploadAsync(memoryStream, overwrite: true);
// Fix for bug. Please see https://stackoverflow.com/questions/67239116/how-do-you-decrypt-blobs-with-azure-keyvault-keys-in-azure-storage-v12
if (blobEncryptionClient.GetProperties().Value.Metadata.ContainsKey("encryptiondata"))
{
if (metadata.ContainsKey("encryptiondata"))
metadata["encryptiondata"] = blobEncryptionClient.GetProperties().Value.Metadata["encryptiondata"];
else
metadata.Add("encryptiondata", blobEncryptionClient.GetProperties().Value.Metadata["encryptiondata"]);
}
await blobEncryptionClient.SetMetadataAsync(metadata);I've investigated this metadata bug (which no one seems to have filed in this repo). If I understand the linked stack overflow answer correctly, I can confirm there is a bug with client-side encryption and metadata, but it is not what is linked and don't think it's at play here. The stack overflow answer's claimed bug is the known and accepted result of metadata design (note the warning here). What interests me about this sample, though, is that it introduces a race condition that wouldn't exist if you just put the metadata on the upload call, e.g. It may be your end-to-end system is hitting this race condition. Because the encrypted data and the encryption metadata aren't being uploaded as one, something else could be getting in there and changing one of those two things in between steps. This would result in a mismatch between the encrypted data and its encryption metadata. This would also explain why you can't manage to replicate it with an isolated test. |
amishra-dev
added
bug
|
@jaschrep-msft any progress on this? |
|
Closing this issue. Feel free to reopen if the problem persists after eliminating this race condition. |
Describe the bug
We are doing client side encryption for storage blob data using keyvault key. This is the bug "#16298" i intially filled for SNI. We were able to get it done but we are getting "Padding is invalid and cannot be removed" error now
Expected behavior
Should successfully decrypt and encrypt the data
Actual behavior (include Exception or Stack Trace)
Padding is invalid and cannot be removed error
To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)
code snippet is exactly explained here except we are using SNI
#16298
Environment:
Azure.Storage.Blobs : 12.8.0
Azure.Identity is 1.4.0-beta.1
Azure.Security.KeyVault.Keys 4.2.0-beta.2
dotnet --infooutput for .NET Core projects): [e.g. Azure AppService or Windows 10 .NET Framework 4.8].Net core
Visual Studi 2019
Call stack:
The text was updated successfully, but these errors were encountered: