az storage blob upload-batch returns ErrorCode:BlobAlreadyExists error v2.34.0 on upload

[nosvalds]

Describe the bug
Trying to upload a file to the '$web' container for a static site with az storage blob upload-batch from GitHub Actions is failing on the latest version of the cli v2.34.0 saying that the blob already exists. The errors says to use --overwrite.

ERROR: The specified blob already exists.
RequestId:5a838987-601e-002b-6577-2d31d6000000
Time:2022-03-01T14:17:46.5388178Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.

Adding --overwrite command results in this error:

ERROR: argument --overwrite: expected one argument

To Reproduce
Setup a GitHub Actions workflow to upload directory to a storage blob;

      - name: Login to Azure CLI
        uses: azure/login@v1
        with:
          creds: ${{ env.AZURE_CREDENTIALS }}

      - name: Upload to blob storage
        uses: azure/CLI@v1
        env:
          STORAGE_NAME: sadatastoresearchdev
        with:
          azcliversion: latest
          inlineScript: |
            az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} -d '$web' -s dist/

Expected behavior
You should be able to upload the blob and overwrite as you could before.

Environment summary

https://github.com/Azure/cli in GitHub Actions

Additional context

Specifying an older azcliversion version v2.33.1 in the workflow allow you to upload the blob.

      - name: Login to Azure CLI
        uses: azure/login@v1
        with:
          creds: ${{ env.AZURE_CREDENTIALS }}

      - name: Upload to blob storage
        uses: azure/CLI@v1
        env:
          STORAGE_NAME: sadatastoresearchdev
        with:
          azcliversion: 2.33.1
          inlineScript: |
            az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} --auth-mode key -d '$web' -s dist/

Detailed logs from GitHub Actions:

022-03-01T14:17:06.8368067Z ##[group]Run azure/login@v1
2022-03-01T14:17:06.8368444Z with:
2022-03-01T14:17:06.8370124Z   creds: ***
2022-03-01T14:17:06.8370507Z   enable-AzPSSession: false
2022-03-01T14:17:06.8370913Z   environment: azurecloud
2022-03-01T14:17:06.8371321Z   allow-no-subscriptions: false
2022-03-01T14:17:06.8371762Z   audience: api://AzureADTokenExchange
2022-03-01T14:17:06.8372170Z env:
2022-03-01T14:17:06.8373821Z   AZURE_CREDENTIALS: ***
2022-03-01T14:17:06.8374471Z   STORAGE_KEY: ***
2022-03-01T14:17:06.8374823Z ##[endgroup]
2022-03-01T14:17:15.8274876Z [command]/usr/bin/az cloud set -n azurecloud
2022-03-01T14:17:21.4767300Z Done setting cloud: "azurecloud"
2022-03-01T14:17:23.0710370Z Login successful.
2022-03-01T14:17:23.0894436Z ##[group]Run azure/CLI@v1
2022-03-01T14:17:23.0894808Z with:
2022-03-01T14:17:23.0895130Z   azcliversion: latest
2022-03-01T14:17:23.0895667Z   inlineScript: az storage blob upload-batch --account-name sadatastoresearchdev -d '$web' -s dist/

2022-03-01T14:17:23.0896146Z env:
2022-03-01T14:17:23.0897712Z   AZURE_CREDENTIALS: ***
2022-03-01T14:17:23.0898317Z   STORAGE_KEY: ***
2022-03-01T14:17:23.0898675Z   AZURE_HTTP_USER_AGENT: 
2022-03-01T14:17:23.0899158Z   AZUREPS_HOST_ENVIRONMENT: 
2022-03-01T14:17:23.0899547Z   STORAGE_NAME: sadatastoresearchdev
2022-03-01T14:17:23.0899947Z ##[endgroup]
2022-03-01T14:17:23.1710590Z Starting script execution via docker image mcr.microsoft.com/azure-cli:latest
2022-03-01T14:17:45.3477491Z WARNING: 
2022-03-01T14:17:45.3478765Z There are no credentials provided in your command and environment, we will query for account key for your storage account.
2022-03-01T14:17:45.3583025Z It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.
2022-03-01T14:17:45.3583675Z 
2022-03-01T14:17:45.3598136Z You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
2022-03-01T14:17:45.3599424Z For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.
2022-03-01T14:17:45.3599847Z 
2022-03-01T14:17:45.3600793Z In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
2022-03-01T14:17:46.6529259Z ERROR: The specified blob already exists.
2022-03-01T14:17:46.6530651Z RequestId:5a838987-601e-002b-6577-2d31d6000000
2022-03-01T14:17:46.6531594Z Time:2022-03-01T14:17:46.5388178Z
2022-03-01T14:17:46.6532370Z ErrorCode:BlobAlreadyExists
2022-03-01T14:17:46.6533513Z If you want to overwrite the existing one, please add --overwrite in your command.
2022-03-01T14:17:46.9476322Z ##[error]Error: az cli script failed.
2022-03-01T14:17:46.9487839Z cleaning up container...
2022-03-01T14:17:46.9851133Z MICROSOFT_AZURE_CLI_1646144243123_CONTAINER
2022-03-01T14:17:46.9851724Z 
2022-03-01T14:17:46.9910045Z (node:1739) UnhandledPromiseRejectionWarning: Error: az cli script failed.
2022-03-01T14:17:46.9913655Z     at /home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:23964
2022-03-01T14:17:46.9914390Z     at Generator.throw (<anonymous>)
2022-03-01T14:17:46.9915105Z     at rejected (/home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:20436)
2022-03-01T14:17:46.9915949Z     at processTicksAndRejections (internal/process/task_queues.js:93:5)
2022-03-01T14:17:46.9917419Z (node:1739) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
2022-03-01T14:17:46.9919329Z (node:1739) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2022-03-01T14:17:46.9996553Z Post job cleanup.

[bacongobbler]

Can confirm - this is a recent change introduced in 2.34.0, breaking several of my pipelines using azure-blob-storage-upload which uses az storage blob upload-batch under the covers.

In addition to the use case @nosvalds provided earlier, I use this action to upload nightly releases of my program to Azure. With this recent change, I have no way to work around this short of deleting the object before uploading the new asset.

Reverting to 2.33.1 resolves the issue for now.

[yonzhan]

storage

[evelyn-ys]

@calvinhzy Can you help take a look? Might be a bug introduced by track2 migration

[calvinhzy]

Hotfix is getting merged today. You can specify --overwrite to overwrite existing blobs. This is a breaking change as we believe overwriting silently is not a good default. Thanks.

[LynAutumn]

Overwrite is not working for me either, failing with:

ERROR: argument --overwrite: expected one argument

[DragonBale]

Overwrite is not working for me either, failing with:

ERROR: argument --overwrite: expected one argument

You need to do
--overwrite true

[ozcanbecoms]

Thanks you @DragonBale

--overwrite true
works well

[bacongobbler]

I'm curious why true is a required argument for the --overwrite flag. Isn't the presence of the --overwrite flag enough to assert that the value is true, or is there an implied third value the CLI can provide?

In other words...

az storage blob upload-batch ...               # implies overwrite==false
az storage blob upload-batch --overwrite ...   # implies overwrite==true

Most CLI parsers work in this manner, so I'm curious why azure-cli chose to approach this differently.

[evelyn-ys]

The hotfix is on the way, both --overwrite and --overwrite true will be accepted after hotfix

[jeffpdotone]

Azure CLI task for release pipeline is not updated to the latest version is there a workaround to make the task use the latest one?

[evelyn-ys]

Which release pipeline do you mean? Is the latest version you mentioned v2.34.1?

[zwolf]

Which release pipeline do you mean? Is the latest version you mentioned v2.34.1?

"latest", insofar as the azure-cli github action is concerned, is still 2.33.1 as of this post. It was 2.34.1 for at least a few hours last week, but was apparently reverted. That is, a github action such as the one in the OP shows something like this in the console:

Run azure/CLI@v1
  with:
    inlineScript: 
      az storage blob upload-batch --account-name my_account -d '$web' -s dist/

Starting script execution via docker image mcr.microsoft.com/azure-cli:2.33.1

I have pinned the older 2.33.1 version in my workflows because the update to 2.34 breaks them in several different ways (newly requiring the --overwrite flag as in this issue, and the new content-type issue described here). This is instead of keeping the azcliversion argument undefined as suggested here.

[wahyuen]

Yet more breaking changes marked with 'patch' numbers. Is it so hard to follow SemVer and mark it as new release and not cause customers workloads to fail which have been working for YEARS!

[calvinhzy]

We will GA the --overwrite flag in May/June time frame. Thanks.

[JCPuerto]

Is anybody else getting the error az : WARNING: Argument '--overwrite' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus?

It overwrites the file but it throws an exception with a warning message.

Should I open a new issue?

[venikx]

Adding the --overwrite doesn't seem to have any effect, as overriding the blog storage still results in a ERROR: The specified blob already exists. error.