Problem with listing blobs by using of ListBlobsHierarchy() #16295

an2deg · 2021-11-26T11:20:30Z

Bug Report

import path of package in question github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
SDK version
github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.12.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.2.0
github.com/Azure/azure-sdk-for-go/sdk/to v0.1.1 // indirect
github.com/aws/aws-sdk-go v1.40.45
Go version: go version go1.16.6 darwin/amd64
Azurite: mcr.microsoft.com/azure-storage/azurite:3.14.3

In my code I'm invoking ContainerClient.ListBlobsHierarchy() to iterate over a list of blobs in an Azure Blob container.
I'm using azure-sdk-for-go in combination with Azurite.

However, tests fail with error get response from Azure API: no Host in request URL.

I started troubleshooting and found out that containerClient.ListBlobHierarchySegment() on the next call of NextPage() is passing test-fixtures-TestAzBlobClient_ListObjects-2021-11-26T114346.107911+0100/test.csv.1 into runtime.NewRequest() as value of parameter endpoint.

I'm wondering is it problem of Azurite which doesn't compose NextMarker properly or it's problem of ListBlobHierarchySegment() which doesn't respect containerClient configuration ?

Here is a screenshot of debugger demonstrating the process of troubleshooting:

My code:

                 perPageConv := int32(2)

		listOptions := &azblob.ContainerListBlobHierarchySegmentOptions{
			Prefix: &prefix,
			Maxresults: &perPageConv,
		}
		azPager := azClient.ListBlobsHierarchy("/", listOptions)

                for {
			hasNext := azPager.NextPage(ctx)
                         if azErr := azPager.Err(); azErr != nil { 
				fmt.Println("get response from Azure API: ", azErr.Error())
				break
                        }

			page := azPager.PageResponse()

			for i, commonPrefix := range page.Segment.BlobPrefixes {
                                 fmt.Println("Blob prefix ", commonPrefix.Name)
			}

			for i, keyObject := range page.Segment.BlobItems {
                                 fmt.Println("Blob ", keyObject.Name)
			}

			if !hasNext {
				break
			}

			pageNo++
		}

The text was updated successfully, but these errors were encountered:

an2deg · 2021-11-26T12:50:16Z

I have just checked on real Azure env and the problem is the same: after requesting the second page I get :

2021/11/26 13:48:25 list blobs:  operational: get response from Azure API: no Host in request URL

The same problem is with ListBlobsFlat()

an2deg · 2021-11-30T08:17:16Z

@kenegozi may I ask you to take a look to this issue? Labels in this issue are incorrect: it's not a question but a bug report and the issue is a serious one.

adreed-msft · 2021-11-30T22:58:27Z

Currently looking.

adreed-msft · 2021-11-30T23:12:03Z

Looks like the generated code is incorrect. Seems to be supplying the entire marker as the new URI. I'm going to shoot out a PR to override this incorrect behavior.

an2deg · 2021-12-02T08:26:13Z

Thank you @adreed-msft for looking at this issue. I cannot test your PR as there are broken dependencies. Will test when your PR be merged.

$ go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@adreed/fix-pagers
go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.1.1-0.20211201002034-a166c1381cd2
go: downloading github.com/Azure/azure-sdk-for-go v58.3.1-0.20211201002034-a166c1381cd2+incompatible
# github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_append_blob_client.go:17:47: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_blob_client.go:17:9: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_blob_client.go:21:41: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_block_blob_client.go:31:46: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_connection_string.go:38:62: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_container_client.go:18:9: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_container_client.go:27:51: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_page_blob_client.go:23:45: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_service_client.go:30:9: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_service_client.go:40:47: undefined: azcore.Credential
../../../Library/Go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/zc_service_client.go:40:47: too many errors

mohsha-msft · 2021-12-03T06:04:52Z

Hey @an2deg ,

Can you try

go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@dev-azblob

dev-azblob is our development branch and I've merged the changes there.
In general, go get @ is the command to get a specific version or commit-id instead of main branch.

an2deg · 2021-12-06T10:21:39Z

Hi Mohit, I tried to do so but in my project and even in empty project I alway get error message: ✔ ~/Projects/test 11:18 $ go get github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob go get: module ***@***.*** found (v60.0.0+incompatible), but does not contain package github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob ✘-1 ~/Projects/test 11:19 $

…

On Fri, Dec 3, 2021 at 7:05 AM Mohit Sharma ***@***.***> wrote: Hey @an2deg <https://github.com/an2deg> , Can you try go get github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob dev-azblob is our development branch and I've merged the changes there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC47UWWSEFI42BF7MO2CR3UPBMZBANCNFSM5I2N22UQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Best regards, Andrii Degtiarov

mohsha-msft · 2021-12-06T10:35:24Z

Hey @an2deg ,

What's the version of Golang you're using?
Can you try go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@b76e8f674ad2125742c244cfc8faa417202525bf

or go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@dev-azblob

an2deg · 2021-12-06T11:20:55Z

Hi Mohit, Ok, go get ***@***.*** works. However, I still have problems with the fix. I have the following function to load fixtures to Azurite: var containerClient azblob.ContainerClient func setUp() { if azblobHost == "" { log.Fatalf("AZBLOB_HOST env variable is required to run depository/azure tests\n") } if azblobPort == "" { log.Fatalf("AZBLOB_PORT env variable is required to run depository/azure tests") } if azblobKey == "" { log.Fatalf("AZBLOB_KEY env variable is required to run depository/azure tests") } if azblobAccount == "" { log.Fatalf("AZBLOB_ACCOUNT env variable is required to run depository/azure tests") } clientOpts = Opts{ Endpoint: fmt.Sprintf("%s:%s", azblobHost, azblobPort), AccountName: azblobAccount, SecretKey: azblobKey, Container: azblobContainer, DisableSSL: true, } creds, err := azblob.NewSharedKeyCredential(azblobAccount, azblobKey) if err != nil { log.Fatalf("Cannot initialize Azure Blob Storage client: %v\n", err.Error()) } srvClient, err := azblob.NewServiceClientWithSharedKey("http://"+azblobHost+":"+azblobPort+"/"+azblobAccount, creds, nil) if err != nil { log.Fatalf("Cannot initialize Azure Blob Storage client: %v\n", err.Error()) } containerClient = srvClient.NewContainerClient(azblobContainer) ctx := context.Background() _, err = containerClient.Create(ctx, &azblob.CreateContainerOptions{}) if err != nil { azErr := &azblob.StorageError{} if errors.As(err, &azErr) && azErr.Response().Status == "409 The specified container already exists." { // Container is already exist, ignore } else { log.Fatalf("Cannot provision a container in Azure Blob Storage client: %v\n", err.Error()) } } } func loadFixturesForListing(prefix string) error { ctx := context.Background() const alreadyExistingObjectKey = "test.csv" // Create objects to test pagination for i := 0; i < 14; i++ { blobName := path.Join(prefix, alreadyExistingObjectKey+fmt.Sprintf(".%d", i)) blockBlob := containerClient.NewBlockBlobClient(blobName) reader := bytes.NewReader(testKeyContent) if _, err := blockBlob.UploadStreamToBlockBlob(ctx, newNopSeekCloser(reader), azblob.UploadStreamToBlockBlobOptions{}); err != nil { return fmt.Errorf("upload blob %s into Azure Container: %w", blobName, err) } } // create a subfolder blobName := path.Join(prefix, "subfolder", alreadyExistingObjectKey) blockBlob := containerClient.NewBlockBlobClient(blobName) reader := bytes.NewReader(testKeyContent) if _, err := blockBlob.UploadStreamToBlockBlob(ctx, newNopSeekCloser(reader), azblob.UploadStreamToBlockBlobOptions{}); err != nil { return fmt.Errorf("upload blob %s into Azure Container: %w", blobName, err) } return nil } This code provisions 14 blobs with names "test.csv.0" - "test.csv.13" under prefix like "test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100/". And 1 blob test.csv under prefix "test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100/subfolder". I expect that ListBlobsHierarchy() for prefix "test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100" after all will returns 14 BlobItems items and 1 BlobPrefixes. The I invoke azPager := azClient.ListBlobsHierarchy(delimiter, listOptions) with the following params [image: image.png] and iterate over results by using azPager.NextPage(ctx). Please note, even when I'm not specifying Maxresults, behavior is the same. On the first NextPage call() I have the following results: [image: image.png] on a second call I get almost the same but MaxResults now is 5000 and NextPage() returns false [image: image.png] Because NextPage() returns false, I exited from a loop at this point. azPager.Err() is always nil . However I executed a few more times from the debugger, but the response is still the same.

…

On Mon, Dec 6, 2021 at 11:35 AM Mohit Sharma ***@***.***> wrote: Hey @an2deg <https://github.com/an2deg> , What's the version of Golang you're using? Can you try go get ***@***.*** or go get ***@***.*** — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC47UWHBCEOME6Y227L4ETUPSGXPANCNFSM5I2N22UQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Best regards, Andrii Degtiarov

whati001 · 2022-01-24T15:31:51Z

I have faced the same issue when executing the sample code from the README.md
Pulling the dev release go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@dev-azblobfixed the issue for me. Is there already a date fixed when this fix will be merged into the official release?

mohsha-msft · 2022-02-09T16:23:31Z

Fixed in sdk/storage/azblob/v0.3.0

seankane-msft assigned mohsha-msft Nov 30, 2021

tasherif-msft assigned siminsavani-msft and adreed-msft Nov 30, 2021

adreed-msft mentioned this issue Dec 1, 2021

Adreed/fix pagers #16354

Merged

ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Dec 1, 2021

RickWinter added this to the [2022] January milestone Dec 1, 2021

serprex mentioned this issue Feb 7, 2022

azblob: fix ListBlob to preserve prefix while paging #16992

Merged

mohsha-msft closed this as completed Feb 9, 2022

github-actions bot locked and limited conversation to collaborators Apr 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with listing blobs by using of ListBlobsHierarchy() #16295

Problem with listing blobs by using of ListBlobsHierarchy() #16295

an2deg commented Nov 26, 2021 •

edited

Loading

an2deg commented Nov 26, 2021 •

edited

Loading

an2deg commented Nov 30, 2021

adreed-msft commented Nov 30, 2021

adreed-msft commented Nov 30, 2021

an2deg commented Dec 2, 2021

mohsha-msft commented Dec 3, 2021 •

edited

Loading

an2deg commented Dec 6, 2021 via email

mohsha-msft commented Dec 6, 2021

an2deg commented Dec 6, 2021 via email

whati001 commented Jan 24, 2022

mohsha-msft commented Feb 9, 2022

Problem with listing blobs by using of ListBlobsHierarchy() #16295

Problem with listing blobs by using of ListBlobsHierarchy() #16295

Comments

an2deg commented Nov 26, 2021 • edited Loading

Bug Report

an2deg commented Nov 26, 2021 • edited Loading

an2deg commented Nov 30, 2021

adreed-msft commented Nov 30, 2021

adreed-msft commented Nov 30, 2021

an2deg commented Dec 2, 2021

mohsha-msft commented Dec 3, 2021 • edited Loading

an2deg commented Dec 6, 2021 via email

mohsha-msft commented Dec 6, 2021

an2deg commented Dec 6, 2021 via email

whati001 commented Jan 24, 2022

mohsha-msft commented Feb 9, 2022

an2deg commented Nov 26, 2021 •

edited

Loading

an2deg commented Nov 26, 2021 •

edited

Loading

mohsha-msft commented Dec 3, 2021 •

edited

Loading