-
Notifications
You must be signed in to change notification settings - Fork 836
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with listing blobs by using of ListBlobsHierarchy() #16295
Comments
I have just checked on real Azure env and the problem is the same: after requesting the second page I get :
The same problem is with ListBlobsFlat() |
@kenegozi may I ask you to take a look to this issue? Labels in this issue are incorrect: it's not a question but a bug report and the issue is a serious one. |
Currently looking. |
Looks like the generated code is incorrect. Seems to be supplying the entire marker as the new URI. I'm going to shoot out a PR to override this incorrect behavior. |
Thank you @adreed-msft for looking at this issue. I cannot test your PR as there are broken dependencies. Will test when your PR be merged.
|
Hey @an2deg , Can you try go get github.com/Azure/azure-sdk-for-go/sdk/storage/azblob@dev-azblob
|
Hi Mohit,
I tried to do so but in my project and even in empty project I alway get
error message:
✔ ~/Projects/test
11:18 $ go get github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob
go get: module ***@***.*** found
(v60.0.0+incompatible), but does not contain package
github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob
✘-1 ~/Projects/test
11:19 $
…On Fri, Dec 3, 2021 at 7:05 AM Mohit Sharma ***@***.***> wrote:
Hey @an2deg <https://github.com/an2deg> ,
Can you try
go get github.com/Azure/azure-sdk-for-go/sdk/storage/dev-azblob
dev-azblob is our development branch and I've merged the changes there.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#16295 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAC47UWWSEFI42BF7MO2CR3UPBMZBANCNFSM5I2N22UQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Best regards,
Andrii Degtiarov
|
Hey @an2deg , What's the version of Golang you're using? or |
Hi Mohit,
Ok, go get ***@***.***
works.
However, I still have problems with the fix. I have the following function
to load fixtures to Azurite:
var containerClient azblob.ContainerClient
func setUp() {
if azblobHost == "" {
log.Fatalf("AZBLOB_HOST env variable is required to run
depository/azure tests\n")
}
if azblobPort == "" {
log.Fatalf("AZBLOB_PORT env variable is required to run
depository/azure tests")
}
if azblobKey == "" {
log.Fatalf("AZBLOB_KEY env variable is required to run
depository/azure tests")
}
if azblobAccount == "" {
log.Fatalf("AZBLOB_ACCOUNT env variable is required to run
depository/azure tests")
}
clientOpts = Opts{
Endpoint: fmt.Sprintf("%s:%s", azblobHost, azblobPort),
AccountName: azblobAccount,
SecretKey: azblobKey,
Container: azblobContainer,
DisableSSL: true,
}
creds, err := azblob.NewSharedKeyCredential(azblobAccount, azblobKey)
if err != nil {
log.Fatalf("Cannot initialize Azure Blob Storage client: %v\n",
err.Error())
}
srvClient, err :=
azblob.NewServiceClientWithSharedKey("http://"+azblobHost+":"+azblobPort+"/"+azblobAccount,
creds, nil)
if err != nil {
log.Fatalf("Cannot initialize Azure Blob Storage client: %v\n",
err.Error())
}
containerClient = srvClient.NewContainerClient(azblobContainer)
ctx := context.Background()
_, err = containerClient.Create(ctx, &azblob.CreateContainerOptions{})
if err != nil {
azErr := &azblob.StorageError{}
if errors.As(err, &azErr) && azErr.Response().Status == "409 The
specified container already exists." {
// Container is already exist, ignore
} else {
log.Fatalf("Cannot provision a container in Azure Blob
Storage client: %v\n", err.Error())
}
}
}
func loadFixturesForListing(prefix string) error {
ctx := context.Background()
const alreadyExistingObjectKey = "test.csv"
// Create objects to test pagination
for i := 0; i < 14; i++ {
blobName := path.Join(prefix,
alreadyExistingObjectKey+fmt.Sprintf(".%d", i))
blockBlob := containerClient.NewBlockBlobClient(blobName)
reader := bytes.NewReader(testKeyContent)
if _, err := blockBlob.UploadStreamToBlockBlob(ctx,
newNopSeekCloser(reader),
azblob.UploadStreamToBlockBlobOptions{}); err != nil {
return fmt.Errorf("upload blob %s into Azure Container: %w",
blobName, err)
}
}
// create a subfolder
blobName := path.Join(prefix, "subfolder", alreadyExistingObjectKey)
blockBlob := containerClient.NewBlockBlobClient(blobName)
reader := bytes.NewReader(testKeyContent)
if _, err := blockBlob.UploadStreamToBlockBlob(ctx, newNopSeekCloser(reader),
azblob.UploadStreamToBlockBlobOptions{}); err != nil {
return fmt.Errorf("upload blob %s into Azure Container: %w",
blobName, err)
}
return nil
}
This code provisions 14 blobs with names "test.csv.0" - "test.csv.13"
under prefix like
"test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100/".
And 1 blob test.csv under prefix
"test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100/subfolder".
I expect that ListBlobsHierarchy() for prefix
"test-fixtures-TestAzBlobClient_ListObjects-2021-12-06T115616.24455+0100"
after all will returns 14 BlobItems items and 1 BlobPrefixes.
The I invoke azPager := azClient.ListBlobsHierarchy(delimiter,
listOptions) with the following params
[image: image.png]
and iterate over results by using azPager.NextPage(ctx).
Please note, even when I'm not specifying Maxresults, behavior is the same.
On the first NextPage call() I have the following results:
[image: image.png]
on a second call I get almost the same but MaxResults now is 5000 and
NextPage() returns false
[image: image.png]
Because NextPage() returns false, I exited from a loop at this point.
azPager.Err() is always nil .
However I executed a few more times from the debugger, but the
response is still the same.
…On Mon, Dec 6, 2021 at 11:35 AM Mohit Sharma ***@***.***> wrote:
Hey @an2deg <https://github.com/an2deg> ,
What's the version of Golang you're using?
Can you try go get
***@***.***
or go get ***@***.***
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#16295 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAC47UWHBCEOME6Y227L4ETUPSGXPANCNFSM5I2N22UQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Best regards,
Andrii Degtiarov
|
I have faced the same issue when executing the sample code from the README.md |
Fixed in |
Bug Report
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.12.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.2.0
github.com/Azure/azure-sdk-for-go/sdk/to v0.1.1 // indirect
github.com/aws/aws-sdk-go v1.40.45
In my code I'm invoking
ContainerClient.ListBlobsHierarchy()
to iterate over a list of blobs in an Azure Blob container.I'm using azure-sdk-for-go in combination with Azurite.
However, tests fail with error
get response from Azure API: no Host in request URL
.I started troubleshooting and found out that
containerClient.ListBlobHierarchySegment()
on the next call ofNextPage()
is passingtest-fixtures-TestAzBlobClient_ListObjects-2021-11-26T114346.107911+0100/test.csv.1
intoruntime.NewRequest()
as value of parameterendpoint
.I'm wondering is it problem of Azurite which doesn't compose NextMarker properly or it's problem of ListBlobHierarchySegment() which doesn't respect
containerClient
configuration ?Here is a screenshot of debugger demonstrating the process of troubleshooting:
My code:
The text was updated successfully, but these errors were encountered: