New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

`Azure` Azure Blob Interface #99

Merged

ShishirPatil merged 5 commits into main from azure-interface

Jan 27, 2022

Member

ShishirPatil commented Jan 27, 2022 •

edited

Loading

Azure Blob storage interface.

Notes:

No buckets -> we have containers.
For downloading buckets, read_all() faster than stream.chunks()
Works for a single region that is mapped to the AZURE_STORAGE_CONNECTION_STRING read from OS environment.

Test with command:
python skylark/test/test_azure_interface.py

TODO:

Make it compatible to all regions other than us-east1 (Done: Azure Azure interface #102)

ShishirPatil added 4 commits

January 26, 2022 17:30


          [WIP] Azure Interface

80173d0


          Azure interface

7b4f940


          Syntax fizes

3a03109


          Tested Azure Interface for single region

801d1db

ShishirPatil requested review from parasj and sarahwooders

January 27, 2022 08:22

ShishirPatil changed the title ~~[WIP] Azure Interface~~ Azure Interface

ShishirPatil changed the title ~~Azure Interface~~ Azure Azure Blob Interface

ShishirPatil mentioned this pull request

Azure Gateway and replicator interface #100

Merged

1 task

Member Author

ShishirPatil commented Jan 27, 2022

Note: Do not use bob_client.exists() to check for existance of a file in azure_interface.py. During testing I found that the API is buggy and will return true even if the blob does not exist. The workaround is as implemented.

parasj approved these changes

View reviewed changes

Contributor

parasj left a comment

LGTM

skylark/cli/cli_helper.py Show resolved Hide resolved

skylark/obj_store/azure_interface.py Show resolved Hide resolved

skylark/obj_store/azure_interface.py


		self.container_client = None

		# TODO:: Figure this out, since azure by default has 15 workers

Contributor

parasj Jan 27, 2022

Where is parallelism in this setup? Does the Azure CLI internally dispatch parallel requests? Or are you explicitly managing this? Or is this nested parallelism?

Member Author

ShishirPatil Jan 27, 2022

Yes, azure internally dispatches parallel requests. It is specified by the self.max_concurrency parameter.

skylark/obj_store/azure_interface.py Show resolved Hide resolved

skylark/obj_store/azure_interface.py

+                      src_object_name, dst_file_path = str(src_object_name), str(dst_file_path)
+                      src_object_name = src_object_name if src_object_name[0] != "/" else src_object_name
+                      def _download_object_helper(offset, **kwargs):

Contributor

parasj Jan 27, 2022

How is offset used? I thought that the chunk interface you tested was slower than atomic objects?

Member Author

ShishirPatil Jan 27, 2022

Correct. It's not used.

skylark/obj_store/azure_interface.py Outdated Show resolved Hide resolved

skylark/obj_store/azure_interface.py Show resolved Hide resolved

ShishirPatil mentioned this pull request

CLI support for Azure and GCS object stores #101

Closed


          Getting rid of s3 specific mime type

3ff2ca2

ShishirPatil merged commit 2c57c3d into main

ShishirPatil deleted the azure-interface branch

January 27, 2022 20:17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet