You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A write is implemented as a delete and then upload. If reads are concurrent with the write, then there is a chance the readers will get there after the delete but before the upload.
This could be called "not a bug; adlfs doesn't support concurrent reads/writes", but part of the point of object storage is to use on distributed systems with lots of concurrency. The docs guarantee strong consistency, so reads racing writes is supposed to return the most recent write. Adlfs breaks those semantics by deleting before uploading.
I can reliably trigger this problem with the following code:
import dask.distributed, adlfs, azure.identity.aio
parallel_client = dask.distributed.Client()
def test():
abfs = adlfs.AzureBlobFileSystem(account_name="wfregtest", credential=azure.identity.aio.DefaultAzureCredential())
for _ in range(30):
with abfs.open("data4/test", "rb") as fobj:
buffer = fobj.read()
with abfs.open("data4/test", "wb") as fobj:
fobj.write(buffer)
parallel_client.run(test)
This is related to #236 and #264. #264 in particular just ignores the delete if it fails. I was wondering if the delete can be removed entirely?
The text was updated successfully, but these errors were encountered:
A write is implemented as a delete and then upload. If reads are concurrent with the write, then there is a chance the readers will get there after the delete but before the upload.
This could be called "not a bug; adlfs doesn't support concurrent reads/writes", but part of the point of object storage is to use on distributed systems with lots of concurrency. The docs guarantee strong consistency, so reads racing writes is supposed to return the most recent write. Adlfs breaks those semantics by deleting before uploading.
I can reliably trigger this problem with the following code:
This is related to #236 and #264. #264 in particular just ignores the delete if it fails. I was wondering if the delete can be removed entirely?
The text was updated successfully, but these errors were encountered: