S3Path.glob() doesn't work correctly starting from s3path>=0.4
#160
Comments
Alexander-Serov
changed the title
S3Path.glob() doesn't work correctly starting for s3path>=0.4S3Path.glob() doesn't work correctly starting from s3path>=0.4
|
Hi @Alexander-Serov |
|
I merged a fix |
|
Version 0.5.3 deployed with the fix |
|
Thanks @liormizr ! Surprisingly just upgrading to s3path==0.5.2s3path==0.5.3And the specific error looks like which can be fixed by replacing |
|
@Alexander-Serov |
|
Found the issue |
|
@Alexander-Serov version 0.5.5 was deployed with the fix. |
|
@liormizr Thanks for fixing fast! I'll check when I have a moment. |
|
Hi @liormizr, I am experiencing some weird behavior after updating to 0.5.5 with the new from s3path import S3Path
p = S3Path.from_uri("replace-with-s3-uri")
s3_file = p / "some_dir" / "empty.txt"
with s3_file.open("w") as fp:
fp.write("1")
print(list(p.glob("*")))The script above results 2 entries of p / "some_dir" |
|
Hi @michaelvay I just now wrote this test: def test_glob_issue_160_weird_behavior(s3_mock):
"""
from s3path import S3Path
p = S3Path.from_uri("replace-with-s3-uri")
s3_file = p / "some_dir" / "empty.txt"
with s3_file.open("w") as fp:
fp.write("1")
print(list(p.glob("*")))
"""
s3 = boto3.resource('s3')
s3.create_bucket(Bucket='my-bucket')
path = S3Path.from_uri("s3://my-bucket/")
new_file = path / "some_dir" / "empty.txt"
new_file.touch()
assert list(path.glob("*")) == [S3Path('/my-bucket/some_dir')]The test passed properly |
I am using Python 3.10.13 |
|
@michaelvay sounds like a setup issue |
|
Here is a reproduce script with minio container Run minio: docker run -d --name minio \
-p 9000:9000 \
-p 9001:9001 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin123 \
minio/minio server /data# test_s3path.py
import boto3
from botocore.client import Config
from s3path import S3Path, register_configuration_parameter
endpoint_url = "http://localhost:9000"
access_key = "minioadmin"
secret_key = "minioadmin123"
minio_resource = boto3.resource(
's3',
endpoint_url=endpoint_url ,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
config=Config(signature_version='s3v4'),
region_name='us-east-1')
try:
bucket = "my-bucket"
minio_resource.create_bucket(Bucket=bucket)
except:
print("bucket already created")
first_dir = S3Path.from_uri(f"s3://{bucket}/first_dir/")
register_configuration_parameter(first_dir, resource=minio_resource)
new_file = first_dir / "some_dir" / "empty.txt"
new_file.touch()
print(list(first_dir.glob("*")))
second_dir = S3Path.from_uri(f"s3://{bucket}/first_dir/second_dir/")
register_configuration_parameter(second_dir, resource=minio_resource)
new_file = second_dir / "some_dir" / "empty.txt"
new_file.touch()
print(list(second_dir.glob("*")))Run: Results: |
|
Hi @michaelvay |
|
Hi @liormizr, |
|
I confirm: v0.5.6 doesn't have the "boto3" import error anymore. Thanks @liormizr ! |
Hello!
I have observed an
s3pathbug with the glob operation in S3. Have a look at me counting the number of "output" subfolders in a specific S3 folder:Do you know what this could be due to? Nothing changes other than the version of
s3pathI'm using. Are you able to reproduce? Could you fix the glob?The text was updated successfully, but these errors were encountered: