Files.list() returns empty if path has trailing slashes

[n3v3rf411]

getContainedObjectBatches is filtering on:

    .filter(s3Object -> s3Object.key().equals(prefix) || s3Object.key().startsWith(prefix + S3Path.PATH_SEPARATOR))

This fails when you are going to Files.list() a Path returned from a Files.list() response:

val path = Files
      .list(basePath)
      .filter(Files.isDirectory(_))
      .sorted((a, b) => b.compareTo(a))
      .filter(it => Files.exists(it.resolve(successFileName)))
      .findFirst()

Files.list(path.get) // This returns empty even if there are files in the "folder"

[markjschreiber]

Apologies for the slow response,

I wonder if this s3Object.key().startsWith(prefix + S3Path.PATH_SEPARATOR) should maybe be s3Object.key().equals(prefix + S3Path.PATH_SEPARATOR))

It would need some testing, especially with walking a tree from a starting node.

[markjschreiber]

The failure seems to occur when filtering on Files.exists()

[markjschreiber]

The source of the problem seems to be that S3FileSystemProvider.checkAccess() calls HeadObject on S3. This only succeeds if the path points to an object. If the path is just a prefix (s3 Directory) then it will fail. A better solution might be to use client.listObjectsV2() and ensure that the path is included in the response.

[markjschreiber]

This should now be fixed in main. Please re-open if there are still issues.