Skip unsupported object names in list output

[gargnitingoogle]

Description

Skip unsupported directory names in list prefixes. Warn on unsupported object names returned on listing.

GCS supports objects of path-type <bucket>/<path1>//<path2>
and treats it different from <bucket>/<path1>/<path2>, while they are both the same in linux filesystem.
GCSFuse being a POSIX-compliant file-system only support the latters and throws error on former. This error can disallow the listing of all directories in the parent directory i.e. <path1>.
The current change ignores the listing of prefixes (directory names) which are empty such
as "/" above to ignore the error and logs the above event as a warning.

This is followed up in

1. Add unsupported directory substrings . and .. #2001, which adds . and .. and their derivative object names to the list of unsupported object names, And in
2. Add e2e tests for fix for unsupported directory list #2011 which adds e2e tests for this change.

Link to the issue in case of a bug fix.

NA

Testing details

1. Manual -
   • GCS bucket structure: 1. <bucket>//hello.txt, 2. <bucket>/a/hello.txt
   • GCSFuse mount command: gcsfuse --implicit-dirs --log-file=$logfile --debug_fuse --log-format=text $bucket $mountpath
   • Without the fix,

     ls $mountpath
     ls: reading directory $mountpath: Input/output error
     cat $logfile
     - No warning/error log -

   • With the fix,

         ls $mountpath
         a
         cat $logfile
         ...
         time=... severity=WARNING message="Ignoring unsupported prefix \"/\""
         ...

2. Unit tests - NA
3. Integration tests - NA

[gargnitingoogle]

Presubmit tests passed: logs

[ashmeenkaur]

Still need to take a look at the tests

[codecov]

Codecov Report

Attention: Patch coverage is 95.12195% with 2 lines in your changes missing coverage. Please review.

Project coverage is 78.39%. Comparing base (3b6804a) to head (5fb1408).

Files with missing lines	Patch %	Lines
internal/util/gcs_util.go	92.30%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1876      +/-   ##
==========================================
+ Coverage   78.38%   78.39%   +0.01%     
==========================================
  Files         107      108       +1     
  Lines       11793    11833      +40     
==========================================
+ Hits         9244     9277      +33     
- Misses       2058     2064       +6     
- Partials      491      492       +1     

Flag	Coverage Δ
unittests	78.39% <95.12%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gargnitingoogle]

added self-review comments.

[gargnitingoogle]

@ashmeenkaur @sethiay I've resolved all your comments. Please review again.

[ashmeenkaur]

Is this expected? There is no valid object having "bar" directory. I think this will cause issues in case of RemoveDir operation as the object won't be listed during the operation. Please check.

[gargnitingoogle]

Is this expected? There is no valid object having "bar" directory.

Yes. This is the cost of using MaxResults=1 for lookUpInode.

If we have objects bar//1, and bar/2 in a bucket, gcs list call during lookUpInode with MaxResults=1 will return bar//1 . If we reject unsupported object bar//1 at the time of lookup-inode, then bar will never be identified as a directory by GCSFuse, and thus bar/2 will never be accessible through GCSFuse, which is no better than the current bug. Now, even if there is no object named bar/2, we can not know from gcs list call with MaxResults=1, and we end up having to identify bar as a directory, albeit empty.

[gargnitingoogle]

I think this will cause issues in case of RemoveDir operation as the object won't be listed during the operation

I'm not sure I understand what you mean by issues. Are you saying that bar//1 will not be deleted, or that it'll be deleted without being listed ?

I'll test out RemoveDir manually to see what the behaviour is.

[gargnitingoogle]

I'll test out RemoveDir manually to see what the behaviour is.

I tested this out manually, in the above example, if I try and delete/rename bar in the gcsfuse mount, it only deletes/renames bar/2 and has no impact on bar//1 in GCS . This looks like the correct behaviour to me as we intend to have bar//1 be invisible to a GCSFuse mount.

[gargnitingoogle]

This PR is causing a problem with rm -r on a nested directory, so marking it as draft until that issue is resolved.

[gargnitingoogle]

Undo this.