[Datasets] Handle any error code when encountering AWS S3 credential error

[c21]

Why are these changes needed?

As a followup of #26619 (comment) and #26619 (comment), here we change from PermissionError to OSError, to be consistent as original error, and also change function name from _handle_read_s3_files_error to _handle_read_os_error, which is more general that we can handle other file systems such as GCS in the future.

Also change to hanlde any error message with pattern AWS Error [code xxx]: No response body as new issue with error code 100 is raised in #26672 .

Tested the example in #26672 and verified it shows new error message now:

>>> import ray
>>> ds = ray.data.read_parquet([
...     "s3://anonymous@ursa-labs-taxi-data/2009/01/data.parquet",
...     "s3://anonymous@ursa-labs-taxi-data/2009/02/data.parquet",
... ])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/chengsu/ray/python/ray/data/read_api.py", line 365, in read_parquet
    return read_datasource(
  File "/Users/chengsu/ray/python/ray/data/read_api.py", line 267, in read_datasource
    requested_parallelism, min_safe_parallelism, read_tasks = ray.get(
  File "/Users/chengsu/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/Users/chengsu/ray/python/ray/_private/worker.py", line 2196, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OSError): ray::_get_read_tasks() (pid=37757, ip=127.0.0.1)
  File "/Users/chengsu/ray/python/ray/data/datasource/file_meta_provider.py", line 330, in prefetch_file_metadata
    return _fetch_metadata(pieces)
  File "/Users/chengsu/ray/python/ray/data/datasource/parquet_datasource.py", line 341, in _fetch_metadata
    piece_metadata.append(p.metadata)
  File "pyarrow/_dataset.pyx", line 1315, in pyarrow._dataset.ParquetFileFragment.metadata.__get__
  File "pyarrow/_dataset.pyx", line 1304, in pyarrow._dataset.ParquetFileFragment.ensure_complete_metadata
  File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
OSError: When reading information for key '2009/02/data.parquet' in bucket 'anonymous@ursa-labs-taxi-data': AWS Error [code 100]: No response body.

During handling of the above exception, another exception occurred:

ray::_get_read_tasks() (pid=37757, ip=127.0.0.1)
  File "/Users/chengsu/ray/python/ray/data/read_api.py", line 1129, in _get_read_tasks
    reader = ds.create_reader(**kwargs)
  File "/Users/chengsu/ray/python/ray/data/datasource/parquet_datasource.py", line 142, in create_reader
    return _ParquetDatasourceReader(**kwargs)
  File "/Users/chengsu/ray/python/ray/data/datasource/parquet_datasource.py", line 197, in __init__
    _handle_read_os_error(e, paths)
  File "/Users/chengsu/ray/python/ray/data/datasource/file_meta_provider.py", line 345, in _handle_read_os_error
    raise OSError(
OSError: Failing to read AWS S3 file(s): ['ursa-labs-taxi-data/2009/01/data.parquet', 'anonymous@ursa-labs-taxi-data/2009/02/data.parquet']. Please check that file exists and has properly configured access. See https://docs.ray.io/en/latest/data/creating-datasets.html#reading-from-remote-storage for more information.

Related issue number

Closes #26672 .

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[c21]

@matthewdeng - could you help take a look when you have time? Thanks.

[c21]

This is just move the original comment literally without any change. Feel make more sense to comment inside the if branch, as we plan to extend the function to apply to other file system such as GCS in the future.

[matthewdeng]

Add a quick note to make it explicit that this is not comprehensive yet, and should be extended as more errors arise?

[c21]

@matthewdeng - sure, added.

[c21]

@pcmoritz, @dmatrix - this should fix #26672, please let me know if the error message below is intuitive enough, thanks.

OSError: Failing to read AWS S3 file(s): ['ursa-labs-taxi-data/2009/01/data.parquet', 'anonymous@ursa-labs-taxi-data/2009/02/data.parquet']. 
Please check that file exists and has properly configured access.
See https://docs.ray.io/en/latest/data/creating-datasets.html#reading-from-remote-storage for more information.

[c21]

Thanks @scv119 and @matthewdeng for review!

[matthewdeng]

@pcmoritz, @dmatrix - this should fix #26672, please let me know if the error message below is intuitive enough, thanks.

OSError: Failing to read AWS S3 file(s): ['ursa-labs-taxi-data/2009/01/data.parquet', 'anonymous@ursa-labs-taxi-data/2009/02/data.parquet']. 
Please check that file exists and has properly configured access.
See https://docs.ray.io/en/latest/data/creating-datasets.html#reading-from-remote-storage for more information.

Do you know if there are any resources describing what these error codes (e.g. 15, 100) mean? I feel like it's still not clear which issue is occurring.

[pcmoritz]

They are here: https://sdk.amazonaws.com/cpp/api/LATEST/_s3_errors_8h_source.html

[c21]

@matthewdeng - my guess when user encounters the issue, is they will (1). first check the file exists (Please check that file exists) then (2). check access / credential configured properly (has properly configured access).

From @pcmoritz 's link, it looks like error code 15 is ACCESS_DENIED, and 100 isUNKNOWN, not giving out too much information. btw I also tried locally for reading a non-existent file, error code is also 15.

[matthewdeng]

@pcmoritz and I briefly chatted about how we could improve the UX so it is easier for the user to check if the file exists and their credentials.

One idea that comes from Philipp's manual debugging is to show them the s3 CLI/SDK command they can run to get a more verbose error message (e.g. #19799 (comment) or #19799 (comment)).