[data] Stability & accuracy improvements for Data+Train benchmark

[Zandew]

Why are these changes needed?

• Shuffles input images for read_images_train_4_gpu release test, which fixes the issue with accuracy going to 0.
• Add AWS Error NETWORK_CONNECTION and AWS Error ACCESS_DENIED as an Exception type to retry during reads, since this can be a transient error that is fine upon retry.
• Other small fixes for optional parameters in benchmark file, used for debugging purposes.

Results of sample release test run:

• read_images_train_4_gpu:

Result of case cache-none: {'time': 11964.644112934, 'tput': 429.22158930338344, 'accuracy': 0.4667895757295709, 'extra_metrics': {}}

• read_images_train_16_gpu:

Result of case cache-none: {'time': 5400.357632072, 'tput': 1593.6668981608586, 'accuracy': 0.5293150227295434, 'extra_metrics': {}}

• read_images_train_16_gpu_preserve_order:

Result of case cache-none: {'time': 5566.524269388, 'tput': 1571.1312653719967, 'accuracy': 0.5295374787691078, 'extra_metrics': {}}

(The difference is accuracy is because the 4 worker test only runs for 3 epochs, the 16 worker test runs for 5 epochs, using the entire dataset per epoch.)

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[raulchen]

This has been repeated multiple times in the code base. can we consolidate them with a util function?

[scottjlee]

@raulchen i intended to keep these constants separate from the retry errors for opening files, e.g.

ray/python/ray/data/datasource/file_based_datasource.py

Lines 67 to 74 in bfc1f78

	# The errors to retry for opening file.
	OPEN_FILE_RETRY_ON_ERRORS = ["AWS Error SLOW_DOWN"]
	# The max retry backoff in seconds for opening file.
	OPEN_FILE_RETRY_MAX_BACKOFF_SECONDS = 32
	# The max number of attempts for opening file.
	OPEN_FILE_MAX_ATTEMPTS = 10

i'm using the common util function call_with_retry with these constants. do you mean consolidate these sets of constants into their own file?

[raulchen]

ok. might be better to also define those constants in a unified place. but not a big deal.

[scottjlee]

i couldn't find a centralized constants.py or other similar file for ray data constants. should we expose these as parameters from DataContext?

[raulchen]

okay to keep it the current way. let's refactor later if needed.

[raulchen]

I just realized that we already have write_file_retry_on_errors in DataContext`. Let's also follow this pattern?

[raulchen]

ok. might be better to also define those constants in a unified place. but not a big deal.

[raulchen]

the input files list may be very large, and will make the error message too verbose.

[scottjlee]

good point, updated to just use read fn name by default for all cases (since we have no control over the length of even one file name)