[Spark] Support subDirs parameter for VACUUM

[sezruby]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Support subDirs parameter for VACUUM command. Currently VACUUM command builds a list of all files under the root table location and exclude valid files from the list. This operation could cause huge storage cost when >10M of valid files under the table and need to run VACUUM frequently for few files. For now there is no way to avoid listing all files for VACUUM operation.

This PR adds subDirs parameter for VACUUM Scala/Python API to limit the candidate files. If subDirs parameter is given, only file paths under the directories will be considered for VACUUM.

It can be beneficial for the following scenario:

• A table maintains millions of files, keeps getting new data and needs to VACUUM frequently.
• If a table is partitioned by date type, it's clear which partitions need to be vacuumed.
• If there are many invalid files and a user want to run vacuum partially due to of lack of resources, etc.

For WHERE clause support in #1691, we usually use the clause to "filter" files. So we have to make the whole list first for filtering. Otherwise it needs to check the predicate before stepping into each subdirectory recursively, which requires a lot of code changes. This PR could be a workaround for #1691. There was the same request #220

How was this patch tested?

Unit tests, ran in production for months

Does this PR introduce any user-facing changes?

Yes, support new parameter for VACUUM.

[tnyz]

can we release this feature?

[sezruby]

@tnyz closed as better approach - #1932 was going to merge. (the latest design seems bit complex now though)

This PR could be a quick workaround for expensive dir command problem while vacuum.