Spark 3.5: Fix NotSerializableException when migrating Spark tables

[manuzhang]

This fixes #11147

cc @nastra @aokolnychyi @zzeekk

[manuzhang]

@nastra could you help take a look?

[RussellSpitzer]

@manuzhang Can you summerize the usage of ExecutorService on the Spark Executors? It looks like the current fix involves making a new Executor service per task and i'm not sure that's what we want to do. I wonder if it makes more sense to pass in a Supplier so we don't have to implement a wrapper class.

But before we do that I want to make sure we are using the ExecutorService for the right reasons in the code path.

[manuzhang]

ExecutorService is used to parallelize reading files to build manifests on the Spark executors for Spark table migration procedures (add_files, migrate, snapshot).

[RussellSpitzer]

ExecutorService is used to parallelize reading files to build manifests on the Spark executors for Spark table migration procedures (add_files, migrate, snapshot).

I meant specifically, are we using this in listPartitions or in the FileIndex implementation?

The title of this issue says this is an issue for "partitioned tables" so why does it work in that case but not in this case? Is it because the listPartitions code is using the executor service or what?

[manuzhang]

Yes, it's used in listPartitions while the title was not accurate. Migrating unpartitioned Spark tables has the same issue. I've updated the title.

[RussellSpitzer]

After thinking about this for a while, I think you are probably right that we need to build a specific LazyExecutorService like you did originally. I'm sorry I lend you on a goose chase here, Let's make sure it is Spark specific and doesn't touch any of the other implementations.

[manuzhang]

@RussellSpitzer I can revert to previous commit and this is Spark specific, but can you elaborate on why LazyExecutorService is better thanSerializableSupplier? I agree with you that people can pass around a non-lazy ExecutorService in future implementations.

I submitted #11417 to add warning in the doc since this PR can't get into 1.7.0

[RussellSpitzer]

Main reason was that the API that is specified in the API Module allows withExecutorService(executor service) so unless we want to break that api (which we could) we need to stick with just passing through an executor service. We could alternatively just change the API if you think that's warrrented. If we did that, I'd probably remove "executor service" all together

[manuzhang]

@RussellSpitzer I've reverted to lazy executor service. Please check again. Thanks.