[SPARK-34039][SQL] ReplaceTable should invalidate cache

[sunchao]

What changes were proposed in this pull request?

This changes ReplaceTableExec/AtomicReplaceTableExec, and uncaches the target table before it is dropped. In addition, this includes some refactoring by moving the uncacheTable method to DataSourceV2Strategy so that we don't need to pass a Spark session to the v2 exec.

Why are the changes needed?

Similar to SPARK-33492 (#30429). When a table is refreshed, the associated cache should be invalidated to avoid potential incorrect results.

Does this PR introduce any user-facing change?

Yes. Now When a data source v2 is cached (either directly or indirectly), all the relevant caches will be refreshed or invalidated if the table is replaced.

How was this patch tested?

Added a new unit test.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38370/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38370/

[SparkQA]

Test build #133782 has finished for PR 31081 at commit d8ea7b6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thank you, @sunchao .
According to the affected version of SPARK-34039, this is merged to master.

cc @cloud-fan , @aokolnychyi

[dongjoon-hyun]

@sunchao . I tested this case with Apache Spark 3.1.0 RC1 with Iceberg 0.10. This is a correctness issue. How do you think about that? If you agree, please make a backport. (Also, cc @HyukjinKwon )

{code}
spark-sql> SELECT * FROM local.db.table;
4	5

spark-sql> CACHE TABLE v AS SELECT * FROM local.db.table;

spark-sql> SELECT * FROM v;
4	5

spark-sql> REPLACE TABLE local.db.table AS SELECT 100 id, '100' string;

spark-sql> SELECT * FROM v;
4	5

spark-sql> SELECT * FROM local.db.table;
100	100

[sunchao]

Thanks @dongjoon-hyun . Did the above include this PR? There is another issue SPARK-34052 on handling temporary view in DSv2 which could cause the above as well.

[dongjoon-hyun]

No~ As I wrote, I used Apache Spark 3.1.0 RC1 with Iceberg 0.10 to see the existing behavior.

[sunchao]

Got you. Yes I think we should backport this (I'll create one soon). In addition, we may need SPARK-34052 to completely address the issue.

In the second:

spark-sql> SELECT * FROM v;
4	5

IMO, the temporary view v should become invalid once the source table local.db.table is replaced.

[aokolnychyi]

It would be great to see this fixed in 3.1.0. Thanks for the work!

[dongjoon-hyun]

I agree with you, @sunchao .

[HyukjinKwon]

Sure, I think RC1 is failed. Thanks for letting me know @dongjoon-hyun. +1 for porting it back.

[HyukjinKwon]

BTW, do we have more issues to fix like this? I have seen similar cache/uncache fixes often lately. If we have some issues to fix already, it might be best to delay the next RC until we have the fixes.

[sunchao]

@HyukjinKwon I'm working on #31107 which also seems to be a bug on caching. Yes it seems there are many issues related to this area recently, and I feel we haven't identified all of them yet.

[sunchao]

BTW some of the cache fixes are not in branch-3.0, @HyukjinKwon @dongjoon-hyun let me know if it's necessary to make the backports.

[dongjoon-hyun]

Thanks, @sunchao . Yes. It would be great if we have it if the UT fails there in the same way.

[HyukjinKwon]

@sunchao and @MaxGekk, can we have an umbrella ticket or epic ticket (if an umbrella ticket is not possible) to group these caching / uncaching issues? Let's block RC until we feel sure these issues are fixed then.

[sunchao]

@HyukjinKwon yes I've tried to put all these under https://issues.apache.org/jira/browse/SPARK-33507. @MaxGekk do you mind adding yours to here too?

[HyukjinKwon]

@MaxGekk
if your goal is different from SPARK-33507, you can also file another umbrella ticket and link SPARK-33507 with incooperates link.
If the goal looks similar or a subset, please move and file JIRAs under that one.

[MaxGekk]

I have converted my tickets to sub-tasks of SPARK-33507