fs load succeeded but no file processed

[wwq2333]

Alluxio Version:
2.9.3

Describe the bug

for the preheated alluxio cluster, when the worker is restarted due to oom, use fs loadmeta and fs load --verify to warm up again;

./bin/alluxio fs load / --submit --partial-listing --verify
./bin/alluxio fs load / --progress
Progress for loading path '/':
	Settings:	bandwidth: unlimited	verify: true
	Job State: SUCCEEDED
	Files Processed: 0
	Bytes Loaded: 0B
	Throughput: 0B/s
	Block load failure rate: 0.00%
	Files Failed: 0

But using fs ls can still find that there are no cached files. and alluxio cluster has enough capacity:

./bin/alluxio fs ls /write_newtrain_facepoints_temp.py
           1507       PERSISTED 07-26-2023 12:54:49:659   0% /write_newtrain_facepoints_temp.py


fsadmin report
Total Capacity: 3120.00GB
        Tier: SSD  Size: 3120.00GB
    Used Capacity: 777.11GB
        Tier: SSD  Size: 777.11GB
    Free Capacity: 2342.89GB

master, worker no err log, try warming up a specific file instead of /

./bin/alluxio fs ls /write_newtrain_facepoints_temp.py
           1507       PERSISTED 07-26-2023 12:54:49:659   0% /write_newtrain_facepoints_temp.py


./bin/alluxio fs load /write_newtrain_facepoints_temp.py --submit --partial-listing --verify
Load '/write_newtrain_facepoints_temp.py' is successfully submitted. JobId: c2facde4-d3f4-4f1d-be0a-4a26aa9ca78e


./bin/alluxio fs load /write_newtrain_facepoints_temp.py --progress
Progress for loading path '/write_newtrain_facepoints_temp.py':
	Settings:	bandwidth: unlimited	verify: true
	Job State: FAILED (alluxio.exception.runtime.InternalRuntimeException: Job failed because it's not healthy.)
	Files Processed: 2052
	Bytes Loaded: 7.36KB
	Block load failure rate: 99.76%
	Files Failed: 1


./bin/alluxio fs ls /write_newtrain_facepoints_temp.py
           1507       PERSISTED 07-26-2023 12:54:49:659 100% /write_newtrain_facepoints_temp.py

worker no err log, master err log:

2023-08-17 06:18:36,790 WARN  [master-rpc-executor-TPE-thread-297](DefaultBlockMaster.java:1133) - Rejecting attempt to change block length from 1507 to 0
2023-08-17 06:18:36,792 WARN  [master-rpc-executor-TPE-thread-333](DefaultBlockMaster.java:1133) - Rejecting attempt to change block length from 1507 to 0
2023-08-17 06:18:36,796 WARN  [master-rpc-executor-TPE-thread-402](DefaultBlockMaster.java:1133) - Rejecting attempt to change block length from 1507 to 0
2023-08-17 06:18:36,798 WARN  [master-rpc-executor-TPE-thread-405](DefaultBlockMaster.java:1133) - Rejecting attempt to change block length from 1507 to 0

Additional context

[ssz1997]

@jja725 Could you take a look?

[jja725]

As we discussed in the meeting, this seems like a master metadata issue since the master thinks the data is already in the cluster but the new worker actually doesn't have the data(no Persistency). This is not reproducible in the test environment.
In the new architecture this should not happen since we check file cache status at worker side

[jja725]

Fixed by #18133