Lightning cannot ignore broken symlink

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

mkdir dsd
cd dsd
ln -s non-existing-file.png file-that-should-be-ignored.png
cd ..
./tidb-lightning -d ./dsd --backend tidb --tidb-port 4000

2. What did you expect to see? (Required)

tidb lightning exit successfully

(nothing should be imported because the directory is effectively empty, the file name of the broken symlink does not match the pattern db.tbl.123.csv for instance)

3. What did you see instead (Required)

tidb lightning encountered error: [Lightning:Storage:ErrStorageUnknown]unknown storage error: stat ./dsd/file-that-should-be-ignored.png: no such file or directory

4. What is your TiDB version? (Required)

Release Version: v7.5.0
Git Commit Hash: 069631e2ecfedc000ffb92c67207bea81380f020
Git Branch: heads/refs/tags/v7.5.0
Go Version: go1.21.3
UTC Build Time: 2023-11-24 08:46:08
Race Enabled: false

Also affects v6.5.4.

[kennytm]

The issue is caused by trying to obtain the "real size" of a symlink. The error is thrown from line 133.

tidb/br/pkg/storage/local.go

Lines 128 to 136 in 49d1b66

	size := f.Size()
	// if not a regular file, we need to use os.stat to get the real file size
	if !f.Mode().IsRegular() {
	stat, err := os.Stat(filepath.Join(l.base, path))
	if err != nil {
	return errors.Trace(err)
	}
	size = stat.Size()
	}

The fix should also consider the following scenarios:

1. The broken symlink do have the name pattern db.tbl.123.csv, but the table db.tbl got filtered out by -f
2. As above, but the table db.tbl is included.

There should still be no errors in scenario 1 as the file db.tbl.123.csv won't be read anyway. For scenario 2 I think we should not skip the broken file.

So IMO the proper fix is

1. Turn line 133 into a WARNING log.
2. Return a dummy size (0 or 1) for such files.