TiDB report store-not-found error

[youjiali1995]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Force offline a store to turn it into a tombstone store and some regions still have peers on it.
2. TiDB doesn't cache the tombstone store's information so it will query PD.
3. TiDB returns a store-not-found error to the user.

RegionCache can't handle peers on a tombstone store now. If TiDB tries to access the peer, TiDB will report backoff timeout. If the tombstone information is deleted from the PD, TiDB will return a store-not-found error to the user.

tidb/store/tikv/region_cache.go

Lines 1779 to 1795 in 18cbfaa

	store, err = c.pdClient.GetStore(bo.GetCtx(), s.storeID)
	if err != nil {
	metrics.RegionCacheCounterWithGetStoreError.Inc()
	} else {
	metrics.RegionCacheCounterWithGetStoreOK.Inc()
	}
	if err != nil {
	// TODO: more refine PD error status handle.
	if errors.Cause(err) == context.Canceled {
	return
	}
	err = errors.Errorf("loadStore from PD failed, id: %d, err: %v", s.storeID, err)
	if err = bo.Backoff(retry.BoPDRPC, err); err != nil {
	return
	}
	continue
	}

BTW, RegionCache doesn't retry if it fails to call GetStore. The error will be reported to the user too.

tidb/store/tikv/region_cache.go

Lines 198 to 201 in 18cbfaa

	_, err := store.initResolve(retry.NewNoopBackoff(context.Background()), c)
	if err != nil {
	return err
	}

2. What did you expect to see? (Required)

TiDB works normally.

3. What did you see instead (Required)

TiDB report an error to the user.

4. What is your TiDB version? (Required)

All version

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions