sql/gcjob: retry failed GC jobs

[sajjadrizvi]

In the previous implementation, failed GC jobs were not being retried regardless
whether the failure is permanent or transient. As a result, a GC job's failure
risked orphaned data, which cannot be reclaimed.

This commit adds a mechanism to retry failed GC jobs that are not permanent. No
limit is set on the number of retries. For the time being, the failure type is
determined based on the failure categorization of schema-change jobs. This
behavior is expected to change once exponential backoff mechanism is
implemented for failed jobs (#44594).

Release note: None

Fixes: #65000

Release note (<category, see below>):

[cockroach-teamcity]

This change is 

[ajwerner]

One small thing, otherwise

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @sajjadrizvi)

---

pkg/sql/gcjob/gc_job.go, line 86 at r1 (raw file):

	defer func() {
		if err != nil && !r.isPermanentGCError(err) {
			err = jobs.NewRetryJobError(err.Error())

these APIs are confusing, but I think it'd be better to do the following:

err = errors.Mark(err, jobs.NewRetryJobError("gc"))

This way we'll preserve the structure for logging rather than just redacting the entire string.

You can see this if you stick the following in a test in the jobs package:

func TestMarkWorks(t *testing.T) {
	err := errors.New("boom")
	marked := errors.Mark(err, NewRetryJobError("foo"))
	require.True(t, errors.Is(marked, retryJobErrorSentinel))
}

[sajjadrizvi]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner)

---

pkg/sql/gcjob/gc_job.go, line 86 at r1 (raw file):

Previously, ajwerner wrote…

these APIs are confusing, but I think it'd be better to do the following:

err = errors.Mark(err, jobs.NewRetryJobError("gc"))

This way we'll preserve the structure for logging rather than just redacting the entire string.

You can see this if you stick the following in a test in the jobs package:

func TestMarkWorks(t *testing.T) {
	err := errors.New("boom")
	marked := errors.Mark(err, NewRetryJobError("foo"))
	require.True(t, errors.Is(marked, retryJobErrorSentinel))
}

Got it! thanks for pointing it out. I have updated the code.

[sajjadrizvi]

bors r=ajwerner

[craig]

Build succeeded:

• GitHub CI (Cockroach)