sql: GC job may fail with error which should be retried

[ajwerner]

Describe the problem

The GC job should very rarely fail. There is a general fear that some errors may be permanent. Nevertheless, today roughly no errors get retried. That's madness. This can leave orphaned data in the keyspace.

To Reproduce

Kill nodes during the execution of a GC job. See an error.

Expected behavior

The job will get retried.

Additional context

Relates to and may replace #55740.

We could (should?) switch the default to retry everything once we have backoff (#44594).

In the schema changer we have some bespoke retry logic that we could mostly reuse.

cockroach/pkg/sql/schema_changer.go

Lines 141 to 145 in 8bfaaff

	// isPermanentSchemaChangeError returns true if the error results in
	// a permanent failure of a schema change. This function is a allowlist
	// instead of a blocklist: only known safe errors are confirmed to not be
	// permanent errors. Anything unknown is assumed to be permanent.
	func isPermanentSchemaChangeError(err error) bool {

[ajwerner]

For backport to 21.1 and 20.2 we should classify errors using the linked function or some part of it and then for the fuller solution we can just retry everything with backoff.

[ajwerner]

For the backport, something like:

--- a/pkg/sql/gcjob/gc_job.go
+++ b/pkg/sql/gcjob/gc_job.go
@@ -80,7 +80,12 @@ func performGC(
 }
 
 // Resume is part of the jobs.Resumer interface.
+func (r schemaChangeGCResumer) Resume(ctx context.Context, execCtx interface{}) (err error) {
+       defer func() {
+               if !isPermanent(err) {
+                       err = jobs.NewRetryJobError(err.Error())
+               }
+       }()
        p := exe