changefeedccl: do not retry forever without making progress

[miretskiy]

Changefeed will retry forever when sink is unavailable. This behavior might not be desirable
for multiple reasons:

• Failures to write to sink are expensive; they result in the the dist flow process to be torn and recreated every 10 seconds.
• Retrying forever is confusing to end users: the job is "running", yet nothing is happening (though recent changes to surface retries in running status might help).
• Some sinks (s3) already have plenty of retires built into the client -- retrying (forever) might not add that much.

I think the fix might be fairly straightforward: retry limited number of times when we don't advance low watermark (i.e. we're not making forward progress).

Jira issue: CRDB-2768

[amruss]

Open questions: How do we alert the user when there is an action they need to take? What should our "retry limit" be? Should we increase the backoff?

[amruss]

Linking: #58077

[miretskiy]

I think doing something as part of the issue is important since we now hold onto PTS record.
Retrying forever would be bad.