Backport of [NET-10544] bugfix: catalog sync fails repeatedly to deregister a service in some scenarios into release/1.5.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #4266 to be assessed for backporting due to the inclusion of the label backport/1.5.x.

The below text is copied from the body of the original PR.

---

Changes proposed in this PR

• Stop using blocking queries for listing services for potential reaping
  • This can hurt us when we use blocking queries and no change happens during the wait period, which causes the request to error and retry forever when we actually just needed the existing list of catalog services so that we could diff against the existing list of k8s services
• Add a limit to the backoff-retry mechanism so that we don't infinitely retry using a bad client
  • This can hurt us when consul-server-connection-manager isn't able to get a valid IP, for example, and the client will be configured with <nil> for the host and retry forever

How I've tested this PR

1. Install Consul to your k8s cluster with catalog sync enabled and this build for global.imageK8S
2. Create a k8s Service selecting one or more Pods
3. Delete the consul-server Pod
4. Delete the k8s Service
5. Observe that the corresponding Consul service is now deregistered where it was not before this change

values.yaml

global:
  name: consul
  datacenter: dc1

  imageK8S: consul-k8s-control-plane:dev

  tls:
    enabled: true
    enableAutoEncrypt: true
  
  acls:
    manageSystemACLs: true

  metrics:
    enabled: true

connectInject:
  enabled: false

syncCatalog:
  enabled: true
  metrics:
    enabled: true

prometheus:
  enabled: true

ui:
  enabled: true
  metrics:
    provider: "prometheus"
    baseURL: http://prometheus-server.prometheus

How I expect reviewers to test this PR

See above

Checklist

• Tests added
• CHANGELOG entry added

---

Overview of commits

• bbfb5dc - 37f731f - e94849b