client/v3: Add backoff before retry when watch stream returns unavailable

[tomari]

The client retries connection without backoff when the server is gone after the watch stream is established. This results in high CPU usage in the client process. This change introduces backoff when the stream is failed and unavailable.

Signed-off-by: Hisanobu Tomari [email protected]

[ahrtr]

The code is duplicated to

etcd/client/v3/watch.go

Lines 1018 to 1028 in 6a0bbf3

	if isUnavailableErr(w.ctx, err) {
	// retry, but backoff
	if backoff < maxBackoff {
	// 25% backoff factor
	backoff = backoff + backoff/4
	if backoff > maxBackoff {
	backoff = maxBackoff
	}
	}
	time.Sleep(backoff)
	}

Please get it included in a function so that it can be reused by both places.

[tomari]

Thanks for the review. The backoff function has been added.

[ahrtr]

LGTM

Thank you @tomari

[codecov-commenter]

Codecov Report

Merging #14556 (6a0bbf3) into main (6a0bbf3) will not change coverage.
The diff coverage is n/a.

❗ Current head 6a0bbf3 differs from pull request most recent head 428fb96. Consider uploading reports for the commit 428fb96 to get more accurate results

@@           Coverage Diff           @@
##             main   #14556   +/-   ##
=======================================
  Coverage   75.46%   75.46%           
=======================================
  Files         457      457           
  Lines       37187    37187           
=======================================
  Hits        28064    28064           
  Misses       7373     7373           
  Partials     1750     1750           

Flag	Coverage Δ
all	75.46% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[spzala]

lgtm. Thanks @tomari

[ahrtr]

@tomari Could you backport the fix to 3.5 and 3.4?