Add configurable preStop delay to Envoy sidecar

[hamishforbes]

Changes proposed in this PR:

• Add a new annotation consul.hashicorp.com/sidecar-proxy-prestop-delay
• Pods with this annotation have a Lifecycle hook added to delay the Envoy proxy sidecar shutdown by the specified number of seconds

Should probably be some validation on the value of the annotation, atm this would allow random command injection to the sidecar container on shutdown

How I've tested this PR:

How I expect reviewers to test this PR:

Checklist:

• Tests added
• CHANGELOG entry added

  HashiCorp engineers only, community PRs should not add a changelog entry.
  Entries should use present tense (e.g. Add support for...)

Related: #650

[david-yu]

Thanks for the PR. We'd like to support a more graceful way of shutting down Envoy so that if the is still running Envoy shuts down after the app is shut down first. We will take this into consideration for future development of this feature.

I'm curious is this workaround working well in your environment?

[hamishforbes]

Yeah for sure, it's not a particularly elegant solution!

I've been running a fork with this patch and a couple of other tweaks in production since opening the PR though, works perfectly for me.

[hashicorp-cla]

All committers have signed the CLA.

[dschaaff]

I'd also note that I'm running a fork with the changes in this pull request. This feature really needs to get prioritized. There is no way to do rolling deployments without 5xx errors due to the envoy sidecar stopping before the appliation container completes.

[narendrapatel]

We are also testing this changes as we have some very high traffic apps and any issues gets quickly noticed and reported. A solution to this feature would be really helpful.

[alt-dima]

Can't understand, how Consul Connect/Mesh can be used in production without this?

[natitomattis]

We also have the same issue, we created this fork since our version (0.45.0) is behind main. We validated that adding the preStop hook fixed the problem.

[david-yu]

Closing as this is now resolved by #2233 and will be released in upcoming patch releases for 1.0.x, 1.1.x and 1.2.x.