Stability improvements to conversion webhooks

[mbbush]

What problem are you facing?

When I installed provider-aws-kafka v1.0.0 (which contained a v1beta2 resource) into a cluster running crossplane 1.12, a TLS certificate error caused the provider to crash loop, which rendered all the resources it managed basically unusable. The TLS issue was fixed in crossplane-contrib/provider-upjet-aws#1157, and while I have not yet found any other issues that could cause the conversion webhooks to crash, I am concerned that their failure had such a large blast radius.

How could Upjet help solve your problem?

Could we run the conversion webhooks in such a way that if they panic, they don't take down the entire provider with them? Perhaps in a sidecar container?