POC: Enable Istio sidecar injection in NATS Pods

[mfaizanse]

Context:
Currently, the sidecar injection to NATS Pods are disabled due to performance reasons. Meaning that the NATS Pods are not part of service mesh.

We want to now enable sidecar injection to NATS Pods due to security concerns. For that we need to do a POC to check if the NATS will work with sidecars and also will there be any side effects like performance etc.

Task(s):

1. Check official NATS documentation for any guidance for running NATS inside a service mesh (Istio).
2. Update nats-manager to deploy NATS with sidecar injection enabled.

• Changesidecar.istio.io/inject label to true (reference).
• Do not merge these changes to nats-manager main branch. Keep it into your fork.

3. Check if sidecars are injected into NATS Pods and check if NATS is working end-to-end. Make sure everything is working as expected.

• Manual tests + load tests
• If NATS is not working, then investigate the reason and any possible solutions.

4. Check the upgrade flow of existing clusters which have sidecar disabled, to version with sidecar enabled.
   • Test the upgrade with load tests deployed.
   • Do we need any extra step for it.
   • Would there be any downtime? or any risks?
5. Evaluate if we should have a flag in NATS CR to enable/disable sidecar injection or should we enable it by default on all existing and new clusters.

[muralov]

Enabled the mTLS for NATS cluster and here is the result:

• sending events manually worked as expected
• a pod outside the mesh was not able to connect NATS cluster
• load test gave the following log output that looks good
  nats-clsuter.txt
• here is the draft PR for the changes
• Upgrade to NATS cluster with sidecars didn't work due to connection issue among nodes. We may use PERMISSIVE for an initial release and apply STRICT in a next release.

[muralov]

I've tested that upgrade worked with two steps:

1. Having PeerAuthentication in PERMISSIVE mode first, which allows non tls traffic too
2. Change it to have STRICT mode, which only allows mTLS

Allowing non tls allows NATS nodes with sidecars to build cluster first.
I again tested that directly going to STRICT mode breaks the NATS cluster.

Here is the NATS cluster screenshot that shows NATS nodes with sidecars:

[marcobebway]

I've tested that upgrade worked with two steps:

1. Having PeerAuthentication in PERMISSIVE mode first, which allows non tls traffic too
2. Change it to have STRICT mode, which only allows mTLS

Allowing non tls allows NATS nodes with sidecars to build cluster first. I again tested that directly going to STRICT mode breaks the NATS cluster.

Findings looks good.