Make c2c networking work

[viovanov]

Is your feature request related to a problem? Please describe.
c2c networking is not enabled in KubeCF.

Describe the solution you'd like

• implement a CoreDNS plugin that replaces the bosh dns adapter
• enable the CATS group for c2c networking
• support in Quarks

Describe alternatives you've considered
n/a

Additional context
This will likely require more containers with extra privileges and support in Quarks for more BOSH DNS features.

[fargozhu]

@viovanov do we have any use cases to support this spike?

[viovanov]

@fargozhu this is a feature of CF that has many usecases, and is also requested by our IBM friends
ping @qu1queee

[smoser-ibm]

@viovanov this would be the last missing piece for us regarding a CF Bosh -> K8s migration. Can we bump the importance of this maybe ?

[qu1queee]

This is on my radar, trying to find someone from our side to help, I dont have too much bandwidth for this atm.

[viovanov]

@JimmyMa can you please add a comment here with your setup and your findings?

[mook-as]

This is expected to run (and fail) on minikube.

[viovanov]

We need to implement an "App DNS" as described below:

[viovanov]

Ping @JimmyMa ⬆️
@f0rmiga is starting work on that app dns server

[f0rmiga]

A few notes based on our call:

• A patch to the silk-cni job is most likely needed to be able to get the k8s svc cluster IP from the injected env var to be used by the Garden Container /etc/resolv.conf. We have done similar patches in the past, so it should be simple.
• Configuring the Apps DNS to use the Quarks DNS as the upstream server might be tricky unless we make the Apps DNS a BOSH-release and let the quarks-operator do the magic to point its /etc/resolv.conf to the Quarks DNS. That's not going to be my first attempt but I'll certainly fall back to this approach to avoid doing any clever tricks that might compromise future maintainability.
• The Apps DNS should contain a CoreDNS plugin that calls the Service Discovery Controller.

[f0rmiga]

For the Apps DNS configuration file, we don't actually need to use /etc/resolv.conf as upstream, we can just generate the CoreDNS configuration that resolves the Quarks DNS svc IP during the Apps DNS initialization.

[f0rmiga]

For the Apps DNS plugin to talk to the Service Discovery Controller, it uses mTLS, so it's easier if we just call the service-discovery-controller.service.cf.internal as all the certs are generated for this common name already. For that, we can use a custom resolver for the Go HTTPS client used by the svc discovery client. This is doable because at this point we already know the Quarks DNS IP address (see my previous comment).

Thanks to @mook-as for helping me think this through. Also, it integrates well with the Eirini extension that sets the nameservers for Eirini apps. Cross-referencing: #284 and SUSE/eirini-dns-aliases#1.

[smoser-ibm]

@andrew-edgar ^^ for your awareness (and to make sure its not impacting your merge-DNS solution you build for us). I don't think it does, but want your awareness and confirmatio

[viovanov]

I've changed this issue so now it tracks the work that we're doing to make c2c work.

[viovanov]

• onfiguring the Apps DNS to use the Quarks DNS as the upstream server might be tricky unless we make the Apps DNS a BOSH-release and let the q

@f0rmiga the Quarks DNS Kube Service should be available before the Apps DNS is started.
That means you can use the env var for it in the Apps DNS. Is there anything else missing?

Just noticed your other comments :).