Best practice for checking if KF web apps are reachable during integration tests

[orfeas-k]

Context

Having as a starting point this comment by @DnPlas, we started a discussion as to how we should check if a WA is reachable during integration tests. cc @ca-scribner

Issue

Until now, in some WAs (TWA with this PR and JUI (Jupyter-UI) ) we made the above check by sending a request straight to the IP of the WA's unit. This indeed checks that the service backing the WA is working. However, this doesn't ensure that the WA will be reachable during deployment since it won't catch the case where TWA asks for the ingress incorrectly. It doesn't check essentially that requests are routed correctly.

Regarding other WAs, at the moment

• VWA (volumes) doesn't have a test for this right now, but it used to actually have sth like the described above with selenium where it reached the UI through the IP of the istio-ingressgateway-workload service
• Istio-operator bundle tests also deploy VWA and reach VWA's UI through the IP of the istio-ingressgateway-workload service.
• KUI (katib-ui) doesn't have any test to check if the UI is reachable
• Dashboard has a test where it reaches its UI but by using the application's IP which (to my understanding) is equivalent to the unit's IP.

Proposed solution

In order to make sure that requests go to the correct route, we should test through Istio's IngressGateway. However the following question arises:

Will we need to go through authentication?

The question is if this authentication flow will be triggered when we send a request to Istio's IngressGateway. If yes, then we will need to also deploy OIDC-Authservice and Dex-Auth alongside.

I 'm adding a detailed analysis of Istio's workflow that @kimwnasptd provided us.

The Istio/AuthService/Dex authentication flow happens for any request that hits the Istio IngressGateway. This means that when a request hits the Istio IngressGateway:

1. IngressGateway will send this request to AuthService
2. AuthService will decide if the request is authenticated or not (by checking a cookie, or the Authorization header)
3. If not it will do the 302 and tell users to login etc (Authorization Code OIDC flow)
4. if the request is authenticated then it will add an extra kubeflow-userid: header in the request and respond to Istio IngressGateway that the request is authenticated

However, we see that in all tests above there is neither OIDC-Authservice neither Dex-Auth deployed.

[kimwnasptd]

My 2 cents on this one would be:

• I'd keep just testing hitting the unit/k8s-svc IP directly
• Have the Istio IngressGateway test as bundle or Istio Charm test

The main reasons being that:

1. Apps' VirtualService configuration almost never changes, so no need to do these tests on the apps
2. The biggest cause of error could be Istio itself and how we setup the IngressGateway (whenever I say IngressGateway I'm referring to the Pod that gets configured by Istio Gateway CRs)

That's why I think it makes more sense to move this tests closer to where the more frequent breaking changes are, or to the bundle (which usually is the fallback case of testing)

[orfeas-k]

Summarizing @kimwnasptd comment, let's test the web apps by using the unit/k8s-svc IP directly instead of going through istio's ingressgateway.

[ca-scribner]

One thing to keep in mind is that Istio adds the kubeflow-userid header to traffic passing through the gateway toward the webapp. Normally the webapps need this header to function properly. To work around this we could add the header to our traffic manually, or some webapps have a development mode that can be enabled so it ignores the header.

These are not a blocker, but good to keep in mind when writing tests.

[orfeas-k]

An example to implementing this in tests can be seen here https://github.com/canonical/notebook-operators/pull/271/files#diff-d36bbd65e732dee2edad9c591d360a76f178d9c56b6a73aa7ba27eed527bd2d4.