ml-pipeline-ui failing on metadata api

[richardburakowski]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?

• Standalone kubernetes cluster v1.29.4
• kubeflow/manifests-v1.9.1-rc.1
• kubeflow/manifests-v1.9.0

Not seeing this in older versions of kubeflow/manifests, e.g. 1.6.1

• KFP version:
  2.3.0

Steps to reproduce

Navigate to kubeflow dashboard pipelines

An error occurred

upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: 111

Expected result

No dashboard error

Materials and Reference

image: gcr.io/ml-pipeline/frontend:2.3.0

kubectl -n kubeflow logs -l app=ml-pipeline-ui

GET /pipeline/
GET /pipeline/apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc
Proxied request:  /apis/v1beta1/pipelines?page_size=5&sort_by=created_at%20desc
GET /pipeline/apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined
Proxied request:  /apis/v1beta1/runs?page_size=5&sort_by=created_at%20desc&resource_reference_key.type=NAMESPACE&resource_reference_key.id=undefined
GET /pipeline/static/js/main.b980985e.js
GET /pipeline/static/css/main.e10b3034.css
GET /pipeline/apis/v1beta1/healthz
GET /pipeline/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
Proxied request:  /apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
GET /pipeline/system/project-id
GET /pipeline/system/cluster-name

/server/node_modules/node-fetch/lib/index.js:1491
			reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
			       ^
FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:517:28)
    at Socket.socketErrorListener (node:_http_client:501:9)
    at Socket.emit (node:events:517:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND'
}

Node.js v18.18.2

---

As a workaround, adding metadata to dns as an alias for any http server seems to work. It doesn't need to return cloud metadata, just connect.

Impacted by this bug? Give it a 👍.

[luko0610]

@richardburakowski
After some hours wasted and having a look at the code here:
https://github.com/kubeflow/pipelines/blob/master/frontend/server/configs.ts

I found the solution. The ui deployment ml-pipeline-ui needs to have the environment variable DISABLE_GKE_METADATA set to 'true'. This disabled the gke metadata fetch, not available outside of GKE clusters.

[richardburakowski]

Hi @luko0610,
Thanks for the investigation. Appreciate you spending the time on this.
This bug is with the standalone install of kubeflow using the kubeflow manifests, so it's expected to work without gke metadata out of the box. Probably not obvious but the issue is not present in older versions of the install (description updated).

There's no trace of DISABLE_GKE_METADATA in any version of the kustomize output making this look like new behaviour of the container image in the absence of gke metadata rather than a bug in the install manifests.

[joelcomp1]

@richardburakowski After some hours wasted and having a look at the code here: https://github.com/kubeflow/pipelines/blob/master/frontend/server/configs.ts

I found the solution. The ui deployment ml-pipeline-ui needs to have the environment variable DISABLE_GKE_METADATA set to 'true'. This disabled the gke metadata fetch, not available outside of GKE clusters.

thank you this solved my issues!