Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm repo add bitnami gives not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR #8433

Closed
chukka opened this issue Dec 16, 2021 · 79 comments
Assignees
Labels
solved triage Triage is needed

Comments

@chukka
Copy link

chukka commented Dec 16, 2021

Which chart: Not chart but bitnami helm repo - helm repo add bitnami https://charts.bitnami.com/bitnami
The name (and version) of the affected chart

Describe the bug
As part of CI tests, we add the bitnami repo and do some chart-testing. This breaks CI flow. is there any option to retry or is it a known issue

To Reproduce
Steps to reproduce the behavior:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

Expected behavior
A clear and concise description of what you expected to happen.

Version of Helm and Kubernetes: 3.5.4

Additional context
This doesn't happen always but fails 2 out of 10 times atleast

@carrodher
Copy link
Member

I was not able to reproduce the issue, I tried several times and it worked fine for all of them. Maybe it's related to the AWS outage that happened yesterday, see some news https://www.theguardian.com/technology/2021/dec/15/amazon-down-web-services-outage-netflix-slack-ring-doordash-latest

Are you still facing the issue?

@chukka
Copy link
Author

chukka commented Dec 16, 2021

@carrodher This issue happens intermittently , This is not the first time it happened for us yesterday .

@carrodher
Copy link
Member

Hi @chukka, could you tell us since when this problem occurred in your case? My 2cts is that it can be related to any connectivity issue produced by a third-party outage like AWS, DNS provider, etc.
We have several automations to release the charts that are constantly adding the repository and I didn't see any issue, or if there were an issue it was transient and solved with the automatic retries.

@GCHQDeveloper926
Copy link
Contributor

GCHQDeveloper926 commented Dec 17, 2021

I've had this happen 2-3 times on 15 December and 3 times this morning 17 December:

Downloading common from repo https://charts.bitnami.com/bitnami
Save error occurred:  could not find : chart rabbitmq not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR
Deleting newly downloaded charts, restoring pre-update state
Error: could not find : chart rabbitmq not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

@javsalgar
Copy link
Contributor

Hi,

In my case I wasn't able to reproduce the issue either.

@andybjourney
Copy link

andybjourney commented Dec 21, 2021

I'm seeing this as well, and it is intermittent. I got this from running terraform locally against a k3d cluster via the helm provider:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

@javsalgar
Copy link
Contributor

I forwarded this to the engineering team so they can take a look.

@tebaly
Copy link

tebaly commented Dec 26, 2021

Doesn't work now. Already about 30 minutes

ArgoCD

rpc error: code = Unknown desc = Manifest generation error (cached): helm repo add https://charts.bitnami.com/bitnami https://charts.bitnami.com/bitnami failed exit status 1: Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

@carrodher
Copy link
Member

Thanks for letting us know, we are looking into it

@carrodher
Copy link
Member

Thanks to everyone for reporting this issue. We are facing some transient issues with the Helm registry, it should work if you retry it:

$ helm repo add bitnami2 https://charts.bitnami.com/bitnami
"bitnami2" has been added to your repositories

About users trying to access using the browser, please note this URL is designed for work with the Helm CLI, the only path accessible via browser is https://charts.bitnami.com/bitnami/index.yaml where you can download the index.yaml

We will update the thread with more information once properly analyzed the issue on our side. Thanks!

@carrodher
Copy link
Member

We have been doing some tests, for example the following script was running during a couple of days:

#!/bin/bash

set -e

export HELM_DEBUG=1

error() {
    echo "See file /tmp/chart.$$.log for debugging info"
}

command -v helm || exit 1

mkdir -p /tmp/charts/

# Do not fail here
helm repo remove bitnami || true

(helm repo add bitnami https://charts.bitnami.com/bitnami 2> /tmp/charts/$$.log) || error

From the original 20063 probes, about 9.1% failed with this message:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerError: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerhelm.go:88: [debug] stream error: stream ID 1; INTERNAL_ERROR; received from peerlooks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reachedmain.(*repoAddOptions).run helm.sh/helm/v3/cmd/helm/repo_add.go:206main.newRepoAddCmd.func1 helm.sh/helm/v3/cmd/helm/repo_add.go:82github.com/spf13/cobra.(*Command).execute github.com/spf13/[email protected]/command.go:856github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/[email protected]/command.go:974github.com/spf13/cobra.(*Command).Execute github.com/spf13/[email protected]/command.go:902main.main helm.sh/helm/v3/cmd/helm/helm.go:87runtime.main runtime/proc.go:255runtime.goexit runtime/asm_amd64.s:1581

This error seems to be caused by a network issue, it can be potentially caused on both sides: client or server: is not a valid chart repository or cannot be reached

Line https://github.com/helm/helm/blob/39ca699ca790e02ba36753dec6ba4177cc68d417/cmd/helm/repo_add.go#L205 points to be a client error, but it's not a determinant clue.

We have performed some tests and we can fetch the file https://charts.bitnami.com/bitnami/index.yaml 100% of the cases, but sometimes helm fails after curl succeeding:

# Curl 
*   Trying 99.84.160.70...*   Trying 99.84.160.70...* TCP_NODELAY set* Connected to charts.bitnami.com (99.84.160.70) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:*   CAfile: /etc/ssl/cert.pem  CApath: none* TLSv1.2 (OUT), TLS handshake, Client hello (1):} [232 bytes data]* TLSv1.2 (IN), TLS handshake, Server hello (2):{ [100 bytes data]* TLSv1.2 (IN), TLS handshake, Certificate (11):{ [4820 bytes data]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):{ [300 bytes data]* TLSv1.2 (IN), TLS handshake, Server finished (14):{ [4 bytes data]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):} [37 bytes data]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):} [1 bytes data]* TLSv1.2 (OUT), TLS handshake, Finished (20):} [16 bytes data]* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):{ [1 bytes data]* TLSv1.2 (IN), TLS handshake, Finished (20):{ [16 bytes data]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256* ALPN, server accepted to use h2* Server certificate:*  subject: CN=charts.bitnami.com*  start date: Feb 14 00:00:00 2021 GMT*  expire date: Mar 15 23:59:59 2022 GMT*  subjectAltName: host "charts.bitnami.com" matched cert's "charts.bitnami.com"*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon*  SSL certificate verify ok.* Using HTTP2, server supports multi-use* Connection state changed (HTTP/2 confirmed)* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0* Using Stream ID: 1 (easy handle 0x7ffbbb00d600)> GET /bitnami/index.yaml HTTP/2> Host: charts.bitnami.com> User-Agent: curl/7.64.1> Accept: */*> * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!< HTTP/2 200 < content-type: binary/octet-stream< content-length: 11287899< last-modified: Tue, 11 Jan 2022 14:36:14 GMT< x-amz-server-side-encryption: AES256< x-amz-version-id: nmDdVr8LQPExrl5WNUdeu5redpWHeUxw< accept-ranges: bytes< server: AmazonS3< date: Tue, 11 Jan 2022 14:52:52 GMT< etag: "04935efc90d38c851fb4009f3b700563"< x-cache: RefreshHit from cloudfront< via: 1.1 7fa75077148a5e108de410272ea8ccc4.cloudfront.net (CloudFront)< x-amz-cf-pop: ORD52-C2< x-amz-cf-id: oWfuxcKqMqT0AP-kpwCfGBX-IA42jtR6mjFoXGazr0-p2GmBIoFwfQ==< { [15983 bytes data]* Connection #0 to host charts.bitnami.com left intact* Closing connection 0


# Helm
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerhelm.go:88: [debug] stream error: stream ID 1; INTERNAL_ERROR; received from peerlooks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reachedmain.(*repoAddOptions).run helm.sh/helm/v3/cmd/helm/repo_add.go:206main.newRepoAddCmd.func1 helm.sh/helm/v3/cmd/helm/repo_add.go:82github.com/spf13/cobra.(*Command).execute github.com/spf13/[email protected]/command.go:856github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/[email protected]/command.go:974github.com/spf13/cobra.(*Command).Execute github.com/spf13/[email protected]/command.go:902main.main helm.sh/helm/v3/cmd/helm/helm.go:87runtime.main runtime/proc.go:255runtime.goexit runtime/asm_amd64.s:1581

@JoepWeijers
Copy link

I'm also seeing this issue pop up in a CI build that calls Helm about once per day.

@viniciusfacco
Copy link

I'm having the same issue. This is the message I get:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://charts.bitnami.com/bitnami/index.yaml": read tcp 10.163.207.67:61516->108.138.167.54:443: wsarecv: An existing connection was forcibly closed by the remote host.

@Segamegadrive
Copy link

Segamegadrive commented Feb 24, 2022

Hi, I'm also facing a similar type of error. Jenkins build fails and I'm using Terraform HEML Provider to deploy Prometheus HEML chat from bitnami. Tried to get around with it by creating a docker image as well; defining commands to install HELM 3 and adding CMD to add repos, and still didn't help. I also tried replacing the TF HELM repository attribute with "/.cache/helm/repository/" but throws a Protocol Handler error.

Below is the error I have been getting:

module.prometheus.helm_release.grafana: Creating...
module.prometheus.helm_release.prometheus: Creating...
module.prometheus.helm_release.grafana: Still creating... [10s elapsed]
module.prometheus.helm_release.prometheus: Still creating... [10s elapsed]

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: open /.cache/helm/repository/HpOj5uooN5ifybCQee2mfuZUHA4=-index.yaml: no such file or directory

  with module.prometheus.helm_release.prometheus,
  on .terraform/modules/prometheus/main.tf line 16, in resource "helm_release" "prometheus":
  16: resource "helm_release" "prometheus" {


Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: open /.cache/helm/repository/ON43pZ61a8pJuVDiqDkXwhDeE0I=-index.yaml: no such file or directory

  with module.prometheus.helm_release.grafana,
  on .terraform/modules/prometheus/main.tf line 25, in resource "helm_release" "grafana":
  25: resource "helm_release" "grafana" {

My Terraform HEML module below, tried with chart name without bitnami/ path too and still throws same error as above:

resource "helm_release" "prometheus" {
  name             = "prometheus"
  repository       = "https://charts.bitnami.com/bitnami"
  chart            = "bitnami/prometheus"
  namespace        = "monitoring"
  create_namespace = true
}

// Grafana HEML chart for Visualisation of telemetry data
resource "helm_release" "grafana" {
  name             = "grafana"
  repository       = "https://charts.bitnami.com/bitnami"
  chart            = "bitnami/grafana"
  namespace        = "monitoring"
  create_namespace = true
}

Any info and advice would be grateful, please. Thank you.

@carrodher
Copy link
Member

Hi, monitoring didn't show bad connections or alike, on our side, are you still facing this intermittent issue?

@snorlaX-sleeps
Copy link

We are also using Terraform to deploy the external DNS Helm chart and have experienced it several times today.
It did not however occur with the metrics-server Helm Chart.

@JoepWeijers
Copy link

Yes, we are still seeing this issue

@DanielJRusev
Copy link

https://charts.bitnami.com/bitnami

AccessDenied Access Denied SN5Q4NA1V71A1ZKZ /KusJDXgVbJl8HSUg3Q/U45KyzXNVDntRtCPoB34yPbvV4MWBWIXtBe8mQyMNtOHvRVcDfRkQHQ=

@chukka
Copy link
Author

chukka commented Mar 26, 2022

FYI- To workaround this issue, we moved all used (by our team) bitnami charts under https://charts.jfrog.io (powered by artifactory)

if you are using artifactory , you can try Remote Repositories to cache all bitnami charts

@alemorcuq
Copy link
Contributor

I can't reproduce the issue on my side:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

@alemorcuq
Copy link
Contributor

alemorcuq commented Mar 28, 2022

I reproduced the issue now, but it worked immediately after:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "bitnami" chart repository (https://charts.bitnami.com/bitnami):
	stream error: stream ID 1; INTERNAL_ERROR; received from peer
Update Complete. ⎈Happy Helming!⎈

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Our team is looking into this.

@ArgonV
Copy link

ArgonV commented Mar 31, 2022

Howdy all, I would just like to report that I am seeing this issue to, in southeast Texas.

Clone failed: Cloning into 'management-state/catalog-cache/a68a354dfd35755d14bebd02d825feb24ecbbd0496438ceb36691e1eabaa4947'... fatal: Remote branch master not found in upstream origin : exit status 128

Adding via command line still works - but in the above, this is through the Rancher Global Catalog UI.

@carrodher
Copy link
Member

Please note this is a totally different and unrelated issue. Helm client doesn't work with git not cloning any Git repository. It seems something totally different and unrelated to Bitnami.

Regarding the original issue related to the Helm registry, we are doing more research on our side in order to find the root cause of those transient errors and mitigate them.

@ArgonV
Copy link

ArgonV commented Apr 1, 2022

Ah, well the original message I had was: Update failed: fatal: not a git repository (or any parent up to mount point /var/lib) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). : exit status 128

When I totally removed the repo and added it back, the error I got was what I posted here in my previous comment. Nothing in my environments has changed (I see the same issue in Test/Dev/Pre/Prod K8s cluster envs) - that just popped up this week. The setup had been running fine for a couple of years now.

@renier
Copy link

renier commented Aug 24, 2022

$ helm repo add bitnami https://charts.bitnami.com/bitnami
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://charts.bitnami.com/bitnami/index.yaml": EOF

that was just now. it doesn't seem like this issue is resolved.

@hmphu
Copy link

hmphu commented Nov 2, 2022

$ helm repo add bitnami https://charts.bitnami.com/bitnami
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://charts.bitnami.com/bitnami/index.yaml": dial tcp: lookup charts.bitnami.com on 8.8.4.4:53: no such host

This issue is still happening

@BenjaminBrandtner
Copy link

It's fixed: #13313

@tal-ayalon
Copy link

Still getting certificate related issues:
image

@basaveswar-kureti
Copy link

basaveswar-kureti commented Dec 20, 2022

@tal-ayalon You are missing bitnami at the end of the url. It should be:

helm repo add bitnami https://charts.bitnami.com/bitnami

@tal-ayalon
Copy link

@basaveswar-kureti Still getting the error:
image
If I'm using the --insecure-skip-tls-verify flag than it works

@pdf
Copy link

pdf commented Dec 21, 2022

@tal-ayalon this error appears to be macos-specific and not directly related to bitnami:

golang/go#51991

@jfklingler
Copy link

jfklingler commented Dec 21, 2022

Works on my machine. 😛 That's not super helpful though.

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
$ helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.17"}
$ sw_vers
ProductName:		macOS
ProductVersion:		13.1
BuildVersion:		22C65

@jdomag
Copy link

jdomag commented Jan 11, 2023

I'm seeing this error at least 5 time a day in argocd-repo-server:

level=error msg="`helm dependency build` failed exit status 1: Error: error loading /helm-working-dir/repository/https:--charts.bitnami.com-bitnami-index.yaml: empty index.yaml file\nError: plugin \"secrets\

@vijaykalapatapu
Copy link

vijaykalapatapu commented Sep 14, 2023

helm repo add bitnami https://charts.bitnami.com/bitnami
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://charts.bitnami.com/bitnami/index.yaml"

@github-actions github-actions bot added the triage Triage is needed label Sep 14, 2023
@javsalgar
Copy link
Contributor

Hi,

We changed the bitnami/charts helm repository to use OCI. Check the new instructions here https://github.com/bitnami/charts/

@dayo777
Copy link

dayo777 commented Oct 14, 2023

Hello friends, I have a little issue I am facing using Helm charts;
I am using the helm dependency update [folder_name] command to update a chart, the update was successful and all but helm is experiencing issues saving the chart.

edit: I used this same chart repository few minutes ago and it worked.

Getting updates for unmanaged Helm repositories...
...Unable to get an update from the "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" chart repository:
	context deadline exceeded (Client.Timeout or context cancellation while reading body)
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Save error occurred:  could not find : chart mariadb not found in https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami: looks like "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" is not a valid chart repository or cannot be reached: context deadline exceeded (Client.Timeout or context cancellation while reading body)
Error: could not find : chart mariadb not found in https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami: looks like "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" is not a valid chart repository or cannot be reached: context deadline exceeded (Client.Timeout or context cancellation while reading body)

@dayo777
Copy link

dayo777 commented Oct 14, 2023

I retried the command, after deleting the Charts folder, and it ran successfully.

Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading mariadb from repo https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
Deleting outdated charts

Hello friends, I have a little issue I am facing using Helm charts; I am using the helm dependency update [folder_name] command to update a chart, the update was successful and all but helm is experiencing issues saving the chart.

edit: I used this same chart repository few minutes ago and it worked.

Getting updates for unmanaged Helm repositories... ...Unable to get an update from the "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" chart repository: context deadline exceeded (Client.Timeout or context cancellation while reading body) Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "bitnami" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 1 charts Save error occurred: could not find : chart mariadb not found in https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami: looks like "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" is not a valid chart repository or cannot be reached: context deadline exceeded (Client.Timeout or context cancellation while reading body) Error: could not find : chart mariadb not found in https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami: looks like "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami" is not a valid chart repository or cannot be reached: context deadline exceeded (Client.Timeout or context cancellation while reading body)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
solved triage Triage is needed
Projects
None yet
Development

No branches or pull requests