Create a helm chart/package

[MChorfa]

To ease the adoption and deployment, please create helm chart and publish it on https://artifacthub.io

┆Issue is synchronized with this Asana task by Unito

[cgardens]

@MChorfa thanks for creating this issue. While we publish kubernetes manifests, we do not yet publish helmcharts. We will let you know when we do.

[caleb15]

For other people who come to this issue you can find the kubernetes guide here and the code here

[davinchia]

@MChorfa is the helm package a matter of discovery or an operational preference compared to kustomize/bare deployment yamls?

We are looking at helm right now and thinking through whether we support helm or kustomise or both.

[MChorfa]

@MChorfa is the helm package a matter of discovery or an operational preference compared to kustomize/bare deployment yamls?

We are looking at helm right now and thinking through whether we support helm or kustomise or both.

The most common denominator is helm for most deployments. I would prioritize Helm first and then look at kustomize

[jonstacks]

I've got a very raw helm chart I'm working on if there is interest in the contribution: master...jonstacks:master

Its probably missing a few things and needs some work on the configuration, but has some CI checks too.

We are going to be trying out airbyte in the next few weeks to see if it solves some of our pain points and I'll be updating my fork with some fixes and improvements as we land on a configuration we could potentially ship to our production environment if airbyte is a good fit.

[jrhizor]

@jonstacks that's pretty neat! I especially like the readme.

[foogunlana]

@jonstacks amazing work. Don't forget nodeSelectors on deployments :) and tolerations. Happy to make a PR too.

[jonstacks]

@foogunlana, thank you :). Just got it up and running a few days ago, but should have a PR out mid next week.

[jrhizor]

Steps going forward for this:

• @davinchia review 🎉 Create a Helm Chart For Airbyte #5891
• @jrhizor review 🎉 Create a Helm Chart For Airbyte #5891
• merge 🎉 Create a Helm Chart For Airbyte #5891 by early next week
• add acceptance tests for helm
• add the deploy docs
• add equivalent helm values for existing kustomize env
• deprecate kustomize / with a migration strategy to helm
• move cloud to depend on the OSS chart (@davinchia will create ticket)

[philippeboyd]

May I suggest moving the helm charts (charts/airbyte folder) to it’s own git repository for instance: airbytehq/airbyte-helm?

The main reason being that helm charts are usually evolving independently and maintained by a different group of people than the devs of the product such as a devops team. The helm charts should also have it’s own independent versioning (git tags) and not necessarily following the product’s releases.

I've seen this pattern in a couple of OSS repos such as Argo, Concourse, Hashicorp Vault, and others...

Let me know what you think, cheers!

[MChorfa]

I might add publish it on https://artifacthub.io . artifacthub is the new home for the charts

[fernhtls]

Yes, would be better to version the chart on its own repo as, as they evolve / version independently of the product / tool.
And for sure publish on https://artifacthub.io. For us, as we mainly use helm + terraform it's very hard to keep on syncing the helm chart on changes to our own repo. There's an explanation here: https://artifacthub.io/docs/topics/repositories/. You need to sign with artifacthub and add those metadata files to the repo, as explained there. But first thing I think it's to really detach the helm chart from the product / tool repo really.

[davinchia]

Hey all, thanks for your comments. I will be looking much more into this this week and next.

We are definitely planning to publish on artifacthub. We haven't yet decided to split this out or not, very open to doing so.

My current understanding of the arguments for moving the Airbyte charts into a separate repo is around portability - it would make the lives of teams who only want Airbyte helm charts easier since the git pull would only refresh helm related files. Is that correct? Would publishing the charts to artifact hub solve this? Or do folks think there is still benefit from splitting the charts out?

[jonstacks]

When I originally set this up, I was roughly following what Apache Airflow did for their helm chart. They keep it in the repo, but then create specific tags(ex: helm-chart/1.3.0) for releases. I've seen keeping it both in the product repo and outside of the product repo work and don't have a preference either way.

I think just getting the helm chart repository set up will solve a lot of problems for people since they can interact with helm tooling and don't have to sync git repos at all and change directories. For example, one could do:

helm repo add airbyte https://charts.airbyte.io
helm install airbyte airbyte/airbyte --version <specific-version-to-install>

And be off to the races for a test environment(Most would need to specify an override for values.yaml for a production deploy IMO). When it comes time to upgrade, you simply run

helm repo update airbyte
helm upgrade ... --version <some-new-version>

to fetch the latest index.yaml from the helm repo to see which new versions are available and install a different version.

I'm not heavily versed in ArtifactHub, but it looks like its an aggregator for helm repos and doesn't actually host them. It looks like you can point ArtifactHub at a hypothetical charts.airbyte.io and it will index it and provide pretty documentation and version history for all your releases based off the index.yaml and the contents of each tar.gz helm chart version.

IMO, the things that would make a better user experience are(in order of most impactful to least impactful):

1. Functional Publish the chart somewhere(Github Pages, GCS Website, S3 +- CloudFront) and point a DNS record like charts.airbyte.io at it. This enables not dealing with git repos for helm operations. It doesn't matter where the chart lives on GitHub and you can change it later as long as you keep publishing to the same web server or you can even move DNS to point to a different IP if you want to change hosting methods.
2. End User Experience Have ArtifactHub start indexing your helm repo to make it searchable and the documentation easily browsable in a medium meant for helm chart documentation and getting started quickly. It looks like there are extra annotations that can be added to the Chart.yaml to provide even more info on the ArtifactHub site.
3. Developer Experience/Preference - Discuss and possibly move the helm chart to its own repo. I don't think it blocks 1 or 2.

[gajus]

To anyone reading the above comment without any context: charts.airbyte.io does not actually exists. Comment suggests adding it.

[gajus]

Instructions for anyone who want to install this Helm chart:

git clone [email protected]:airbytehq/airbyte.git
cd airbyte
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dep build charts/airbyte
helm upgrade --install \
  --create-namespace \
  --namespace "airbyte" \
  --debug \
  --wait \
  --timeout "600s" \
  contra-airbyte ./charts/airbyte

[gajus]

Did anyone figure out how to add airbyte Helm chart to ArgoCD?

[prein]

Setting up automated publishing to Github Pages should be reasonably low effort. Would a PR for that be welcome?

[jonstacks]

Did anyone figure out how to add airbyte Helm chart to ArgoCD?

You should be able to add the current github repo and specify a path to the helm-chart. This Argo section should work.

[prein]

@jonstacks this approach would not work when using an umbrella chart where Airbyte chart is a dependency. What might work is git sub-module approach with dependency referenced as file

cat .gitmodules
[submodule "dependencies/airbyte"]
  path = dependencies/airbyte
  url = https://github.com/airbytehq/airbyte.git

cat my_local_airbyte/Chart.yaml
dependencies:
  name: airbyte
  version: 0.3.0
    repository: file://dependencies/airbyte/charts/airbyte

[thomas-vl]

Is there any update on Airbyte hosting the helm chart somewhere?

[mohitreddy1996]

Hi any updates or ETA regarding Airbyte hosting the helm chart? Thanks!

[Riezebos]

It looks like helm-charts hosted by airbyte are available here: https://docs.airbyte.com/helm-charts/
Installation using the commands shown there also works for me.

[davinchia]

Hi all, yes we have slowly been making progress cleaning up the charts. We have held off announcing it for now as we are still making Helm part of our regular builds. Hoping to get this out in the next week or so!

[olivermeyer]

Hi all, yes we have slowly been making progress cleaning up the charts. We have held off announcing it for now as we are still making Helm part of our regular builds. Hoping to get this out in the next week or so!

That's good news. Do you have an update on when we can expect stable charts? I've been trying to upgrade our charts and the latest few versions are very unstable. If a stable chart is coming soon I'll hold off on the upgrade.

[davinchia]

@olivermeyer

We are running into some issues making the helm charts part of the PR build tests. Once we do that we can 'call' this stable. I would expect this to be done in the next week or so.

@xpuska513 is working on this.

CC @supertopher @git-phu

[simon-oneacrefund]

@davinchia does this mean that the charts at https://docs.airbyte.com/helm-charts/ should not be used yet?

We are trying them,helm install works but helm upgrade fails during recreation of PVCs. Should I create an issue or wait until the charts are more stable?

[marcosmarxm]

@xpuska513 can you check this?

@davinchia does this mean that the charts at https://docs.airbyte.com/helm-charts/ should not be used yet?

We are trying them,helm install works but helm upgrade fails during recreation of PVCs. Should I create an issue or wait until the charts are more stable?

[davinchia]

@simon-oneacrefund you can use them, however they are in beta while we iron out the testing. You should wait till the charts are stable.

@xpuska513 can we confirm the charts support upgrading? Last I checked I believe they do.

[xpuska513]

Hmm, probably right now upgrade is broken after we switched from using bitnami charts to using just a single deployment. It’s a known issue with PVC management by Helm.
I’ll be out for the most time next week, but I took laptop with me, so I’ll try try to fix it. If not I’ll rollback changes made to db and minio back to using bitnami.
cc: @davinchia , @simon-oneacrefund

[vincentkoc]

The latest version is 0.40.12 in artifacthub not 0.41.2, the are a number of PRs to clean up the helm charts. There were a number of broken chart releases.

[xpuska513]

About versions, I already made PR that addresses this issue, but it's still waiting for review(it's 2nd week since it was created)

#16642

The main issue with versioning is with the fact that the current workflow doesn't stick to the airbyte versioning, which causes action that is being used right now to break versioning for helm charts.

My PR addresses this by simply integrating release process of Helm charts into airbyte release process.

[xpuska513]

Hi, happy to inform you that the release process got updated, so there should be no more issues with publishing and version disappearing from time to time.

[stang]

hello 👋

Can someone help me clarify if whether or not it is recommended to use https://artifacthub.io/packages/helm/airbyte/airbyte ? 🙏

[davinchia]

Not yet. We are still ironing out some issues with the charts in certain situations. They are generally stable for now, however might not work in all cases.

[olivermeyer]

Hi @davinchia, can you confirm whether the charts are ready for use?

[gajus]

We are using them in production, for what it is worth.

[vincentkoc]

Likewise, i made the switch and seems stable

[davinchia]

These should be more or less stable now. It's open since we are doing some minor clean up, however we should no longer run into issues with them.

[cgardens]

@supertopher can we close this issue? i think the work prodeng did to publish helm charts completes it.

[supertopher]

yes we can. there are functioning helm charts, cloud is built on top of OSS helm charts which should help to keep the OSS charts up to date by necessity