Helm chart are published before images are ready

[makkes]

Describe the bug

With each Weave GitOps release the chart is updated prematurely so that upgrades fail as long as the images haven't been built, yet.

Environment

• Weave-Gitops Version: all
• Flux Version: n/a
• Kubernetes version: n/a

To Reproduce
Steps to reproduce the behavior:

Immediately after a Weave GitOps release is published, try to install that release.

Expected behavior

The release installs successfully.

Actual Behavior

Installation fails:

weave-gitops-648b7f9655-ll4zn   0/1     ImagePullBackOff   0          11m
weave-gitops-648b7f9655-ll4zn   0/1     ErrImagePull       0          11m

[tsunamaru]

Just faced the same issue when flux applied weave-gitops helm chart.
Looks like image build job is stuck: https://github.com/weaveworks/weave-gitops/actions/runs/5738127454/job/15551440034

[makkes]

@tsunamaru thanks for verifying. The build job usually takes ~1.5h so that's expected.

[adelmoradian]

seems to be working now again!

[enekofb]

Thanks all for reporting and following up! We will look into it to avoid it happening in the future.

[stefanprodan]

The build job usually takes ~1.5h so that's expected.

This can take just 20m if you use tonistiigi/xx, example https://github.com/fluxcd/kustomize-controller/blob/main/Dockerfile

[stefanprodan]

I would change the title to

GitHub release and Helm chart are published before images are ready

The release workflow should be:

• build and push container image
• build and push Hem chart
• publish release to GitHub

[enekofb]

Previous spike effort #3157

[enekofb]

Release PR examples

• v0.28
• v0.29

[enekofb]

Done so far:

• We have implemented guardrails to avoid this issue from happening (as in v0.29 release with chart published before image with users impact). Attached PRs for full context on the solution.
• Today's release v0.30 should help us to validate it.

Validation of the solution after merging:

• works as expected for push on main
• works as expected for non-release pr
• works as expected for release pr 8d13b12

Complete 🆙 validation would close the ticket

The other suggestions on build time and github release order will be tackled in a follow-up issue #3945

[makkes]

Strictly speaking this hasn't been completely resolved as the GitHub release is still created and announced (e.g. on Slack) prematurely, i.e. before the new chart version is actually available.

[enekofb]

Strictly speaking this hasn't been completely resolved as the GitHub release is still created and announced (e.g. on Slack) prematurely, i.e. before the new chart version is actually available.

You are right, i have removed that expectation from the issue title and reflected in the follow up ticket #3945