Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.29.0 never becomes ready #3419

Closed
dgn opened this issue Oct 25, 2024 · 15 comments · Fixed by #3426
Closed

0.29.0 never becomes ready #3419

dgn opened this issue Oct 25, 2024 · 15 comments · Fixed by #3426
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@dgn
Copy link

dgn commented Oct 25, 2024

Bug Report

What did you do?
We regularly install the olm-operator on KinD using operator-sdk olm install. This has started failing since 0.29.0 has come out. 0.28.0 installs just fine. We're using the latest version of operator-sdk (v1.37.0)

Here's a log:

INFO[0010] Creating OLM resources...                    
INFO[0010]   Creating Namespace "olm"                   
INFO[0012]   Namespace "olm" created                    
INFO[0012]   Creating Namespace "operators"             
INFO[0013]   Namespace "operators" created              
[...snip...]
INFO[0024]   Creating CatalogSource "olm/operatorhubio-catalog" 
INFO[0025]   CatalogSource "olm/operatorhubio-catalog" created 
INFO[0025] Waiting for deployment/olm-operator rollout to complete 
INFO[0026]   Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available 
FATA[0120] Failed to install OLM version "latest": deployment/olm-operator failed to rollout: context deadline exceeded 

The containers fail to start with CreateContainerConfigError. Here's the relevant event log:

  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Warning  Failed     5s (x11 over 2m1s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "olm-operator-7b6ffc567-nnlm7_olm(5b55cb5e-8d14-4b71-abd7-8fd8644c28be)", container: olm-operator)
  Normal   Pulled     5s (x10 over 2m)    kubelet            Container image "quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37" already present on machine

What did you expect to see?
A working installation of olm-operator using operator-sdk olm install

What did you see instead? Under which circumstances?
Deployment never becomes ready.

Environment

  • operator-lifecycle-manager version:

0.29.0

  • Kubernetes version information:

v1.29.2

  • Kubernetes cluster kind:

KinD cluster

Possible Solution
Looks like container has runAsNonRoot and image will run as root from the event log is the issue, but there might be more.

Additional context
Add any other context about the problem here.

@jmazzitelli
Copy link
Contributor

FWIW I see the same problem on minikube:

$ minikube version
minikube version: v1.34.0
commit: 210b148df93a80eb872ecbeb7e35281b3c582c61

jmazzitelli added a commit to jmazzitelli/kiali that referenced this issue Oct 29, 2024
There is a bug in OLM v0.29.0 that causes OLM not to start. Use v0.28.0 for now until OLM is fixed.

See: operator-framework/operator-lifecycle-manager#3419
@dgn dgn changed the title 0.29.0 never becomes ready on kind 0.29.0 never becomes ready Oct 29, 2024
@jmazzitelli
Copy link
Contributor

We regularly install the olm-operator on KinD using operator-sdk olm install

I also want to say that we (Kiali team) start it a different way, but get the same error. Here's how we deploy OLM: https://github.com/kiali/kiali/blob/v2.0.0/make/Makefile.olm.mk#L249

REPLICATION PROCEDURES TO SEE THE ERROR:

Basically, download the install.sh and pipe it to bash.

You can replicate the problem easily on minikube:

  1. Start minikube
  2. Download OLM v0.29.0 install script and pipe to bash:
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.29.0/install.sh | bash -s v0.29.0
  1. Notice script hangs and eventually times out waiting:
...
Waiting for deployment "olm-operator" rollout to finish: 0 of 1 updated replicas are available...
  1. Look at the OLM pods fail to start
$ kubectl get pods -n olm
NAME                                READY   STATUS                       RESTARTS   AGE
catalog-operator-568c6d5989-75psv   0/1     CreateContainerConfigError   0          37s
olm-operator-7c4756f796-mdn8p       0/1     CreateContainerConfigError   0          37s
$ kubectl describe pods -n olm -l app=olm-operator
  Normal   Scheduled  99s                default-scheduler  Successfully assigned olm/olm-operator-7c4756f796-mdn8p to minikube
  Normal   Pulling    98s                kubelet            Pulling image "quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37"
  Normal   Pulled     78s                kubelet            Successfully pulled image "quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37" in 19.958s (19.958s including waiting). Image size: 235113812 bytes.
  Warning  Failed     10s (x7 over 78s)  kubelet            Error: container has runAsNonRoot and image will run as root (pod: "olm-operator-7c4756f796-mdn8p_olm(fd01319b-d680-48e8-8d03-5a66385ae1d9)", container: olm-operator)
  Normal   Pulled     10s (x6 over 78s)  kubelet            Container image "quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37" already present on machine

jmazzitelli added a commit to kiali/kiali that referenced this issue Oct 29, 2024
There is a bug in OLM v0.29.0 that causes OLM not to start. Use v0.28.0 for now until OLM is fixed.

See: operator-framework/operator-lifecycle-manager#3419
@tyxeron
Copy link

tyxeron commented Oct 30, 2024

We face the same on k3d:

$ k3d version                   
k3d version v5.7.4
k3s version v1.30.4-k3s1 (default)

We used the same procedure to install OLM as @jmazzitelli and faced exactly the same issue.

@acornett21
Copy link

The issue appears to be that the sha in the olm.yaml file points to to a sha for v0.19.0 instead of v0.29.0

containerStatuses:
  - image: quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37
    imageID: ""
    lastState: {}
    name: olm-operator
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'container has runAsNonRoot and image will run as root (pod: "olm-operator-7b6ffc567-4svgn_olm(96ffa7d3-4fcb-4ea0-bd19-bae6baf40ea4)",
          container: olm-operator)'
        reason: CreateContainerConfigError
crane digest quay.io/operator-framework/olm:v0.19.0
sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37

@m1kola
Copy link
Member

m1kola commented Oct 30, 2024

Here is the job which generated the release. There is a step called "Generate quickstart release manifests" and I see some mentions of v0.19.0 release here for some reason:

Generate quickstart release manifests logs
  make release ver=v0.29.0 IMAGE_REPO=quay.io/operator-framework/olm
  shell: /usr/bin/bash -e {0}
  env:
    IMAGE_TAG: v0.29.0
docker pull quay.io/operator-framework/opm:v1.47.0
v1.47.0: Pulling from operator-framework/opm
2c77aede9220: Pulling fs layer
2bdf44d7aa71: Pulling fs layer
452e9eed7ecf: Pulling fs layer
0f8b424aa0b9: Pulling fs layer
d557676654e5: Pulling fs layer
c8022d07192e: Pulling fs layer
d858cbc252ad: Pulling fs layer
1069fc2daed1: Pulling fs layer
b40161cd83fc: Pulling fs layer
3f4e2c586348: Pulling fs layer
80a8c047508a: Pulling fs layer
5391af8231d5: Pulling fs layer
357b100e24fd: Pulling fs layer
d858cbc252ad: Waiting
1069fc2daed1: Waiting
b40161cd83fc: Waiting
0f8b424aa0b9: Waiting
d557676654e5: Waiting
c8022d07192e: Waiting
3f4e2c586348: Waiting
80a8c047508a: Waiting
5391af8231d5: Waiting
357b100e24fd: Waiting
035d5108d0c5: Pulling fs layer
5f1c40846974: Pulling fs layer
035d5108d0c5: Waiting
5f1c40846974: Waiting
2c77aede9220: Verifying Checksum
2c77aede9220: Download complete
2c77aede9220: Pull complete
2bdf44d7aa71: Download complete
452e9eed7ecf: Verifying Checksum
452e9eed7ecf: Download complete
2bdf44d7aa71: Pull complete
0f8b424aa0b9: Verifying Checksum
0f8b424aa0b9: Download complete
c8022d07192e: Download complete
d858cbc252ad: Verifying Checksum
d858cbc252ad: Download complete
1069fc2daed1: Verifying Checksum
1069fc2daed1: Download complete
452e9eed7ecf: Pull complete
0f8b424aa0b9: Pull complete
b40161cd83fc: Verifying Checksum
b40161cd83fc: Download complete
3f4e2c586348: Verifying Checksum
3f4e2c586348: Download complete
5391af8231d5: Verifying Checksum
5391af8231d5: Download complete
80a8c047508a: Verifying Checksum
80a8c047508a: Download complete
357b100e24fd: Verifying Checksum
357b100e24fd: Download complete
035d5108d0c5: Verifying Checksum
035d5108d0c5: Download complete
d557676654e5: Verifying Checksum
d557676654e5: Download complete
d557676654e5: Pull complete
c8022d07192e: Pull complete
d858cbc252ad: Pull complete
1069fc2daed1: Pull complete
b40161cd83fc: Pull complete
3f4e2c586348: Pull complete
80a8c047508a: Pull complete
5f1c40846974: Verifying Checksum
5f1c40846974: Download complete
5391af8231d5: Pull complete
357b100e24fd: Pull complete
035d5108d0c5: Pull complete
5f1c40846974: Pull complete
Digest: sha256:077e654da39d055ea56edc91faa2c36d5e8f4dec58db2017fed24a80246b6ca7
Status: Downloaded newer image for quay.io/operator-framework/opm:v1.47.0
quay.io/operator-framework/opm:v1.47.0
./scripts/copy_crds.sh
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_catalogsources.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_clusterserviceversions.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_installplans.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_olmconfigs.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_operatorconditions.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_operatorgroups.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_operators.yaml
copying ./scripts/../vendor/github.com/operator-framework/api/crds/operators.coreos.com_subscriptions.yaml
Generating the v0.19.0  release
docker pull quay.io/operator-framework/olm:v0.19.0 
v0.19.0: Pulling from operator-framework/olm
b49b96595fd4: Pulling fs layer
df949455c4e5: Pulling fs layer
41e48f6b1bd9: Pulling fs layer
42c0b83a7e87: Pulling fs layer
efd54022ebb6: Pulling fs layer
71621b4144cb: Pulling fs layer
3397cf438d9a: Pulling fs layer
efd54022ebb6: Waiting
71621b4144cb: Waiting
42c0b83a7e87: Waiting
3397cf438d9a: Waiting
b49b96595fd4: Verifying Checksum
b49b96595fd4: Download complete
41e48f6b1bd9: Verifying Checksum
41e48f6b1bd9: Download complete
b49b96595fd4: Pull complete
efd54022ebb6: Verifying Checksum
efd54022ebb6: Download complete
df949455c4e5: Verifying Checksum
df949455c4e5: Download complete
df949455c4e5: Pull complete
41e48f6b1bd9: Pull complete
3397cf438d9a: Verifying Checksum
3397cf438d9a: Download complete
71621b4144cb: Verifying Checksum
71621b4144cb: Download complete
42c0b83a7e87: Verifying Checksum
42c0b83a7e87: Download complete
42c0b83a7e87: Pull complete
efd54022ebb6: Pull complete
71621b4144cb: Pull complete
3397cf438d9a: Pull complete
Digest: sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37
Status: Downloaded newer image for quay.io/operator-framework/olm:v0.19.0
quay.io/operator-framework/olm:v0.19.0
make TARGET=upstream RELEASE_VERSION=v0.19.0  PACKAGE_QUICKSTART=true package
make[1]: Entering directory '/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager'
(re)installing /home/runner/go/bin/yq-v3.0.0-20201202084205-8846255d1c37
go: downloading github.com/mikefarah/yq/v3 v3.0.0-20201202084205-8846255d1c37
go: downloading github.com/spf13/cobra v1.0.0
go: downloading gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
go: downloading gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
go: downloading github.com/fatih/color v1.9.0
go: downloading github.com/goccy/go-yaml v1.8.1
go: downloading github.com/mattn/go-colorable v0.1.7
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae
(re)installing /home/runner/go/bin/helm-v3.15.4
go: downloading helm.sh/helm/v3 v3.15.4
go: downloading github.com/Masterminds/semver/v3 v3.2.1
go: downloading github.com/gofrs/flock v0.8.1
go: downloading github.com/gosuri/uitable v0.0.4
go: downloading github.com/moby/term v0.5.0
go: downloading github.com/spf13/cobra v1.8.0
go: downloading golang.org/x/term v0.22.0
go: downloading golang.org/x/text v0.16.0
go: downloading k8s.io/apimachinery v0.30.3
go: downloading k8s.io/client-go v0.30.3
go: downloading k8s.io/klog/v2 v2.120.1
go: downloading k8s.io/kubectl v0.30.3
go: downloading github.com/fatih/color v1.13.0
go: downloading golang.org/x/sys v0.22.0
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.3
go: downloading github.com/Masterminds/sprig/v3 v3.2.3
go: downloading k8s.io/api v0.30.3
go: downloading k8s.io/cli-runtime v0.30.3
go: downloading github.com/cyphar/filepath-securejoin v0.2.4
go: downloading github.com/mitchellh/copystructure v1.2.0
go: downloading github.com/xeipuuv/gojsonschema v1.2.0
go: downloading k8s.io/apiextensions-apiserver v0.30.3
go: downloading github.com/evanphx/json-patch v5.7.0+incompatible
go: downloading github.com/Masterminds/vcs v1.13.3
go: downloading github.com/opencontainers/image-spec v1.1.0-rc6
go: downloading oras.land/oras-go v1.2.5
go: downloading github.com/containerd/containerd v1.7.12
go: downloading github.com/Masterminds/squirrel v1.5.4
go: downloading github.com/jmoiron/sqlx v1.3.5
go: downloading github.com/rubenv/sql-migrate v1.5.2
go: downloading github.com/imdario/mergo v0.3.13
go: downloading github.com/go-logr/logr v1.4.1
go: downloading github.com/mattn/go-isatty v0.0.17
go: downloading github.com/mattn/go-runewidth v0.0.9
go: downloading github.com/russross/blackfriday/v2 v2.1.0
go: downloading github.com/BurntSushi/toml v1.3.2
go: downloading github.com/gobwas/glob v0.2.3
go: downloading golang.org/x/crypto v0.25.0
go: downloading github.com/Masterminds/goutils v1.1.1
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/huandu/xstrings v1.4.0
go: downloading github.com/shopspring/decimal v1.3.1
go: downloading github.com/spf13/cast v1.5.0
go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading golang.org/x/sync v0.7.0
go: downloading k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340
go: downloading sigs.k8s.io/kustomize/api v0.13.5-0.20230601165947-6ce0bf390ce3
go: downloading sigs.k8s.io/kustomize/kyaml v0.14.3-0.20230601165947-6ce0bf390ce3
go: downloading golang.org/x/net v0.23.0
go: downloading github.com/mitchellh/reflectwalk v1.0.2
go: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
go: downloading github.com/fvbommel/sortorder v1.1.0
go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d
go: downloading k8s.io/component-base v0.30.3
go: downloading github.com/docker/cli v25.0.1+incompatible
go: downloading github.com/docker/docker v25.0.6+incompatible
go: downloading github.com/lann/builder v0.0.0-20180802200727-47ae307949d0
go: downloading golang.org/x/oauth2 v0.10.0
go: downloading github.com/go-gorp/gorp/v3 v3.1.0
go: downloading github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535
go: downloading k8s.io/apiserver v0.30.3
go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible
go: downloading google.golang.org/protobuf v1.33.0
go: downloading golang.org/x/time v0.3.0
go: downloading github.com/go-openapi/jsonreference v0.20.2
go: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
go: downloading github.com/go-openapi/swag v0.22.3
go: downloading github.com/chai2010/gettext-go v1.0.2
go: downloading github.com/MakeNowJust/heredoc v1.0.0
go: downloading github.com/mitchellh/go-wordwrap v1.0.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading google.golang.org/grpc v1.58.3
go: downloading github.com/docker/docker-credential-helpers v0.7.0
go: downloading github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0
go: downloading github.com/distribution/reference v0.5.0
go: downloading github.com/go-errors/errors v1.4.2
go: downloading github.com/go-openapi/jsonpointer v0.19.6
go: downloading github.com/klauspost/compress v1.16.0
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.45.0
go: downloading go.opentelemetry.io/otel v1.19.0
go: downloading go.opentelemetry.io/otel/trace v1.19.0
go: downloading github.com/emicklei/go-restful/v3 v3.11.0
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/xlab/treeprint v1.2.0
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/felixge/httpsnoop v1.0.3
go: downloading go.opentelemetry.io/otel/metric v1.19.0
go: downloading github.com/gorilla/mux v1.8.0
go: downloading go.starlark.net v0.0.0-20230525235612-a134d8f9ddca
go: downloading github.com/moby/spdystream v0.2.0
go: downloading github.com/prometheus/client_golang v1.16.0
go: downloading github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
go: downloading github.com/cespare/xxhash/v2 v2.2.0
go: downloading github.com/prometheus/client_model v0.4.0
go: downloading github.com/prometheus/common v0.44.0
go: downloading github.com/prometheus/procfs v0.10.1
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d
Getting operator registry image
docker pull quay.io/operator-framework/opm:v1.47.0
v1.47.0: Pulling from operator-framework/opm
Digest: sha256:077e654da39d055ea56edc91faa2c36d5e8f4dec58db2017fed24a80246b6ca7
Status: Image is up to date for quay.io/operator-framework/opm:v1.47.0
quay.io/operator-framework/opm:v1.47.0
/home/runner/go/bin/yq-v3.0.0-20201202084205-8846255d1c37 w -i deploy/upstream/values.yaml olm.image.ref quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37
/home/runner/go/bin/yq-v3.0.0-20201202084205-8846255d1c37 w -i deploy/upstream/values.yaml catalog.image.ref quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37
/home/runner/go/bin/yq-v3.0.0-20201202084205-8846255d1c37 w -i deploy/upstream/values.yaml package.image.ref quay.io/operator-framework/olm@sha256:cfdda06700fae2962ac89e677c1968517f8c5fae838ca31774b909f545d7be37
/home/runner/go/bin/yq-v3.0.0-20201202084205-8846255d1c37 w -i deploy/upstream/values.yaml -- catalog.opmImageArgs "--opmImage=quay.io/operator-framework/opm@sha256:077e654da39d055ea56edc91faa2c36d5e8f4dec58db2017fed24a80246b6ca7"
./scripts/package_release.sh v0.19.0 deploy/upstream/manifests/v0.19.0 deploy/upstream/values.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-catalogsources.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-clusterserviceversions.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-installplans.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-olmconfigs.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-operatorconditions.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-operatorgroups.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-operators.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/crds/0000_50_olm_00-subscriptions.crd.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_00-namespace.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_00-namespace.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_01-olm-operator.serviceaccount.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_01-olm-operator.serviceaccount.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_09-aggregated.clusterrole.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_09-aggregated.clusterrole.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_01-olm-operator.serviceaccount.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_07-olm-operator.deployment.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_08-catalog-operator.deployment.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_17-upstream-operators.catalogsource.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_15-packageserver.clusterserviceversion.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_02-olmconfig.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_13-operatorgroup-default.yaml
wrote /tmp/tmp.uGEFWqNAGv/chart/olm/templates/0000_50_olm_13-operatorgroup-default.yaml

ln -sfFn ./v0.19.0 deploy/upstream/manifests/latest
./scripts/package_quickstart.sh deploy/upstream/manifests/v0.19.0 deploy/upstream/quickstart/olm.yaml deploy/upstream/quickstart/crds.yaml deploy/upstream/quickstart/install.sh
Wrote manifests to deploy/upstream/quickstart/olm.yaml and deploy/upstream/quickstart/crds.yaml
make[1]: Leaving directory '/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager'

@m1kola
Copy link
Member

m1kola commented Oct 30, 2024

OLM_VERSION in the repo root still contains 0.19.0. It seems like make release depends on this:

release: RELEASE_VERSION=v$(shell cat OLM_VERSION) #HELP Generate an OLM release (NOTE: before running release, bump the version in ./OLM_VERSION and push to master, then tag those builds in quay with the version in ./OLM_VERSION)

But it is not clear to me how all the releases after 0.19.0 were created correctly.

@m1kola
Copy link
Member

m1kola commented Oct 30, 2024

Ok, I think I understand the issue now. In the release workflow we pass the actual version here:

run: make release ver=${{ env.IMAGE_TAG }} IMAGE_REPO=quay.io/operator-framework/olm

But we no longer read it in the Makefile after this change in #3280

@m1kola m1kola mentioned this issue Oct 30, 2024
11 tasks
@acornett21
Copy link

@m1kola Thanks for diving into this, It seems like #3280 should have been broken into more manageable pieces so it could have been better reviewed. A PR with ~1800 changes can't be reviewed properly IMO.

Is the plan to merge in #3426 and re-release 0.29.0? ie delete the current release, images, etc, and re-do a new 0.29.0 release?

@m1kola
Copy link
Member

m1kola commented Oct 31, 2024

It seems like #3280 should have been broken into more manageable pieces so it could have been better reviewed. A PR with ~1800 changes can't be reviewed properly IMO.

I'm a big fan of small PRs and this one could've probably been spliced into smaller ones. But to be fair - most of the file changes are in vendor directory.

We just missed the fact that ver is being used in goreleaser.yaml workflow (which is not in the PR's diff).

Is the plan to merge in #3426 and re-release 0.29.0? ie delete the current release, images, etc, and re-do a new 0.29.0 release?

I'm not a big fan of replacing releases and I'm not sure that our tooling supports this. Probably will be better to create a new release and potentially remove the old one. Once #3426 is ready to go in I'll think about it.

@acornett21
Copy link

I'm not a big fan of replacing releases and I'm not sure that our tooling supports this. Probably will be better to create a new release and potentially remove the old one. Once #3426 is ready to go in I'll think about it.

Maybe all that's needed is to remove the assets from 0.29.0 and make a note that it was broken, so the assets were removed. Not sure if that's a more confusing experience or not.

@jon-nfc
Copy link

jon-nfc commented Oct 31, 2024

is it only olm.yaml that's affected or are crds.yaml affected too?

@jmazzitelli
Copy link
Contributor

So -- what's the final solution? Is 0.29.0 fixed or is another release coming?

@m1kola
Copy link
Member

m1kola commented Nov 1, 2024

We had a discussion in K8s slack in this thread and it was decided that the best course of action is:

  1. Add a note to 0.29.0 release and remove artefacts from it. This part is already done.
  2. Release a new version. I'll be looking into this today.

@m1kola
Copy link
Member

m1kola commented Nov 4, 2024

We have a number of issues with release automation tooling which we need to resolve. I fixed some, but we are still not 100% recovered.

So far I merged the following:

But it looks like our manifests get populated with arch specific image refs based on the CI arch instead of a ref of v0.30.0. Still working on it.

I also noticed that this issue got automatically closed. Re-opening for now.

@m1kola
Copy link
Member

m1kola commented Nov 12, 2024

We just released v0.30.0.

I'm sorry it took so long to address this: we had several issues due to changes in the release automation which needed to be addressed and while fixing these we also discovered some other issues.

@m1kola m1kola closed this as completed Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants