Image automation has stopped #2219

jwerre · 2021-12-15T22:46:03Z

jwerre
Dec 15, 2021

Today I pushed a new image to DockerHub (as I have many times before) and it was not reconciled. I thought this may have something to do with the semver of the ImagePolicy being out of bounds but it seems ok:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: app
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: app
  policy:
    semver:
      range: ">=4.0.0-0"

The latest Tag in DockerHub is 4.8.3. I'm at a loss here since everything else seems to be in order. If I update the semver range and push it to my master branch that's all I need to do correct? Or do I need to reconcile after that?

flux check
► checking prerequisites
✔ Kubernetes 1.19.13-eks-8df270 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.14.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.18.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.14.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.18.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.19.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.19.2
✔ all checks passed

flux logs --since 15m --name app
2021-12-15T22:19:12.846Z info ImageRepository/app.flux-system - reconciliation finished in 766.940532ms, next run in 1m0s
2021-12-15T22:20:13.559Z info ImageRepository/app.flux-system - reconciliation finished in 712.611008ms, next run in 1m0s
2021-12-15T22:21:14.319Z info ImageRepository/app.flux-system - reconciliation finished in 758.814849ms, next run in 1m0s
2021-12-15T22:22:15.032Z info ImageRepository/app.flux-system - reconciliation finished in 711.489965ms, next run in 1m0s
2021-12-15T22:23:15.778Z info ImageRepository/app.flux-system - reconciliation finished in 745.53011ms, next run in 1m0s
2021-12-15T22:24:16.465Z info ImageRepository/app.flux-system - reconciliation finished in 687.11394ms, next run in 1m0s
2021-12-15T22:25:17.220Z info ImageRepository/app.flux-system - reconciliation finished in 753.750375ms, next run in 1m0s
2021-12-15T22:26:17.949Z info ImageRepository/app.flux-system - reconciliation finished in 729.136004ms, next run in 1m0s
2021-12-15T22:27:18.690Z info ImageRepository/app.flux-system - reconciliation finished in 739.822106ms, next run in 1m0s
2021-12-15T22:28:19.383Z info ImageRepository/app.flux-system - reconciliation finished in 691.69087ms, next run in 1m0s
2021-12-15T22:29:20.082Z info ImageRepository/app.flux-system - reconciliation finished in 699.2615ms, next run in 1m0s
2021-12-15T22:30:20.798Z info ImageRepository/app.flux-system - reconciliation finished in 715.670753ms, next run in 1m0s
2021-12-15T22:31:21.590Z info ImageRepository/app.flux-system - reconciliation finished in 790.360675ms, next run in 1m0s
2021-12-15T22:32:22.351Z info ImageRepository/app.flux-system - reconciliation finished in 760.800424ms, next run in 1m0s
2021-12-15T22:33:23.090Z info ImageRepository/app.flux-system - reconciliation finished in 738.723793ms, next run in 1m0s
2021-12-15T22:21:14.355Z info ImagePolicy/app.flux-system - Discarding event, no alerts found for the involved object
2021-12-15T22:26:17.970Z info ImagePolicy/app.flux-system - Discarding event, no alerts found for the involved object
2021-12-15T22:31:21.613Z info ImagePolicy/app.flux-system - Discarding event, no alerts found for the involved object

flux get sources git
NAME       	READY	MESSAGE                         REVISION        SUSPENDED
flux-system	True 	Fetched revision: master/xxx	master/xxx	False

flux get image repository
NAME     	READY	MESSAGE                       	LAST SCAN                	SUSPENDED
app      	True 	successful scan, found 10 tags	2021-12-15T15:41:31-07:00	False
...

There are 10 tags so Flux knows there are newer images but my app manifest is not being updated.

Answered by jwerre

Dec 17, 2021

On the advice of @stefanprodan I restarted the image-automation-controller and it resolved the issue.

kube get pods -n flux-system

NAME                                          READY   STATUS    RESTARTS   AGE
helm-controller-96dc99bfd-68477               1/1     Running   0          39h
image-automation-controller-64c955c59-drglp   1/1     Running   0          72s  << -- This is the one.
image-reflector-controller-55fb7f765d-c2l2f   1/1     Running   0          10d
kustomize-controller-7bc878f768-kj5bj         1/1     Running   0          39h
notification-controller-945795558-57lc6       1/1     Running   0          10d
source-controller-65665fd68f-nv4j5            1/1     Running   0  …

View full answer

jwerre · 2021-12-17T00:10:28Z

jwerre
Dec 17, 2021
Author

I'm just going to document my progress here just in case anyone else runs into this issue. I took another stab at this and everything appears ok yet I'm still running the wrong semver on my pods. There are no errors in the logs; docker secrets checkouts out; flux knows the latest tag of the image.

flux get image policy

NAME     	READY	MESSAGE                                                                 LATEST IMAGE
app      	True 	Latest image tag for 'docker.io/xxx/app' resolved to: 4.8.3       	docker.io/xxx/app:4.8.3
[...]

Yet when I get the pods the AGE is days old and is running version 4.8.1. I confirmed today that this is happening for different deployments in my cluster, not just the app deployment.

I just manually changed the version in my manifest and pushed it to github then ran flux reconcile image repository app and that updated the image to the correct one. But still no automation.

14 replies

jwerre Dec 17, 2021
Author

yep:

    spec:
      imagePullSecrets:
        - name: my-secret
      containers:
        - image: docker.io/xxx/app-staging:4.8.3 # {"$imagepolicy": "flux-system:app"}
          name: app
          ports:

jwerre Dec 17, 2021
Author

If you restart the controller does it do anything?

I've re-run flux bootstrap, would that restart the controller? I'm not sure how I would restart it otherwise.

stefanprodan Dec 17, 2021
Maintainer

Restart with kubectl delete pod for the image automation controller pod.

jwerre Dec 17, 2021
Author

ahh... duh. I'll give that at try

jwerre Dec 17, 2021
Author

Yes! that did it! Thank you so much @stefanprodan.

jwerre · 2021-12-17T13:52:35Z

jwerre
Dec 17, 2021
Author

On the advice of @stefanprodan I restarted the image-automation-controller and it resolved the issue.

kube get pods -n flux-system

NAME                                          READY   STATUS    RESTARTS   AGE
helm-controller-96dc99bfd-68477               1/1     Running   0          39h
image-automation-controller-64c955c59-drglp   1/1     Running   0          72s  << -- This is the one.
image-reflector-controller-55fb7f765d-c2l2f   1/1     Running   0          10d
kustomize-controller-7bc878f768-kj5bj         1/1     Running   0          39h
notification-controller-945795558-57lc6       1/1     Running   0          10d
source-controller-65665fd68f-nv4j5            1/1     Running   0          39h

kubectl delete pods image-automation-controller-64c955c59-drglp -n flux-system

pod "image-automation-controller-64c955c59-drglp" deleted

After that pod spun back up my pods updated shortly thereafter.

1 reply

stefanprodan Dec 17, 2021
Maintainer

I've created an issue here fluxcd/image-automation-controller#282

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image automation has stopped #2219

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 15 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Image automation has stopped #2219

jwerre Dec 15, 2021

Replies: 2 comments · 15 replies

jwerre Dec 17, 2021 Author

jwerre Dec 17, 2021 Author

jwerre Dec 17, 2021 Author

stefanprodan Dec 17, 2021 Maintainer

jwerre Dec 17, 2021 Author

jwerre Dec 17, 2021 Author

jwerre Dec 17, 2021 Author

stefanprodan Dec 17, 2021 Maintainer

jwerre
Dec 15, 2021

Replies: 2 comments 15 replies

jwerre
Dec 17, 2021
Author

jwerre Dec 17, 2021
Author

jwerre Dec 17, 2021
Author

stefanprodan Dec 17, 2021
Maintainer

jwerre Dec 17, 2021
Author

jwerre Dec 17, 2021
Author

jwerre
Dec 17, 2021
Author

stefanprodan Dec 17, 2021
Maintainer