images are not fetched

[kadzielawa]

Hello,
thank you for your powerful tool, I really like it! great job ! it's big milestone to gitops :)
I have problem with fluxcd2 because during creating processes which would be automatically after pushing images to ECR repository, HelmRelease is not upgrading containers however ImageUpdateAutomation works great and helm values file is updated on git repository very well.

not upgrading containers - I mean that even in repository the image is in the newest version eg.(sdd9) container is still in wrong version...
do you have any ideas what it's wrong with those files?

flux logs doesn't return nothing special... I see only that every one minute updating working.

something like values.yaml are not valid recognize by kubernetes

Every 2.0s: flux get images all                                                                                                                                                                                                                                                                                            iMac-Jakub.local: Sun Apr 11 09:45:36 2021

NAME                    READY   MESSAGE                         LAST SCAN                       SUSPENDED
imagerepository/podinfo True    successful scan, found 34 tags  2021-04-11T09:44:46+02:00       False

NAME                    READY   MESSAGE                                                                                                 LATEST IMAGE
imagepolicy/podinfo     True    Latest image tag for '247635813899.dkr.ecr.eu-central-1.amazonaws.com/hey-flux' resolved to: sdd9      247635813899.dkr.ecr.eu-central-1.amazonaws.com/hey-flux:sdd9

NAME                                    READY   MESSAGE                                                         LAST RUN                        SUSPENDED
imageupdateautomation/flux-system       True    no updates made; last commit 58dd783 at 2021-04-11T07:35:46Z    2021-04-11T09:44:53+02:00       False

structure (my-private-repo)
-file1.yml
-file2.yml
-file3.yml
---eks(directory)
-------heyflux(directory)
----------values.yaml
----------Chart.yaml
----------rest chart files

file1.yml

apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  checkout:
    branch: master
    gitRepositoryRef:
      name: podinfo
  commit:
    authorEmail: [email protected]
    authorName: fluxcdbot
    messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
  interval: 1m0s
  push:
    branch: master
  update:
    path: ./eks/heyflux
    strategy: Setters

file2.yml

---
apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImageRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  image: 247635813899.dkr.ecr.eu-central-1.amazonaws.com/hey-flux
  interval: 1m0s
  secretRef:
    name: aws-ecr
---
apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: podinfo
  filterTags:
    pattern: '^sdd[a-fA-F0-9]+'
  policy:
    alphabetical:
      order: asc

file3.yml:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: ssh://[email protected]/muumerepos/my-private-repo.git
  secretRef:
    name: ssh-credentials
  ref:
    branch: master
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: heyflux
  namespace: flux-system
spec:
  interval: 1m
  upgrade:
    remediation:
      retries: -1
  chart:
    spec:
      chart: ./eks/heyflux
      valuesFile: ./eks/heyflux/values.yaml
      sourceRef:
        kind: GitRepository
        name: podinfo
        namespace: flux-system
      interval: 1m

eks/heyflux/values.yaml

# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1
envName: "default"
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%
muumeerp:
  image:
    repository: 247635813899.dkr.ecr.eu-central-1.amazonaws.com/hey-flux
    tag: sdd9 # {"$imagepolicy": "flux-system:podinfo:tag"}
    pullPolicy: Always
  service:
    type: ClusterIP
    port: 8080

[martijnjanssen]

I'm having the same issue, but with a deployment. My ImagePolicy is able to retrieve the correct image and tag, but the image itself is not updated for some reason. The ImageUpdateAutomation doesn't see any updates and constantly has the no updates made message. My Deployment also has a strategy which is set to Recreate, but the RollingUpdate gives the same result. Something is wrong, but I'm unsure what. If needed I can attach my files here too. Otherwise I'll just see what the response to kadzielawa is.

[kadzielawa]

@martijnjanssen please take a look on the discussion which we have here:
fluxcd/flux2#1249

[squaremo]

@martijnjanssen It sounds like you have a different problem. As I parse the discussion here, the automated updates are working (writing to git) but do not have an effect when synced to the cluster. I think your problem is that the automation does not make a commit, is that right?

[martijnjanssen]

Yes, I will take a look at it again later, maybe I can spot the issue. If not I can file an issue or start a discussion.

It does indeed not make a commit, however, it does detect the newer image. Anyway, I'll stop bumping this issue 😉

Update: I have fixed my issue by moving my ImageRepository, ImagePolicy and ImageUpdateAutomation to the flux-system namespace. In the someimage namespace I set the imagepolicy in the kustomization as such: newTag: "48" # {"$imagepolicy": "flux-system:someimage:tag"}, where someimage is the name of the image. It's according to the documentation on fluxcd. In essence, from the documentation it wasn't entirely clear to me that these resources SHOULD be in the flux-system namespace.

[sanjvij]

@squaremo Did you managed to fix your problem? I am having the same issue whereby the containers are not getting updated with new image. In my case the flux is not even updating the deployment yaml with newer version. However when I run get image policy, I can see that flux is able to identify that a newer version is available. I have logged my issue here . Will appreciate any help or pointers [https://github.com/fluxcd/flux2/issues/1274]

[nomeelnoj]

I think i might know what is going on here, if I understand the issue correctly.

You are having Flux update the values.yaml file in your helm chart when a new image is pushed to ECR, which is working properly, but then you are expecting flux to redeploy your helm chart into the cluster. From what I understand, this is not going to work. In order for flux to see changes in a helm chart that is hosted in a git repository, the version of that chart in Chart.yaml needs to change. This is happening because fluxv2 pulls helm charts from git as versioned artifacts based on the version you have defined in Chart.yaml, versus using git metadata as it did in fluxv1.

Suggest a slightly different approach:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: heyflux
  namespace: flux-system
spec:
  chart:
    spec:
      chart: ./eks/heyflux
      valuesFile: ./eks/heyflux/values.yaml
      values: # Not sure if you can provide both valuesFile and values...haven't tested this
        muumeerp:
          image:
            tag: sdd9 # {"$imagepolicy": "flux-system:podinfo:tag"}
# ... removed for brevity

As I understand it, Flux will commit back to your repo into file3.yaml due to the marker above whenever you push an image to ECR that matches your ImagePolicy definition. When the git repository is updated with the new image tag, the source-controller will pick up the changes and sync them into your cluster, passing off to the kustomize-controller which will update the HelmRelease object with the new tag from your file3.yaml, and then the helm-controller will run the equivalent of a helm upgrade. That may not be 100% correct, but I believe this is all due to the version in your Chart.yaml not getting updated, per the docs here: https://toolkit.fluxcd.io/guides/faq-migration/#helm-integration (see the bottom row in the table).

[stefanprodan]

If you use the spec.values like suggested @nomeelnoj it will work without bumping the chart version in Git.

[dmytro-rachkov]

Hey there guys! Enjoyed so far my fluxcd experience, though similarly to kadzielawa I have issues with automatic image updates to git repo.

Here is some info around it:

Repo structure

 ── cd
│   ├── flux-system
│   │   ├── gotk-components.yaml
│   │   ├── gotk-sync.yaml
│   │   └── kustomization.yaml
│   └── sync
│       ├── flagger
│       │   ├── blueGreen.yaml
│       │   └── load-tester
│       └── flux
│           ├── helm_release.yaml
│           └── helm_source.yaml
├── flagger
└── rest-api-app
    ├── Chart.yaml
    ├── charts
    ├── templates
    └── values.yaml

bootstrap was done like this

         flux bootstrap gitlab \
         --owner=dmitry_rachkov/cd \
         --repository=BlueGreen \
         --branch=master \
         --path=cd \
         --components-extra=image-reflector-controller,image-automation-controller \
         --token-auth

image creation

         flux create image repository rest-api-app \
         --image=registry.hub.docker.com/dmitryrachkov/rest-api \
         --interval=1m

image policy

          flux create image policy rest-api-app \
         --image-ref=rest-api-app \
         --select-semver=">=0.0.6"

image update automation

         flux create image update rest-api-app \
          --git-repo-ref=rest-api-app \
          --git-repo-path="./rest-api-app" \
          --checkout-branch=master \
          --push-branch=master \
          --author-name=dmitry_rachkov \
          [email protected] \
          --commit-template="{{range .Updated.Images}}{{println .}}{{end}}" 

The update is expected to happen over helm release (based on GitRepository - helm source object)

helm_source.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: rest-api-app
  namespace: flux-system
spec:
  interval: 1m
  url: https://gitlab.com/dmitry_rachkov/cd/BlueGreen.git
  ref:
    branch: master
  ignore: |
    # exclude all
    /*
    # include charts directory
    !/rest-api-app/
    !/cd/sync/flux

helm_release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: rest-api-app
  namespace: rest-api-app
spec:
  interval: 1m0s
  chart:
    spec:
      chart: ./rest-api-app
      sourceRef:
        kind: GitRepository
        name: rest-api-app
        namespace: flux-system
      valuesFiles:
        - ./rest-api-app/values.yaml
  values:
    deployment:
      image:
        tag: 0.0.16 # {"$imagepolicy": "rest-api-app:rest-api-app:tag"}

Unfortunately automatic commit just doesn't happen with this setup for me though as I've read above and in documentation it should, would be grateful for ideas or hints.
Cheers

[kingdonb]

@dmitry-rachkov Thank you for your report, but I do not think your issue is the same as the original poster of this issue.

It appears you may have placed your GitRepository source in a different namespace than the Image Automation resources (ImagePolicy, ImageRepository, ImageUpdateAutomation). At least the $imagepolicy tag indicates it to be so – if your ImagePolicy and related resources are in flux-system then the tag YAML setter comment should look like this instead:

tag: 0.0.16 # {"$imagepolicy": "flux-system:rest-api-app:tag"}

If your resources were created in the flux-system namespace, as I suspect from the commands you showed, then updating the tag to this structure should cause the automation to spring to life fairly quickly. Please give that a try.

Cross-namespace secret references are not currently a valid configuration in Flux and variations on this issue seem to be collectively the top most common error facing users of ImageUpdateAutomation. (You're not even the first person in this thread to have hit this problem: #180 (comment))

I have submitted a PR to add a note to the API docs (#200), but if this turns out to not have been your problem, please open a separate issue, as I think the top poster did not have this misconfiguration issue. I am also working on a way to elevate this to a higher prominence in the FAQ, or even better, with an error message, but some variants of the issue like the way this problem seems to be presenting for you will unfortunately not be able to signal an error. (You apparently referenced in your kyaml setter an ImagePolicy at a location that does not exist, and ImagePolicy/ImageUpdateAutomation resources only look for references to themselves when scanning for tags that they should update.)

Please also a note from the community support policy: if you are not sure if you are using the API correctly, rather than filing it as an issue or bug report, we think the Q&A section of the discussion board on fluxcd/flux2/discussions is the most appropriate place to post an inquiry like this. (Thanks for using Flux!)

[kingdonb]

I marked this as a documentation, because there is documentation I would like to see added as we've discussed before, the "recipe" book showing which ways you can configure ImageUpdateAutomation in formulaic ways, and how cross-namespace support is or is not ready for use in general availability.

But I've also given this issue the 'bug' label as for usability, we want Flux's image automation to be as easy to configure from the CLI as it was in Flux v1 (or even, say, in Weave Cloud.) Users should be able to detect images which can be updated, select from whether they have a semver format or some regex, receive feedback and guidance at the CLI which helps them know when image tags match the format, which images can be selected from, and which candidate is considered as the latest before they apply the image update policy to the cluster, and so on.

Maybe all that belongs in a separate issue, but there are already a couple of different issues open related to the usability, I think this story is already found in one or more of them.

It would be even better if the CLI could help you determine what updates can be applied to a file, and trace for you what image update automations would apply to this gitrepository path when reconciled, or lead you to notice when IUA is not correctly associated with the path you wanted. And even when your sourceRef is inappropriately specified with an invalid or not present GitRepo, which can often happen when users are first trying automation and have not been appropriately warned about the cross-namespace issues, namely that cross-namespace secret access is not currently supported, so as it was maybe already explained above, image update which are not in the same namespace as ImageRepo, or similar issues, #201

Right now the CLI helps you build the YAML and interact with the status, but it does not do most of all that.

[bdols]

I spent some time determining why my ImageUpdateAutomations stopped working after I moved my Deployment objects. For me, it was that I didn't have a Kustomization for the new location setup.

I found that flux logs -f wasn't verbose enough and turned up debug logging via flux install --log-level=debug and checked the pod logs in flux-system.
For example , this flux logs entry:
2021-11-22T23:43:19.873Z info Kustomization/dev-env.flux-system - server-side apply completed
corresponded to what I was expecting to happen
In the kustomize-controller logs, the output field in the JSON is what I found to be useful:
{"level":"info","ts":"2021-11-22T23:43:19.873Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"dev-env","namespace":"flux-system","output":{"Deployment/dev-env/dev-env-app1":"configured","Deployment/dev-env/dev-env-app2":"unchanged","Deployment/dev-env/dev-env-app3":"unchanged","Deployment/dev-env/dev-env-app4":"unchanged","ImageUpdateAutomation/flux-system/smoke-test":"unchanged"}}

in the earlier failed attempts to update the image, the log doesn't indicate failure:
{"level":"info","ts":"2021-11-22T18:32:49.146Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"dev-env","namespace":"flux-system","output":{"ImageUpdateAutomation/flux-system/smoke-test":"unchanged"}}
for this, I would need to check the path of the Kustomization to see that it wasn't checking the right directory.

for ImageUpdateAutomation, the flux logs entries are also sparse:
2021-11-22T23:42:46.409Z debug ImageUpdateAutomation/smoke-test.flux-system - updating with setters according to image policies
2021-11-22T23:42:46.410Z debug ImageUpdateAutomation/smoke-test.flux-system - ran updates to working dir
which corresponds to:

{"level":"debug","ts":"2021-11-22T23:42:46.409Z","logger":"controller-runtime.manager.controller.imageupdateautomation","msg":"updating with setters according to image policies","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"smoke-test","namespace":"flux-system","count":6,"manifests-path":"/tmp/flux-system-app-helm525809833/clusters/dev"}
{"level":"debug","ts":"2021-11-22T23:42:46.410Z","logger":"controller-runtime.manager.controller.imageupdateautomation","msg":"ran updates to working dir","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"smoke-test","namespace":"flux-system","working":"/tmp/flux-system-app-helm525809833"}

for this I wanted to be able to check what files were getting pulled down into the working dir

I'm using the latest flux, 0.23.0. I found that I was able to run ImageUpdateAutomation in another namespace if I had all of the following in the different namespace: GitRepository, ImageRepository, ImagePolicy, ImageUpdateAutomation, and Kustomization.