Image APIs v1alpha2 (-> v1beta1)

[squaremo]

The goal is to design the image-{reflector,automation} APIs for v1beta1, which will then be kept backward-compatible. To make sure changes have time in the field before that, the first part of the proposal is this:

Any changes here will be implemented in an API version v1alpha2, which breaks compatibility with v1alpha1; if successful, it will become v1beta1; otherwise, changes will be made to v1alpha2 until satisfactory.

The rest of the proposal explains features we want to make room for in the API, and where they will fit. I'll put together a suggested API with YAML examples, in an answer.

Image automation API

Use of sourcev1.GitRepository

At present the image automation object refers to a sourcev1.GitRepository, as a way to provide the location and credentials for the git repository to which to make commits. The discussion here explains the problem with doing this -- in short, since GitRepository is part of the source API, it has fields which aren't relevant to automation, which introduces dissonance and (as demonstrated in e.g., fluxcd/image-automation-controller#113, fluxcd/image-automation-controller#72).

There are two practical alternatives:

• introduce a new type in the API representing git repository used for automation;
• inline the fields needed into the ImageUpdateAutomation object

These each have their own downsides.

For the first: it is expensive to introduce another type into the API, for the user's understanding as well as for UI and so on.

For the second: if there is a desire to add, say, (S3) buckets as targets for automation, the argument for reusing types is stronger.

For both of them: the motivation for cross-namespace references is essentially to avoid duplicating secrets; and similarly, so that a GitRepository definition created for syncing can be reused. From that point of view, it would undermine that requirement to either introduce a new type or to duplicate the fields.

On balance, despite it being a triangular piece sideways through a rectangular hole, I think it's better to lean into using the sourcev1 type(s). That suggests these changes:

• use the field name sourceRef (making way for using buckets, potentially)
• consider respecting the ignore and timeout fields from GitRepository
• consider having .spec.checkout.reference (with semantics as for git repository, and possibly a restricted set of possible values) instead of .spec.checkout.branch
• consider making .spec.checkout optional and using .spec.reference from the GitRepository as a fallback value

Namespaces and source references

There is an issue (fluxcd/image-automation-controller#85) about how to organise image policies and image update objects. The problems explained there are:

• you may want to keep your policy objects alongside the workloads they pertain to, which implies the automation object is in the same namespace, which implies that the git repository object is in the same namespace, leading to duplication;
• you may wish to keep image automation object with the git repository object and secret, but the policies alongside the workloads they pertain to.

An extra motivation is that the markers used for automation are namespaced, and if you want to have only some of them updated at a time, you have only the namespace the policies are in, and the path given in the automation object as controls. Being able to split the policies into different namespaces without duplicating secrets everywhere would be helpful.

The proposed solutions to these are, respectively:

• Let the gitRepositoryRef (/sourceRef) refer to other namespaces
• Give a target namespace in the image automation object

It is unclear to me which of these is the right way to go, or if both are needed to serve the requirements without compromising. The good news is that there is room to add either or both of cross-namespace references and target namespaces after the fact, since they will be backward-compatible extensions.

Use of sourcev1.Bucket

Hypothetically there are uses for writing changes back to an S3 bucket or equivalent, rather than making commits to git -- perhaps as a way of interacting with CI tooling. Using sourceRef (with the expected sourcev1 type) instead of gitRepositoryRef makes room for this, since a bucket can be referenced there as well as a git repository.

However, the ImageUpdateAutomation fields .spec.commit, .spec.checkout, and .spec.push are all particular to using a git repository. Ideally, all the things that are particular to git would go in a "union" member; in other words, you supply the git field with all its particulars, or you supply the bucket field with all its particulars.

spec:
  sourceRef:
    kind: GitRepository
    name: foo
  update:
    strategy: Setters
  git:
    checkout:
      ref: # currently .checkout.branch, but other kinds of ref are possible
        branch: main
    commit:
      author: # current authorEmail and authorName
        name: Michael
        email: [email protected]
      messageTemplate: |
        Automated update
    push:
      branch: auto-update

or

spec:
  sourceRef:
    kind: Bucket
    name: config
  update:
    strategy: Setters
  bucket:
    # anything here?

The sourceRef itself is particular, but in the interest of keeping with other controllers it's less surprising to keep it at the top level.

Signing commits

Issue fluxcd/image-automation-controller#122 asks for commit signing. This can be introduced, backward-compatibly, with a secret ref field in the git particulars.

Verifying git ref

The sourcev1.GitRepository type has a field .spec.verify for supplying GPG keys to verify the git HEAD; this could be respected by the image automation controller without a change to the API.

Commit author

In #952 (comment) Stefan suggested in passing that the commit author finds be tidied up, from

spec:
  commit:
    authorEmail: [email protected]
    authorName: Flux automation

to

spec:
  commit:
    author:
      email: [email protected]
      name: Flux automation

and subsequently, we agreed that the email would be mandatory and the name optional. Since the commit section will move in a breaking change, this seems like a good tidy to make at the same time. If possible, the original fields could be included but deprecated.

Image reflector API

There are fewer feature requests outstanding against the image reflector controller and API -- it is simple and works fine in practice.

Pin the image tag or digest

It's possible to use a regular expression to pin an image policy to an exact image, but this issue argues that it's worth making this operation explicit in the API fluxcd/image-reflector-controller#117. This can be added backward-compatibly.

Exclude bad images at the repository level

Sometimes a particular image is just unusable, and when that happens it's useful to be able to exclude it from consideration by any policy (i.e., in the ImageRepository): fluxcd/image-reflector-controller#62. This can be added backward-compatibly.

Git revision image policy and build time policy

Flux v1 used the build timestamp (from a label, or by looking at image container config) as the default image policy, and this is requested as a feature for Flux v2: fluxcd/image-reflector-controller#43.

A slightly more far-out suggestion is to sort images according to the commits from which they were built: fluxcd/image-reflector-controller#15.

New kinds of policy for selecting images can be added backward-compatibly, by design.

Authentication for specific platforms

Kubernetes platforms usually have their own ambient access control (e.g., IAM in AWS), and this is used for granting access to their respective container image registries (ACR, GCR, ECR). We have a good, documented solution for at least those mentioned, which is to use a CronJob to create a docker secret. This is neat because it uses an off-the-shelf mechanism (Kubernetes' cron jobs) and works in sympathy with multi-tenancy (you have to be able to create a CronJob with permissions in the ambient access control to get a container registry secret).

But it's still possible that there will need to be platform-specific authentication help somewhere in the API, probably in the ImageRepository object. This can likely be done backward-compatibly -- i.e., doesn't need room made for it in the API.

This issue tracks image registry auth for different platforms: fluxcd/image-reflector-controller#11.

[stefanprodan]

I propose we add apiVersion as an optional field to sourceRef, this has been implemented across the GOTK APIs (KC, HC, NC). Even if now it seems redundant, this will help us in the future when the source API will be promoted to v1.

[squaremo]

Here are examples of the image.toolkit.fluxcd.io types with all the discussed bits put in, as marked by comments:

apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImageUpdateAutomation
metadata:
  name: foo
  namespace: foo-system
spec:
  interval: 5m
  suspend: false
  targetNamespace: app # "Namespaces and source references"; optional, use policies in this namespace, default is same namespace
  sourceRef: # renamed from .spec.checkout.gitRepositoryRef
    apiVersion: source.toolkit.fluxcd.io/v1beta1
    kind: GitRepository # | Bucket
    name: tenant-repo
    namespace: tenant # NB "Namespaces and source references" cross-namespace source ref
  update: # a union, other strategies could go in here
    path: ./config
    strategy: Setters
  git: # git repository source specific; | bucket: {...}
    checkout:
      # (gitRepositoryRef was here)
      ref: # optional; falls back to using the ref given in the git repository
        branch: main # | tag: v1 | commit: 321cba | semver: '1.x', if semantics for these allow
    commit:
      author: # required because .spec.commit.author.email is required
        name: Fluxbot
        email: [email protected]
      messageTemplate: |
        Another commit from fluxbot!
      signing: # "Signing commits" above
        secretRef:
          name: gpg-keys
    push:
      branch: auto-updates

apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImagePolicy
metadata:
  name: app-version
  namespace: app
spec:
  imageRepositoryRef:
    name: app-image
  policy:
    semver: # | alphabetical | numerical | timestamp | git, with their own fields
      range: '1.x'
  filterTags: {...} # as before
  pinVersion:  # "Pin the image tag or digest", not sure of design here, just a placeholder
    tag: v1.2.5 # | digest

apiVersion: image.toolkit.fluxcd.io/v1alpha2
kind: ImageRepository
metadata:
  name: app-image
  name: app
spec:
  image: podinfo
  exclude: # "Exclude bad images at the repository level" above, design TBD
  - v1.2.4
  interval: 10m
  timeout: 30s
  secretRef:
    name: dockerconfig
  certSecretRef:
    name: certs
  platform: {...}  # "Authentication for specific platforms", for platform-specific auth; requirement TBD
  suspend: false

[Frizlab]

The cross namespace reference of the sourceRef did not make it to beta1 did it? Will it make it in later beta or has it been completely abandoned?

[stefanprodan]

I'm working on ACLs for cross-namespace access, see this first RFC fluxcd/image-reflector-controller#162

[simongottschlag]

One thing that you are writing about is the secret duplication and enabling cross-namespace references. Another option (that may make the spec easier) could be to recommend a third party solution for that and leave it to the end users to decide if they want to manually duplicate something or use automation.

In my case, especially when using GitOps, duplication usually isn't a problem. If I need a secret per namespace the automation we use can do that instead of relying on Flux to take care of that using cross-namespace references.

If we don't use the automation for it, something else like the Azure KeyVault CSI driver can be used, or something like kubed to sync between namespaces: https://github.com/appscode/kubed

At least worth thinking about 😊

[simongottschlag]

Example from cert-manager docs: https://cert-manager.io/docs/faq/kubed/

[squaremo]

There's another use for cross-namespace references other than preventing duplication, which is to provide mediated access (being able to run automated updates) to a git repository without giving away full write permissions. You may or may not agree that is this is worth the trade-off ...

For the minute, the changes in fluxcd/image-automation-controller#139 do not add cross-namespace references, they just keep space open for them.

[squaremo]

Update on graduating to v1beta1

We've now released the image controllers, and a flux which will install them, that uses v1alpha2. There are some issues logged since then, but I do not think they are problems with the API itself:

• Empty commits image-automation-controller#166 -- this is not debugged yet, but my best guess is that it's a bug in the git implementation
• Commits happening too frequently after v1alpha2 image-automation-controller#168 -- this is due to PR 143, which occurred around the same time as the API update but is not a consequence of it

(and for the sake of completeness, bugs that have been closed were closed without changing the API)

Some issues logged suggest possible new features, so let's cover those off:

• Commits happening too frequently after v1alpha2 image-automation-controller#168 talks about batching changes from several image policies into one commit -- I'm fairly confident any new flags controlling batching can be added backward-compatibly, since they will describe a new behaviour
• Support sha256 digest pinning image-automation-controller#165 asks for the ability to use SHA256 to name images. This would be additive and backward-compatible.
• Support commit message template functions image-automation-controller#160 asks for additional templating functions, which is unlikely to have an effect on the API (since the template is just a string). Although it changes the meaning of the template field, it would be backward-compatible (assuming the functions added don't supersede what's in bare go text templating).
• Image markers are not usable with configuration transpilers  image-automation-controller#134 points out that the update markers are not usually preserved by tools that process YAMLs, making update by setters tricky. The mode of update is designed as an extension point -- another kind of update would be an alternative to setters, and additive.

(If I've missed any you think are worth mentioning, please tell me)

I propose the v1alpha2 image API is promoted to v1beta1. I think that this can be done by keeping both versions in the CRD, and letting the Kubernetes API server transparently replace the version -- i.e., it doesn't need a webhook, or for people to rewrite their manifests. But I will make sure that's the case, experimentally.

[stefanprodan]

it doesn't need a webhook, or for people to rewrite their manifests

We should encourage people to change the API version in Git for all manifests. We also have to decide when to remove the v1alpha1 and v1alpha2 from Flux distribution. Maybe a 3 months period would be enough for people to upgrade?