Intermittent unexpected EOF

[RonnanSouza]

The Issue

Several ImageUpdateAutomations are intermittently failing our clusters. We use git sources pointing to our Gitlab using deploy keys to authenticate through SSH.

Artifacts

Flux Check

► checking prerequisites
✗ flux 0.37.0 <0.38.3 (new version is available, please upgrade)
✔ Kubernetes 1.23.12 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.27.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.27.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.23.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.31.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.29.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.32.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git Source

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: provisioning-api
  namespace: provisioning-api
  gitImplementation: go-git
  ignore: |
    /*
    !/kubernetes/kube-stag-us-cilium.tfgco.com
  interval: 15m
  ref:
    branch: master
  secretRef:
    name: provisioning-api-git-credentials
  timeout: 60s
  url: ssh://[email protected]/infrastructure/provisioning-api

ImageUpdateAutomation

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: provisioning-api
  namespace: provisioning-api
spec:
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: [email protected]
        name: flux2-bot
      messageTemplate: Automated image update by Flux
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: provisioning-api
    namespace: provisioning-api
  update:
    path: /kubernetes/kube-stag-us-cilium.tfgco.com
    strategy: Setters

Logs

image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.486Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"}}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.486Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"},"ref":{"branch":"master"},"working":"/tmp/provisioning-api-provisioning-api1860084948"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.862Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","count":1,"manifests-path":"/tmp/provisioning-api-provisioning-api1860084948/kubernetes/kube-stag-us-cilium.tfgco.com"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.864Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","working":"/tmp/provisioning-api-provisioning-api1860084948"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"error","ts":"2023-01-19T23:22:45.020Z","msg":"unexpected EOF","name":"provisioning-api","namespace":"provisioning-api","reconciler kind":"ImageUpdateAutomation","annotations":null,"error":"error","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).AnnotatedEventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:137\ngithub.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:114\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).event\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:593\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile.func2\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:163\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.020Z","logger":"events","msg":"unexpected EOF","type":"Warning","object":{"kind":"ImageUpdateAutomation","namespace":"provisioning-api","name":"provisioning-api","uid":"c4762892-cde9-4ac4-a32d-af8bf726449f","apiVersion":"image.toolkit.fluxcd.io/v1beta1","resourceVersion":"1051457684"},"reason":"error"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"error","ts":"2023-01-19T23:22:45.044Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","error":"unexpected EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.809Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"}}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.810Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"},"ref":{"branch":"master"},"working":"/tmp/provisioning-api-provisioning-api2243875591"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:46.168Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","count":1,"manifests-path":"/tmp/provisioning-api-provisioning-api2243875591/kubernetes/kube-stag-us-cilium.tfgco.com"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:46.169Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","working":"/tmp/provisioning-api-provisioning-api2243875591"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"info","ts":"2023-01-19T23:22:47.573Z","msg":"Committed and pushed change 8e20661c6840a226463667890b5bfec937fda422 to master\nAutomated image update by Flux","name":"provisioning-api","namespace":"provisioning-api","reconciler kind":"ImageUpdateAutomation","reason":"info","annotations":null}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:47.573Z","logger":"events","msg":"Committed and pushed change 8e20661c6840a226463667890b5bfec937fda422 to master\nAutomated image update by Flux","type":"Normal","object":{"kind":"ImageUpdateAutomation","namespace":"provisioning-api","name":"provisioning-api","uid":"c4762892-cde9-4ac4-a32d-af8bf726449f","apiVersion":"image.toolkit.fluxcd.io/v1beta1","resourceVersion":"1051457715"},"reason":"info"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"info","ts":"2023-01-19T23:22:47.575Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","revision":"8e20661c6840a226463667890b5bfec937fda422","branch":"master"}

[aryan9600]

we introduced some git related changes in v0.28.0 of image-automation-controller. could you try updating your flux installation to the latest version and see if that helps?
also, how intermittent are these errors? these errors do tend to happen when the network is patchy or the git server is misbehaving for some reason.

[RonnanSouza]

could you try updating your flux installation to the latest version and see if that helps?

Hey @aryan9600 👋, sure! I'll update the controller and leverage it do update all flux controllers. I'll let you know if it got fixed soon.

also, how intermittent are these errors? these errors do tend to happen when the network is patchy or the git server is misbehaving for some reason.

Not actually, I tried with a test application and it happened in 1/2 of the times I triggered the update. And I couldn't notice any issue with either gitlab or the network, also the source controller didn't present any issues at the time.