Github Enterprise SSH handshake failing #652

steveizzle · 2022-03-31T16:32:51Z

Describe the bug

We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:

k get gitrepositories -n flux-system

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   False   failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

With version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):

k get gitrepositories -n flux-system 

NAME         URL                                                              AGE   READY   STATUS
flux2-sync   ssh://git@foo/abc/def   64d   True    Fetched revision: main/4afa5937b48b12a42c0739c13870c282826b3979

If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message

Steps to reproduce

We installed flux with the community helm charts without any relevant change to the values.yaml

Expected behavior

It is working again ^^

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

► checking prerequisites
✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used
✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.18.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.21.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.17.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.22.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.22.3
✔ all checks passed

Git provider

GitHub Enterprise

Container Registry provider

No response

Additional context

No response

Code of Conduct

I agree to follow this project's Code of Conduct

steveizzle · 2022-04-01T09:15:51Z

It seems that the bug was introduced in v0.22.0 of the sourcecontroller. Should i open the bug there?

pjbgf · 2022-04-01T09:27:19Z

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

pjbgf · 2022-04-01T09:32:43Z

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

steveizzle · 2022-04-01T09:34:41Z

@steveizzle thank you for reporting this. Do you mind sharing what gitImplementation are you using?

I used go-git and tried now libgit2. With libgit2 it is working.

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

Sure, i will give it a try and report back

steveizzle · 2022-04-01T10:07:47Z

Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often?

Would you mind running the latest version of source-controller and enabling the experimental transport to see whether the issue persists? Here are more information about it: #636 (comment)

The error seems to be consistent.

As soon as i enable the experimental transport neither libgit2 nor go-git is working:

        env:
        - name: EXPERIMENTAL_GIT_TRANSPORT
          value: "true"
        - name: RUNTIME_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: ghcr.io/fluxcd/source-controller:v0.22.5

We found out that one repository is using an ED25519 Key which seems to be still working. Normally we use RSA keys.

pjbgf · 2022-04-01T10:14:52Z

Thanks for that. Do you mind sharing the errors you get when using libgit2 and the experimental transport? The pod logs may have some extra information that could be useful.

ghost · 2022-05-25T08:14:43Z

We are getting the same error suddenly after redeploying a test cluster that we deploy often (almost every day) and doesn't give any errors with Flux.

Is there any resolution for this, or just keep trying until it works?

steveizzle · 2022-05-25T15:45:44Z

I currently pinned an old version, but will spent some time on getting more debug Infos the next days

pjbgf · 2022-05-27T13:54:58Z

@steveizzle @campanellaf this may have to do with some of the crypto level settings that may have changed upstream.
So to confirm, you are using RSA for your authentication keypair, right? What is the algorithm used for the known_host you are providing?

We have just released a RC with some improvements specially around the libgit2 implementation. Would you mind to also checking whether it fixes your problem?
ghcr.io/fluxcd/source-controller:rc-4b3e0f9a

hazim1093 · 2022-06-02T15:51:21Z

The issue still exists on v0.30.2.

@pjbgf I tried your RC with the go-git implementation, but no luck.
The algorithm we're using for known_hosts is ecdsa-sha2-nistp256

steveizzle · 2022-06-02T15:59:56Z

@pjbgf Regarding the rc, Unfortunately the rc didnt fix the problem:

This is a example of a git repository in my test cluster:

k get deploy -n flux-system  -o wide 

helm-controller               1/1     1            1           132d   manager      ghcr.io/fluxcd/helm-controller:v0.21.0                   app=helm-controller
image-automation-controller   1/1     1            1           132d   manager      ghcr.io/fluxcd/image-automation-controller:rc-48bcca59   app=image-automation-controller
image-reflector-controller    1/1     1            1           132d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.18.0        app=image-reflector-controller
kustomize-controller          1/1     1            1           132d   manager      ghcr.io/fluxcd/kustomize-controller:v0.25.0              app=kustomize-controller
notification-controller       1/1     1            1           132d   manager      ghcr.io/fluxcd/notification-controller:v0.23.5           app=notification-controller
source-controller             1/1     1            1           132d   manager      ghcr.io/fluxcd/source-controller:rc-4b3e0f9a             app=source-controller


k get gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync -o yaml

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  annotations:
    meta.helm.sh/release-name: flux2-sync
    meta.helm.sh/release-namespace: flux-system
    reconcile.fluxcd.io/requestedAt: "2022-06-02T17:34:40.013251+02:00"
  creationTimestamp: "2022-01-21T12:20:41Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 9
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: flux
    helm.sh/chart: flux2-sync-0.4.0
  name: flux2-sync
  namespace: flux-system
  resourceVersion: "556712592"
  uid: 8f76a6b4-cf42-49e1-af14-9608a1dc25c2
spec:
  gitImplementation: libgit2
  interval: 5m0s
  ref:
    branch: main
  secretRef:
    name: flux2-secret
  timeout: 20s
  url: ssh://git@SECRET_ENTERPRISE_URL
status:
  conditions:
  - lastTransitionTime: "2022-06-02T15:16:18Z"
    message: no artifact for resource in storage
    observedGeneration: 9
    reason: NoArtifact
    status: "True"
    type: Reconciling
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "False"
    type: Ready
  - lastTransitionTime: "2022-06-02T15:32:34Z"
    message: 'failed to checkout and determine revision: unable to fetch-connect to
      remote ''ssh://SECRET_ENTERPRISE_URL'': ssh:
      handshake failed: ssh: unable to authenticate, attempted methods [none publickey],
      no supported methods remain'
    observedGeneration: 9
    reason: GitOperationFailed
    status: "True"
    type: FetchFailed
  lastHandledReconcileAt: "2022-06-02T17:34:40.013251+02:00"
  observedGeneration: 8

steveizzle · 2022-06-02T16:05:27Z

I also checked out the algorithms for the known_hosts and got the following:

Working:

gitrepo1:
- known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256;
- key: ssh-ed25519
gitrepo2:
- known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
- key: ecdsa-sha2-nistp384

Not Working:

gitrepo3:
- known_hosts: 1 entry; keyB with ecdsa-sha2-nistp256
- identity key: ssh-rsa
gitrepo4:
- known_hosts: 2 entries; keyA with ssh-rsa and keyB with ecdsa-sha2-nistp256
- identity key: ssh-rsa

So i would say its not related to the known_hosts.. And still everything is working when i switch the source controller back to v0.21.2

steveizzle · 2022-06-02T17:22:39Z

I did not set the Experimental env Variable in the test with the rc. Is this still necessary with the rc?

stefanprodan · 2022-06-02T17:56:41Z

@steveizzle we have released source-controller yesterday with managed transport enabled by default. The official image is here: https://github.com/fluxcd/source-controller/pkgs/container/source-controller/23808902?tag=v0.25.0

Please test it with go-git and libgit2, no env var needed.

steveizzle · 2022-06-03T10:11:48Z

Unfortunately we have got the same problem with v0.25.0 :(

k get deploy -n flux-system source-controller -o yaml | grep -i image:
        image: ghcr.io/fluxcd/source-controller:v0.25.0

k get pods -n flux-system source-controller-d987bdf4f-mt8c8
NAME                                READY   STATUS    RESTARTS   AGE
source-controller-d987bdf4f-mt8c8   1/1     Running   0          23m

k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"libgit2"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched (no change)

flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to fetch-connect to remote 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

k patch gitrepositories.source.toolkit.fluxcd.io -n flux-system flux2-sync --type='json' -p='[{"op": "replace", "path":"/spec/gitImplementation", "value":"go-git"}]'
gitrepository.source.toolkit.fluxcd.io/flux2-sync patched

flux reconcile source git -n flux-system flux2-sync
► annotating GitRepository flux2-sync in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_ENTERPRISE_URL': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

steveizzle · 2022-06-09T12:55:50Z

I also let it ran with loglevel debug, but i think stacktrace doesnt help much:

{"level":"error","ts":"2022-06-09T12:45:48.671Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_URL.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to clone 'ssh://git@SECRET_URL.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:204\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:233\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

pjbgf · 2022-06-10T16:52:27Z

@steveizzle the repositories that are not working, are all of them on the same Git server?

Also, I noticed that the failing repos are using ssh-rsa for Auth-N, can you confirm that is enabled on the server?

We have released another release candidate that further improves on the connection management and observability.
Can you please give it a try using --log-level=trace?

ghcr.io/fluxcd/source-controller:rc-b877bc21

steveizzle · 2022-06-13T08:41:42Z

About the auth-N:

ssh -fNvv steveizzle@SECRET_ENTERPRISE_URL
.....
debug2: host key algorithms: [email protected],ecdsa-sha2-nistp256,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
...

That should be a yes. And the rsa key is still working with the old version of the source controller..

Regarding the new rc:

k get deploy -n flux-system source-controller -o wide
source-controller   1/1     1            1           142d   manager      ghcr.io/fluxcd/source-controller:rc-b877bc21   app=source-controller

# ecdsa-sha2-nistp384:
k apply -f flux-secret-new.yaml 
flux reconcile source git -n testing-cmo-flux flux-app-test

► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022

# ssh-rsa 4096:
k delete -f flux-secret-new.yaml
k apply -f flux-secret-new-rsa.yaml

flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to list remote for 'ssh://git@'SECRET_ENTERPRISE_URL: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'


# Relevant Reconcilliation Loop errors: 
k logs -n flux-system source-controller-67f4567c-7bz5z
{"level":"error","ts":"2022-06-13T08:36:38.301Z","logger":"controller.gitrepository","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","cid":"83666aeb-96e2-4a90-aaf1-6b3abe5f5bb1","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"github.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:78\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2022-06-13T08:36:38.301Z","msg":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","name":"flux-app-test","namespace":"testing-cmo-flux","reconciler kind":"GitRepository","annotations":null,"error":"GitOperationFailed","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:113\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.recordEvent\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:123\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.ErrorActionHandler\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/processor.go:80\ngithub.com/fluxcd/source-controller/internal/reconcile/summarize.(*Helper).SummarizeAndPatch\n\tgithub.com/fluxcd/source-controller/internal/reconcile/summarize/summary.go:182\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile.func1\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:211\ngithub.com/fluxcd/source-controller/controllers.(*GitRepositoryReconciler).Reconcile\n\tgithub.com/fluxcd/source-controller/controllers/gitrepository_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-06-13T08:36:38.302Z","logger":"events","msg":"Warning","object":{"kind":"GitRepository","namespace":"testing-cmo-flux","name":"flux-app-test","uid":"7e4d321f-b7b9-4370-bd92-70f825d88f42","apiVersion":"source.toolkit.fluxcd.io/v1beta2","resourceVersion":"567995259"},"reason":"GitOperationFailed","message":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain"}
{"level":"error","ts":"2022-06-13T08:36:38.306Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-app-test","namespace":"testing-cmo-flux","error":"failed to checkout and determine revision: unable to list remote for 'ssh://git@SECRET_ENTERPRISE_URL/flux-app-test.git': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

# v0.21.2: 
k edit deploy -n flux-system source-controller 
deployment.apps/source-controller edited

k get deploy -n flux-system source-controller -o wide
NAME                READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                     SELECTOR
source-controller   1/1     1            1           142d   manager      ghcr.io/fluxcd/source-controller:v0.21.2   app=source-controller

flux reconcile source git -n testing-cmo-flux flux-app-test
► annotating GitRepository flux-app-test in testing-cmo-flux namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master/bd3e6d2d8b24c97aa57dd8cb66db6fa9c113d022

Kevinwoolworth · 2022-06-14T01:06:39Z

i meet same issue. My flux env :

flux check
► checking prerequisites
✗ flux 0.24.1 <0.31.1 (new version is available, please upgrade)
✔ Kubernetes 1.21.9 >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.12.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.15.5
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.17.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.16.0
✔ all checks passed

kubectl get gitrepositories -n flux-system
NAME          URL                                                                 READY   STATUS                                                                                                                                                                                                            AGE
flux-system   ssh://[email protected]/xxxx/xxx-xxx.git   False   unable to clone 'ssh://[email protected]/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain   98d

flux reconcile source git flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'unable to clone 'ssh://[email protected]/xxxx/xxx-xxx.git', error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain'

aryan9600 · 2022-06-16T10:26:05Z

@Kevinwoolworth @steveizzle would it be possible for you to post the description of the secret that the GitRepository object refers to by running kubectl describe -n <your-namespace> secrets <your-secret>? Thanks

steveizzle · 2022-06-23T13:01:21Z

@aryan9600 Sure, we switched to working authentications, but i surely will help to assist on this issue:

Name:         flux-app-test
Namespace:    testing-cmo-flux
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      2602 bytes
identity.pub:  571 bytes
known_hosts:   581 bytes

hazim1093 · 2022-07-26T09:22:13Z

Hi is there any update or potential timeline for this issue?

cdenneen · 2022-08-17T16:19:23Z

Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue:
@aryan9600 @stefanprodan

❯ k -n flux-system describe secret flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      306 bytes
identity.pub:  205 bytes
known_hosts:   171 bytes

❯ k get gitrepository -A
NAMESPACE     NAME          URL                                        AGE    READY   STATUS
flux-system   flux-system   ssh://[email protected]/gitops/k8s-configs    139d   False   failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
flux-system   gilogging     https://gitlab.example.com/gi/apps/gilogging.git   139d   True    stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'

cdenneen · 2022-08-17T16:35:12Z

❯ flux check
► checking prerequisites
✗ flux 0.31.5 <0.32.0 (new version is available, please upgrade)
✔ Kubernetes 1.23.7-eks-4721010 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.11
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta1
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta1
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed
❯ flux get sources git
NAME       	REVISION                                     	SUSPENDED	READY	MESSAGE
flux-system	main/5160fe3801b485575424d53616dff76fcc74e478	False    	False	failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
gilogging  	main/adb4d09                                 	False    	True 	stored artifact for revision 'main/adb4d0932cf6366e435b5b540c48414b143c40cf'
❯ flux logs --level=error
❯ kubectl describe gitrepository flux-system -n flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  reconcile.fluxcd.io/requestedAt: 2022-08-17T10:52:38.057356-04:00
API Version:  source.toolkit.fluxcd.io/v1beta2
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2022-03-30T18:50:51Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  90866210
  UID:               6e766830-bebd-4f26-a1e2-07232736d405
Spec:
  Git Implementation:  go-git
  Interval:            1m0s
  Ref:
    Branch:  main
  Secret Ref:
    Name:   flux-system
  Timeout:  20s
  URL:      ssh://[email protected]/gitops/k8s-configs
Status:
  Artifact:
    Checksum:          0bf60b79888c89ee8ba48d42a296f8fafb1a27fb8b5b8d319e253c1fc981299e
    Last Update Time:  2022-08-15T18:49:12Z
    Path:              gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
    Revision:          main/5160fe3801b485575424d53616dff76fcc74e478
    Size:              37517
    URL:               http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/5160fe3801b485575424d53616dff76fcc74e478.tar.gz
  Conditions:
    Last Transition Time:     2022-08-16T16:40:35Z
    Message:                  failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   False
    Type:                     Ready
    Last Transition Time:     2022-08-15T16:56:08Z
    Message:                  stored artifact for revision 'main/5160fe3801b485575424d53616dff76fcc74e478'
    Observed Generation:      1
    Reason:                   Succeeded
    Status:                   True
    Type:                     ArtifactInStorage
    Last Transition Time:     2022-08-16T16:40:35Z
    Message:                  failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
    Observed Generation:      1
    Reason:                   GitOperationFailed
    Status:                   True
    Type:                     FetchFailed
  Content Config Checksum:    sha256:fcbcf165908dd18a9e49f7ff27810176db8e9f63b4352213741664245224f8aa
  Last Handled Reconcile At:  2022-08-17T10:52:38.057356-04:00
  Observed Generation:        1
  URL:                        http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
  Type     Reason              Age                  From               Message
  ----     ------              ----                 ----               -------
  Warning  GitOperationFailed  12m (x106 over 23h)  source-controller  failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/gitops/k8s-configs': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

darkowlzz · 2022-08-17T17:48:59Z

Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue

@cdenneen Can you share more details about your GitLab instance? I'm running GitLab 15.2.2-ee and it's working fine.

anetogit · 2022-08-25T23:06:40Z

I did these steps, using AWS EKS and CodeCommit, from the scratch:

eksctl create cluster --name flux-demo --nodegroup-name ng-flux

git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo

cd flux-demo

mkdir -p ./clusters/flux-demo/flux-system

flux install \
  --components-extra=image-reflector-controller,image-automation-controller \
  --export > ./clusters/flux-demo/flux-system/gotk-components.yaml

git add -A && git commit -m "add components" && git push

kubectl apply -f ./clusters/flux-demo/flux-system/gotk-components.yaml

flux check

aws iam create-user --user-name fluxserviceaccount

aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AWSCodeCommitFullAccess --user-name fluxserviceaccount

PUBLIC_KEY=$(cat ~/.ssh/flux_rsa.pub)
aws iam upload-ssh-public-key --user-name fluxserviceaccount --ssh-public-key-body $PUBLIC_KEY

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo \
  --branch=master \
  --private-key-file=flux_rsa \
  --interval=1m

and I got this error:

✗ failed to checkout and determine revision: unable to clone 'ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/flux-demo': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I am using:

flux version 0.32.0
source-controller:v0.26.1
eksctl 0.109.0-dev+78243b4c.2022-08-19T12:49:29Z
kubernetes 1.22

aryan9600 · 2022-08-27T08:04:18Z

@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using flux create secret git (ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use the flux create source --secret-ref instead of --private-key-file to authenticate.

stefanprodan · 2022-08-27T09:03:47Z

@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/

anetogit · 2022-08-28T02:03:48Z

@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using flux create secret git (ref: https://fluxcd.io/docs/cmd/flux_create_secret_git/) and then use the flux create source --secret-ref instead of --private-key-file to authenticate.

@aryan9600 No, my key is not protected with a password, there is no passphrase. I was using private key because of the CodeCommit of AWS. I am going to try flux create secret git. Thank you.

anetogit · 2022-08-28T02:08:05Z

@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/

@stefanprodan Yes. The tutorial works with the HTTPS:

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=https://git-codecommit.<region>.amazonaws.com/v1/repos/<my-repository> \
 ...

but not with SSH, because o the issue:

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=ssh://Your-SSH-Key-ID@git-codecommit.<region>.amazonaws.com/v1/repos/<my-repository> \
  ...

thanks.

caige-kelly · 2022-10-28T20:14:10Z

I hope this is placed in the correct thread. I am facing a similar issue with Github Organizations where I can successfully connect to private repositories via ssh, but after roughly 5-10 minutes 'flux get source git' will show failure with error message: failed to checkout and determine revision: unable to fetch-connect to remote 'https://github.com/{org}/{repo}': unhandled HTTP error 401 Authorization Required. I can run 'flux create source {flags}' right after and it will re-establish the connection without issue, then repeat failure.

LockedThread · 2023-06-08T17:53:14Z

This is still happening, on public Github, non-enterprise:

kubectl get deploy -n flux-system  -o wide
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                               SELECTOR
helm-controller               1/1     1            1           48d   manager      ghcr.io/fluxcd/helm-controller:v0.32.1               app=helm-controller
image-automation-controller   1/1     1            1           48d   manager      ghcr.io/fluxcd/image-automation-controller:v0.32.0   app=image-automation-controller
image-reflector-controller    1/1     1            1           48d   manager      ghcr.io/fluxcd/image-reflector-controller:v0.27.0    app=image-reflector-controller
kustomize-controller          1/1     1            1           48d   manager      ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.1      app=kustomize-controller
notification-controller       1/1     1            1           48d   manager      ghcr.io/fluxcd/notification-controller:v1.0.0-rc.1   app=notification-controller
source-controller             1/1     1            1           48d   manager      ghcr.io/fluxcd/source-controller:v1.0.0-rc.1         app=source-controller

kubectl -n flux-system describe secret flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
identity:      306 bytes
identity.pub:  205 bytes
known_hosts:   171 bytes

flux get source git flux-system

NAME            REVISION                SUSPENDED       READY   MESSAGE

flux-system     main@sha1:098b74a4      False           False   failed to checkout and determine revision: unable to list remote for 'ssh://[email protected]/REDACTED/REDACTED': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

hiddeco · 2023-06-08T18:54:35Z

@LockedThread given this issue is kind of stale, can you please share your source-controller version?

LockedThread · 2023-06-08T18:56:48Z

@LockedThread given this issue is kind of stale, can you please share your source-controller version?

The version does show in my comment but here is the output of flux version

helm-controller: v0.32.1
image-automation-controller: v0.32.0
image-reflector-controller: v0.27.0
kustomize-controller: v1.0.0-rc.1
notification-controller: v1.0.0-rc.1
source-controller: v1.0.0-rc.1

LockedThread · 2023-06-08T19:07:38Z

I also just decided to update to see if the issue fixed itself, still broken:

flux: v2.0.0-rc.5
helm-controller: v0.34.1
image-automation-controller: v0.34.1
image-reflector-controller: v0.28.0
kustomize-controller: v1.0.0-rc.4
notification-controller: v1.0.0-rc.4
source-controller: v1.0.0-rc.5

LockedThread · 2023-06-08T19:09:47Z

This is the bootstrap command I am using:

GITHUB_TOKEN=REDACTED flux bootstrap github --owner=REDACTED --repository=REDACTED --path=REDACTED --read-write-key --private --components-extra=image-reflector-controller,image-automation-controller

LockedThread · 2023-06-08T19:13:44Z

Here's more context from flux events:

4m43s (x25 over 6m22s)  Warning error                   ImageUpdateAutomation/flux-system       unable to clone 'ssh://[email protected]/REDACTED/REDACTED': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
3m7s (x9 over 6m24s)    Warning GitOperationFailed      GitRepository/flux-system               failed to checkout and determine revision: unable to clone 'ssh://[email protected]/REDACTED/REDACTED': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

hiddeco · 2023-06-08T19:28:33Z

Can you please try to rotate the credentials (and things like known_hosts)? The instructions for this can be found here: https://fluxcd.io/flux/installation/#deploy-key-rotation

LockedThread · 2023-06-08T19:34:47Z

Can you please try to rotate the credentials (and things like known_hosts)? The instructions for this can be found here: https://fluxcd.io/flux/installation/#deploy-key-rotation

I really appreciate the quick response. That worked. You're really saving me right now.

hiddeco · 2023-06-08T19:41:42Z

You're welcome!

If that worked (and based on the error), I suspect the deploy key got deleted from the repository. As from the top of my head, a known_hosts entry mismatch would yield another error (and the last change made by GitHub there was at the end of March). Given this, you may want to figure out who and/or what's to blame for this :-).

LockedThread · 2023-06-08T19:53:28Z

You're welcome!

If that worked (and based on the error), I suspect the deploy key got deleted from the repository. As from the top of my head, a known_hosts entry mismatch would yield another error (and the last change made by GitHub there was at the end of March). Given this, you may want to figure out who and/or what's to blame for this :-).

The deploy key wasn't deleted and it was being used up until 6 days ago when it randomly broke. This was in our production cluster and I was freaking out why my changes weren't getting deployed.

stefanprodan · 2023-06-08T20:27:02Z

Did your PAT expired 6 days ago? The deploy key expires when the GH token used for bootstrap expires. If you use PATs with an expiry date, you need to make sure to rotate the key before they expire.

LockedThread · 2023-06-08T20:27:57Z

Did your PAT expired 6 days ago? The deploy key expires when the GH token used for bootstrap expires. If you use PATs with an expiry date, you need to make sure to rotate the key before they expire.

None of the keys or PATs expired

yehoshuadimarsky · 2023-08-04T15:59:34Z

I had the same issue was driving me crazy. Turned out that I specified the URL wrong for SSH:

-  url: ssh://[email protected]/MY-REPO.git
+  url: ssh://[email protected]:22/MY-ORG/MY-REPO

stefanprodan transferred this issue from fluxcd/flux2 Apr 1, 2022

pjbgf added the area/git Git related issues and pull requests label Jun 10, 2022

Github Enterprise SSH handshake failing #652

Github Enterprise SSH handshake failing #652

Comments

steveizzle commented Mar 31, 2022

Describe the bug

Steps to reproduce

Expected behavior

Screenshots and recordings

OS / Distro

Flux version

Flux check

Git provider

Container Registry provider

Additional context

Code of Conduct

steveizzle commented Apr 1, 2022

pjbgf commented Apr 1, 2022

pjbgf commented Apr 1, 2022

steveizzle commented Apr 1, 2022

steveizzle commented Apr 1, 2022

pjbgf commented Apr 1, 2022 • edited Loading

ghost commented May 25, 2022

steveizzle commented May 25, 2022

pjbgf commented May 27, 2022

hazim1093 commented Jun 2, 2022

steveizzle commented Jun 2, 2022

steveizzle commented Jun 2, 2022 • edited Loading

steveizzle commented Jun 2, 2022

stefanprodan commented Jun 2, 2022

steveizzle commented Jun 3, 2022 • edited Loading

steveizzle commented Jun 9, 2022

pjbgf commented Jun 10, 2022

steveizzle commented Jun 13, 2022 • edited Loading

Kevinwoolworth commented Jun 14, 2022 • edited Loading

aryan9600 commented Jun 16, 2022

steveizzle commented Jun 23, 2022

hazim1093 commented Jul 26, 2022

cdenneen commented Aug 17, 2022

cdenneen commented Aug 17, 2022

darkowlzz commented Aug 17, 2022 • edited Loading

anetogit commented Aug 25, 2022

aryan9600 commented Aug 27, 2022

stefanprodan commented Aug 27, 2022

anetogit commented Aug 28, 2022 • edited Loading

anetogit commented Aug 28, 2022

caige-kelly commented Oct 28, 2022

LockedThread commented Jun 8, 2023

hiddeco commented Jun 8, 2023

LockedThread commented Jun 8, 2023

LockedThread commented Jun 8, 2023

LockedThread commented Jun 8, 2023

LockedThread commented Jun 8, 2023

hiddeco commented Jun 8, 2023

LockedThread commented Jun 8, 2023

hiddeco commented Jun 8, 2023

LockedThread commented Jun 8, 2023

stefanprodan commented Jun 8, 2023

LockedThread commented Jun 8, 2023

yehoshuadimarsky commented Aug 4, 2023

pjbgf commented Apr 1, 2022 •

edited

Loading

steveizzle commented Jun 2, 2022 •

edited

Loading

steveizzle commented Jun 3, 2022 •

edited

Loading

steveizzle commented Jun 13, 2022 •

edited

Loading

Kevinwoolworth commented Jun 14, 2022 •

edited

Loading

darkowlzz commented Aug 17, 2022 •

edited

Loading

anetogit commented Aug 28, 2022 •

edited

Loading