Support Service Account Keys defined in ImagePullSecrets #857

jnauska · 2021-12-16T14:09:19Z

What steps did you take and what happened:

We're using Google Artifact Registry to host our containers.

We have configured the access according to the Google Documentation: https://cloud.google.com/artifact-registry/docs/access-control

So we're using Service Account keys in our ImagePullSecrets. ImagePullSecrets would look like this (SA_KEY is encoded):

{
  "auths": {
    "repository_name": {
      "username": "_json_key_base64",
      "password": "SA_KEY"
    }
  }
}

Decoded SA_KEY would look like

{
  "type": "service_account",
  "project_id": "XYZ",
  "private_key_id": "XYZ",
  "private_key": "XYZ",
  "client_email": "XYZ@XYZ",
  "client_id": "XYZ",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "XYZ"
}

When Vulnerability Scan tries to Pull the image for scanning it gives out error:

"error":"reading .dockerconfigjson field of \"APPLICATION_NAME/image-pull-secrets\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

starboard/pkg/docker/config.go

Lines 26 to 36 in 8944e54

    
           func (v *BasicAuth) Decode() (string, string, error) { 
        
           	bytes, err := base64.StdEncoding.DecodeString(string(*v)) 
        
           	if err != nil { 
        
           		return "", "", err 
        
           	} 
        
           	split := strings.Split(string(bytes), ":") 
        
           	if len(split) != 2 { 
        
           		return "", "", fmt.Errorf("expected username and password concatenated with a colon (:)") 
        
           	} 
        
           	return split[0], split[1], nil 
        
           }

Is splitting the string with : and giving out errors if len(split) != 2. As the SA_KEY has multiple : included in the string, it gives out the error and breaks ImagePull

What did you expect to happen:

To be able to use Service Account keys in ImagePullSecrets.

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

Environment:

Starboard version (use starboard version): 0.13.1 (starboard-operator)
Kubernetes version (use kubectl version): v1.20.10-gke.1600
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): N/A

The text was updated successfully, but these errors were encountered:

danielpacak · 2021-12-17T08:51:13Z

How is this issue different from #279 ? In the current implementation Starboard reads imagePullSecrets to get username and password that are eventually passed to Trivy as TRIVY_USERNAME and TRIVY_PASSWORD environment variables. That's the only officially supported authentication scheme.

For GCE, ECR, and other managed registries more work as to be done.

jnauska · 2021-12-17T08:56:22Z

I guess that purpose of the issue is the same, just that the description of the issue seemed that there was a working method, but there was support documentation needed.

And this issue is not specifically tied to GCR, but to other registries that are also using the same scheme for authentication.

danielpacak · 2021-12-17T09:17:21Z

I see what you mean now. The title of this issue confused me. So basically we're talking about a bug in parsing passwords from imagePullSecrets that may contain the colon (:) characters. If we fix the parsing logic would that resolve your issue @jnauska ?

jnauska · 2021-12-17T11:10:13Z

I tested with trivy (0.21.2) binary that it works:

export TRIVY_USERNAME="_json_key_base64"
export TRIVY_PASSWORD="$(cat /home/jnauska/Documents/gcp/registry_key | base64 -w0)"

jnauska@ububox:~/Downloads$ trivy --debug image REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG
2021-12-17T12:52:51.213+0200	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-12-17T12:52:51.215+0200	DEBUG	cache dir:  /home/jnauska/.cache/trivy
....
....
....
2021-12-17T12:52:54.777+0200	DEBUG	Detecting library vulnerabilities, type: node-pkg, path: 

REGION-docker.pkg.dev/PROJECT/FOLDER/IMAGE:IMAGETAG (debian 10.10)
================================================================================================================================
Total: 47 (UNKNOWN: 0, LOW: 24, MEDIUM: 12, HIGH: 9, CRITICAL: 2)

jnauska · 2021-12-17T11:34:47Z

We got it working with the following ImagePullSecret setup:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
    }
  }
}

So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.

So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.

Our ImagePullSecret currently has both options:

{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
      "username": "_json_key_base64"
      "password": "SA_KEY(encoded)"
    }
  }
}

danielpacak · 2021-12-17T12:15:02Z

We got it working with the following ImagePullSecret setup:
{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
    }
  }
}
So the issue is actually that starboard doesn't support username:password, but expects the username:password to be within auth.

So I guess this can be closed, though it would be nice to support username:password aswell w/o auth. And maybe add documentation that SA KEY's can be used with this kind of ImagePullSecret.

Our ImagePullSecret currently has both options:
{
  "auths": {
    "repository_name": {
      "auth": "_json_key_base64:SA_KEY(encoded)"
      "username": "_json_key_base64"
      "password": "SA_KEY(encoded)"
    }
  }
}

Interesting. We've discovered the same limitation this week and reported in #855 . To confirm my understanding, will #855 solve your problem? /cc @deven0t

Resolves: #855 Resolves: #857 Signed-off-by: Daniel Pacak <[email protected]>

dgdevops · 2022-02-03T11:58:07Z

Hey @jnauska,
Thank for sharing this workaround.
I tried using with with the starboard-operator version 0.14.1, however I keep on getting the following errors:
{"level":"error","ts":1643803264.6358004,"logger":"controller.replicaset","msg":"Reconciler error","reconciler group":"apps","reconciler kind":"ReplicaSet","name":"nginx-deployment-5c59b4886f","namespace":"web","error":"reading .dockerconfigjson field of \"web/gcr-registry\" secret: expected username and password concatenated with a colon (:)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

The gcr-registry secret's content looks like this:

{
  "auths": {
    "gcr_repository_name_1": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    },
    "gcr_repository_name_2": {
      "auth": "_json_key_base64:SA_KEY(b64 encoded)",
      "username": "_json_key_base64(cleartext)",
      "password": "SA_KEY(b64 encoded)"
    }
  }
}

Any suggestions?

jnauska · 2022-02-03T12:34:14Z

Remove the username and password entries from the auths, and just rely on the auth

dgdevops · 2022-02-03T14:11:01Z

Hey @jnauska,
Thank you for your response.
By default the gcr-registry secret only has the auth parameter set and it looks like this in practise:

{
  "auths": {
    "gcr_repository_name_1": {
      "auth": "_json_key:SA_KEY(b64 encoded)"
    },
    "gcr_repository_name_2": {
      "auth": "_json_key:SA_KEY(b64 encoded)"
    }
  }
}

I added the username & password as a test, with the secret shared above I still get the same errors.

jnauska · 2022-02-03T16:09:12Z

I think you need to double encode that, so like:

"auth": "base64(_json_key_base64:base64(SA_KEY))"

dgdevops · 2022-02-04T12:01:27Z

Hello @jnauska,
Thank you for your response.
I did a few tests to get closer to the solution:

base64:(_json_key:SA_KEY) - Image pull successful, "Reconciler error" messages in the Operator
cleartext:(_json_key:SA_KEY) - Image pull successful, "Reconciler error" messages in the Operator
base64:(_json_key_base64:base64(SA_KEY)) - Image pull fails, Operator gives the following error:

{"level":"error","ts":1643975080.345095,"logger":"reconciler.vulnerabilityreport","msg":"Scan job container","job":"security/scan-vulnerabilityreport-75d5f7b9d5","container":"alpine","status.reason":"Error","status.message":"2022-02-04T11:44:39.689Z\t\u001b[31mFATAL\u001b[0m\tscan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:\n\t* unable to inspect the image (XY.gcr.io/<PATH>/alpine:3.15.0): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* GET https://XY.gcr.io/v2/token?scope=repository%3A<project_name>%2F<repo_name>%2F<repo_dir>%3Apull&service=XY.gcr.io: UNAUTHORIZED: Not Authorized.\n\n\n","stacktrace":"github.com/aquasecurity/starboard/pkg/operator/controller.(*VulnerabilityReportReconciler).reconcileJobs.func1\n\t/home/runner/work/starboard/starboard/pkg/operator/controller/vulnerabilityreport.go:320\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

The scan job secrets have the following content (when testing option 3):
alpine.password: base64:SA_KEY
alpine.username: _json_key_base64

Note: The image pull secret contains auths.repository_name.auth field only (without username or password specified)

dgdevops · 2022-02-09T11:33:03Z

@jnauska & @danielpacak,
Do you have any suggestions please?

jessequinn · 2022-04-10T14:21:52Z

@dgdevops so based on your comment the solution provided by @jnauska does not work? I tried your 3rd option and also image pull failed for me.

dgdevops · 2022-04-10T14:29:32Z

Hello @jessequinn,
None of the above mentioned workarounds/solutions worked for me unfortunately.

jessequinn · 2022-04-10T18:50:43Z

The following looks like HOW it should work. Basically someone would need to update the plugin to place an empty username and add a new ENV VAR GOOGLE_APPLICATION_CREDENTIALS. I may try to make a PR for this. I tested the docker example given in that PR. It works.

jessequinn · 2022-04-11T13:12:51Z

ok. i have played with the code. Actually i think the problem could be fixed quite easily.

docker/config.go contains the following:

func decodeAuths(auths map[string]Auth) (map[string]Auth, error) {
	decodedAuths := make(map[string]Auth)
	for server, entry := range auths {
		if entry == (Auth{}) {
			continue
		}

		if strings.TrimSpace(string(entry.Auth)) == "" {
			decodedAuths[server] = Auth{
				Username: entry.Username,
				Password: entry.Password,
			}
			continue
		}

		// TODO: issue decoding GCR auth
		username, password, err := entry.Auth.Decode()
		if err != nil {
			return nil, err
		}

		decodedAuths[server] = Auth{
			Auth:     entry.Auth,
			Username: username,
			Password: password,
		}

	}
	return decodedAuths, nil
}

now the Decode() method using SplitN rather than Split removes the issue:

func (v *BasicAuth) Decode() (string, string, error) {
	bytes, err := base64.StdEncoding.DecodeString(string(*v))
	if err != nil {
		return "", "", err
	}
	split := strings.SplitN(string(bytes), ":", 2)
	fmt.Println("split", len(split), split[0], split[1])
	if len(split) != 2 {
		return "", "", fmt.Errorf("expected username and password concatenated with a colon (:)")
	}
	return split[0], split[1], nil
}

I tried testing this through Kind as per the contribution guidelines; however, modifications to the config.go DO NOT APPEAR TO APPLY to the docker images built. Any idea? @danielpacak

Added a PR for the SplitN #1126

danielpacak added the ⏳ additional info required Additional information required to close an issue label Dec 17, 2021

danielpacak mentioned this issue Dec 17, 2021

fix: parse registry credentials when auth is blank #859

Merged

danielpacak self-assigned this Dec 17, 2021

danielpacak removed the ⏳ additional info required Additional information required to close an issue label Dec 17, 2021

danielpacak closed this as completed in #859 Dec 17, 2021

danielpacak added a commit that referenced this issue Dec 17, 2021

fix: parse registry credentials when the auth property is blank (#859)

7687ef9

Resolves: #855 Resolves: #857 Signed-off-by: Daniel Pacak <[email protected]>

danielpacak added this to the Release v0.13.2 milestone Dec 17, 2021

dgdevops mentioned this issue Feb 2, 2022

"Reconciler error" when scanning docker image from GCR private repository #948

Closed

jessequinn mentioned this issue Apr 11, 2022

Modified Decode() to use SplitN rather than Split #1126

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Service Account Keys defined in ImagePullSecrets #857

Support Service Account Keys defined in ImagePullSecrets #857

jnauska commented Dec 16, 2021 •

edited

Loading

danielpacak commented Dec 17, 2021

jnauska commented Dec 17, 2021 •

edited

Loading

danielpacak commented Dec 17, 2021

jnauska commented Dec 17, 2021

jnauska commented Dec 17, 2021

danielpacak commented Dec 17, 2021 •

edited

Loading

dgdevops commented Feb 3, 2022

jnauska commented Feb 3, 2022 •

edited

Loading

dgdevops commented Feb 3, 2022 •

edited

Loading

jnauska commented Feb 3, 2022

dgdevops commented Feb 4, 2022 •

edited

Loading

dgdevops commented Feb 9, 2022

jessequinn commented Apr 10, 2022

dgdevops commented Apr 10, 2022

jessequinn commented Apr 10, 2022

jessequinn commented Apr 11, 2022 •

edited

Loading

Support Service Account Keys defined in ImagePullSecrets #857

Support Service Account Keys defined in ImagePullSecrets #857

Comments

jnauska commented Dec 16, 2021 • edited Loading

danielpacak commented Dec 17, 2021

jnauska commented Dec 17, 2021 • edited Loading

danielpacak commented Dec 17, 2021

jnauska commented Dec 17, 2021

jnauska commented Dec 17, 2021

danielpacak commented Dec 17, 2021 • edited Loading

dgdevops commented Feb 3, 2022

jnauska commented Feb 3, 2022 • edited Loading

dgdevops commented Feb 3, 2022 • edited Loading

jnauska commented Feb 3, 2022

dgdevops commented Feb 4, 2022 • edited Loading

dgdevops commented Feb 9, 2022

jessequinn commented Apr 10, 2022

dgdevops commented Apr 10, 2022

jessequinn commented Apr 10, 2022

jessequinn commented Apr 11, 2022 • edited Loading

jnauska commented Dec 16, 2021 •

edited

Loading

jnauska commented Dec 17, 2021 •

edited

Loading

danielpacak commented Dec 17, 2021 •

edited

Loading

jnauska commented Feb 3, 2022 •

edited

Loading

dgdevops commented Feb 3, 2022 •

edited

Loading

dgdevops commented Feb 4, 2022 •

edited

Loading

jessequinn commented Apr 11, 2022 •

edited

Loading