Not able to scan image in Google Artifact Registry #783

manishjagtap · 2020-12-22T13:43:11Z

Description

trivy version: 0.15.0

I have two GCP projects within our organization:

trivyProject: Where aquasec/trivy:0.15.0 container is run
targetProject: Has Google Artifact Registry that hosts container images to scan.

From aquasec/trivy:0.15.0 container running in trivyProject project, I want to scan images in another project targetProject.
I have created a Service Account in trivyProject and have assigned it artifactregistry.reader as follows. I have already checked that:

Service account has artifactregistry.reader on the target registry
Service Account private key JSON file (Plain-text - not encoded) is available in GOOGLE_APPLICATION_CREDENTIALS file.

  gcloud artifacts repositories add-iam-policy-binding myRegistry \
    --project=targetProject \
    --location=us \
    --member=serviceAccount:[email protected] \
    --role=roles/artifactregistry.reader

Case#1:

In the official aquasec/trivy:0.15.0 container running in trivyProject:

  export GOOGLE_APPLICATION_CREDENTIALS="$SA_KEY_JSON"
  export TRIVY_USERNAME=""

  trivy -d image us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0

What did you expect to happen?

Expected trivy to scan the image.

What happened instead?

Got permission denied error

Output of run with `-debug`:

Case#1 Logs:

+ trivy --debug image --timeout 5m us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0
2020-12-22T13:27:30.026Z    DEBUG   Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2020-12-22T13:27:30.090Z    DEBUG   cache dir:  /root/.cache/trivy
2020-12-22T13:27:30.090Z    DEBUG   DB update was skipped because DB is the latest
2020-12-22T13:27:30.090Z    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-12-22 12:23:57.143363872 +0000 UTC, NextUpdate: 2020-12-23 00:23:57.143363472 +0000 UTC, DownloadedAt: 2020-12-22 13:27:27.320023833 +0000 UTC
2020-12-22T13:27:30.353Z    FATAL   unable to initialize a scanner:
    github.com/aquasecurity/trivy/internal/artifact.run
        /home/circleci/project/internal/artifact/run.go:73
  - unable to initialize a docker scanner:
    github.com/aquasecurity/trivy/internal/artifact.dockerScanner
        /home/circleci/project/internal/artifact/image.go:28
  - 2 errors occurred:
    * unable to inspect the image (us-docker.pkg.dev/targetProject/myRepi/myImage:1.0.0): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    * GET https://us-docker.pkg.dev/v2/token?scope=repository%3AtargetProject%2FmyRegistry%2FmyImage%3Apull&service=us-docker.pkg.dev: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/targetProject/locations/us/repositories/myReg" (or it may not exist)

Output of `trivy -v`:

$ trivy -v
Version: 0.15.0

Additional details (base image name, container registry info...):

Image to be scanned - equivalent to us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0 - is in Google Artifact Resgistry.

Any pointers on why I may be getting permission denied error?

The text was updated successfully, but these errors were encountered:

knqyf263 · 2020-12-23T06:08:47Z

Is $SA_KEY_JSON a path to the credential JSON file? The following error looks like you're just missing the permission.

DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myProject/locations/us/repositories/myReg" (or it may not exist)

manishjagtap · 2020-12-24T13:23:40Z

SA_KEY_JSON points to the private key json of the Service Account. Pruned contents are:

{
  "type": "service_account",
  "project_id": "trivyProject",
  "private_key_id": "55c2xxx6a",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBAD...eqDmc=\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1112xx27",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/registry-reader%40trivyProject.iam.gserviceaccount.com"
}

I have given permissions using gcloud artifacts repositories add-iam-policy-binding... command as seen above. Ref: https://cloud.google.com/artifact-registry/docs/access-control#grant-repo

In GKE dashboard, I have confirmed that this Service Account - [email protected] - is asigned the role roles/artifactregistry.reader that has artifactregistry.repositories.downloadArtifacts permission on targetProject/myRegistry registry.

manishjagtap · 2020-12-24T14:59:31Z

Case#2:

In another custom container with gcloud and trivy binaries, both, if the following commands are executed, the image us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0 is successfully scanned by trivy. Note that the SA key file used in the above case (original bug report) and the following case#2 below is the same - but only case#2 works.

Case#2 code:

  SA_KEY="sa-key.json"    # With private_key as shown above

  gcloud auth activate-service-account --key-file ${SA_KEY}
  gcloud auth configure-docker us-docker.pkg.dev,gcr.io        

  export GOOGLE_APPLICATION_CREDENTIALS="$SA_KEY"
  export TRIVY_USERNAME=""

  $TRIVY_BIN --debug image us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0

Logs of above code:

00:00:56.650  --> Activate registry-reader Service Account
00:00:56.650  + gcloud auth activate-service-account --key-file /opt/registry-reader-sa/key.json
00:00:59.204  Activated service account credentials for: [[email protected]]
00:00:59.466  + gcloud auth configure-docker us-docker.pkg.dev,gcr.io
00:01:00.044  Adding credentials for: us-docker.pkg.dev,gcr.io
00:01:00.044  After update, the following will be written to your Docker config file
00:01:00.044   located at [/home/prod/.docker/config.json]:
00:01:00.044   {
00:01:00.044    "credHelpers": {
00:01:00.044      "us-docker.pkg.dev": "gcloud",
00:01:00.044      "gcr.io": "gcloud"
00:01:00.044    }
00:01:00.044  }
00:01:00.044  
00:01:00.305  Do you want to continue (Y/n)?  
00:01:00.305  Docker configuration file updated.
...
00:01:04.925  + /home/jenkins/agent/workspace/cve-scanner/trivy --debug image us-docker.pkg.dev/myProject/myRegistry/myImage:1.0.0
00:01:04.925  2020-12-24T13:00:14.474Z	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
00:01:05.187  2020-12-24T13:00:14.588Z	DEBUG	cache dir:  /home/prod/.cache/trivy
00:01:05.187  2020-12-24T13:00:14.588Z	DEBUG	DB update was skipped because DB is the latest
00:01:05.187  2020-12-24T13:00:14.588Z	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2020-12-24 12:24:02.943630066 +0000 UTC, NextUpdate: 2020-12-25 00:24:02.943629666 +0000 UTC, DownloadedAt: 2020-12-24 13:00:05.845228399 +0000 UTC
00:01:06.575  2020-12-24T13:00:15.828Z	DEBUG	Vulnerability type:  [os library]
00:01:09.123  2020-12-24T13:00:18.738Z	DEBUG	Artifact ID: sha256:58d3edxxx296
00:01:09.123  2020-12-24T13:00:18.738Z	DEBUG	Blob IDs: [sha256:50644c xxx 7246f7b099]
00:01:09.123  2020-12-24T13:00:18.739Z	INFO	Detecting Alpine vulnerabilities...
00:01:09.123  2020-12-24T13:00:18.739Z	DEBUG	alpine: os version: 3.12

So, I guess I need to run the equivalent of $ gcloud auth configure-docker us-docker.pkg.dev in the official aquasec/trivy container (run some CLI or modify some config file).

tisc0 · 2020-12-24T16:00:55Z

Hi @manishjagtap ,
I've had the same issue a few days ago since we decided to use the fresh Google Artifact registry.

When I force the content of my config.json for docker, everything is fine (that's your equivalent of gcloud auth configure-docker).
So this is what I do in my gitlab-cy.yaml to be sure it works (since I never had the chance to make work export TRIVY_USERNAME="") :

  before_script: # or script:
    - |
      AUTH_ARTIFACTORY=$(echo -n "${ARTIFACTORY_USER}:${ARTIFACTORY_PASSWORD} | base64 | tr -d '\n')
      mkdir -p ~/.docker && \
      cat <<EOF > ~/.docker/config.json
      {
        "auths": {
          "$ARTIFACTORY_REGISTRY": {
            "auth": "${AUTH_ARTIFACTORY}"
          }
        },
        "credHelpers": {
          "eu.gcr.io": "gcloud",
          "europe-west1-docker.pkg.dev": "gcloud",
          "whatever_you_need": "gcloud"
        }
      }
      EOF

(this snippet embed Jfrog artifactory auth. example, not needed in your case)

manishjagtap · 2020-12-24T19:44:45Z

Thanks a lot @tisc0

The way I am running the trivy command is like follows:
In a Jenkins pipeline, using Kubernetes plugin, I start a dynamic pod with two containers in it:
Container-1: Custom container - with Google Cloud SDK (gcloud and stuff) installed
Container-2: aquasec/trivy - the trivy command is run inside this container as follows:

Pipeline goes something like:

    stage("Scan Images") {
      steps {
        script {
          container('trivy') {.  // Run following commands inside the 'trivy' container of the pod
            sh """
              export GOOGLE_APPLICATION_CREDENTIALS="$SA_KEY_JSON"
              export TRIVY_USERNAME=""

              trivy -d image us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0
            """
          }
        }
      }
    }

After force creating ~/.docker/config.json, it gives the following error:

* error getting credentials - err: exec: "docker-credential-gcloud": executable file not found in $PATH, out: ``

because the official aquasec/trivy:0.15.0 container does not have the docker-credential-gcloud file installed.

Note that Case#2 works fine because gcloud and trivy binary are in the same custom container. The custom container had the docker-credential-gcloud util installed as part of Google Cloud SDK and the above 2 gcloud commands created the ~/.docker/config.json, and therefore trivy scan worked fine.

I was expecting the standalone, official aquasec/trivy trivy container to work if I only do the following:

  export GOOGLE_APPLICATION_CREDENTIALS="$SA_KEY"
  export TRIVY_USERNAME=""

manishjagtap · 2020-12-24T20:35:05Z

Using Standalone Credential Helper for authenticating with Google Artifact Registry worked for me. Using some hack, copied the docker-credential-gcr bin into the trivy container.

The config.json got created by running docker-credential-gcr command as:follows

00:00:28.771  + docker-credential-gcr configure-docker '--registries=us-docker.pkg.dev,gcr.io'
00:00:28.771  /root/.docker/config.json configured to use this credential helper for GCR registries
00:00:28.771  + cat /root/.docker/config.json
00:00:28.771  {
00:00:28.771  	"auths": {},
00:00:28.771  	"credHelpers": {
00:00:28.771  		"gcr.io": "gcr",
00:00:28.771  		"us-docker.pkg.dev": "gcr"
00:00:28.771  	}
00:00:28.779  }

So, the final working code is:

export GOOGLE_APPLICATION_CREDENTIALS="$SA_KEY_JSON"
export TRIVY_USERNAME=""

docker-credential-gcr configure-docker --registries=us-docker.pkg.dev,gcr.io
trivy image us-docker.pkg.dev/targetProject/myRegistry/myImage:1.0.0

That said, is this the right way? I am pretty sure I am not the first person to face this issue. :)

manishjagtap · 2020-12-25T09:16:30Z

Ref Links:

https://cloud.google.com/docs/authentication/production
https://cloud.google.com/artifact-registry/docs/docker/authentication#standalone-helper
https://cloud.google.com/container-registry/docs/advanced-authentication

manishjagtap · 2021-01-05T09:21:55Z

From within the trivy container, is there any way to avoid the use of the cred helper?

docker-credential-gcr configure-docker --registries=us-docker.pkg.dev

I am still not convinced that I have to copy the docker-credential-gcr helper util into the official aquasec/trivy container to make this work.
Not sure how others can scan private registries in Google Artifact Registry without having to use cred helpers like docker-credential-gcr.

dinvlad · 2021-01-13T01:16:09Z

It should be possible to auth with docker login using a Service Account key, as documented in
https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key

cat "${SA_KEY_JSON}" | docker login -u _json_key --password-stdin https://us-docker.pkg.dev

manishjagtap · 2021-01-21T11:19:57Z

It should be possible to auth with docker login using a Service Account key, as documented in
https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key
cat "${SA_KEY_JSON}" | docker login -u _json_key --password-stdin https://us-docker.pkg.dev

Official aquasec/trivy container does not have docker util.

github-actions · 2021-03-23T00:13:46Z

This issue is stale because it has been labeled with inactivity.

materemias · 2021-07-08T13:39:47Z

sorry for necrobumping, but I could not find if there is any solution to get authenticated to GCR with a credentials json file... I am trying to run the aquasec/trivy image with GOOGLE_APPLICATION_CREDENTIALS=/gcr.json env var

mykter · 2021-08-10T16:11:12Z

The trivy container doesn't have gcloud, the standalone credential helper, nor docker, but we can manually set up the docker config.json file to contain the necessary authentication information.

In this gitlab job sample GCP_KEY contains the plain json key contents for the service account. Run this prior to invoking trivy in the official trivy container and it works with Google Artifact Registry. You need to replace the registry location with your one.

     # manually reproducing the effect of `docker login -u _json_key --password-stdin < $GOOGLE_APPLICATION_CREDENTIALS`
    - GCR_AUTH=$(echo "_json_key:$GCP_KEY" | base64 | tr -d '[:space:]')
    - |
      mkdir ~/.docker && cat << EOF > ~/.docker/config.json
      {"auths":
        {
          "us-central1-docker.pkg.dev":{"auth":"$GCR_AUTH"}
        }
      }
      EOF

krol3 · 2021-11-24T00:16:10Z

@manishjagtap, are you able to scan the image in Google Artifact Registry? We could add the relevant information to the trivy documentation.

ahsan-raza · 2022-11-18T16:44:38Z

The trivy container doesn't have gcloud, the standalone credential helper, nor docker, but we can manually set up the docker config.json file to contain the necessary authentication information.

In this gitlab job sample GCP_KEY contains the plain json key contents for the service account. Run this prior to invoking trivy in the official trivy container and it works with Google Artifact Registry. You need to replace the registry location with your one.
     # manually reproducing the effect of `docker login -u _json_key --password-stdin < $GOOGLE_APPLICATION_CREDENTIALS`
    - GCR_AUTH=$(echo "_json_key:$GCP_KEY" | base64 | tr -d '[:space:]')
    - |
      mkdir ~/.docker && cat << EOF > ~/.docker/config.json
      {"auths":
        {
          "us-central1-docker.pkg.dev":{"auth":"$GCR_AUTH"}
        }
      }
      EOF

@mykter I am trying to achieve the same with gitlab CI job. It works if I pull the docker image on my Docker for Mac but when I do the same in gitlab-ci.yaml file, it gives me the error of permission denied. My CI stage is as follows:

container_scanning:
  image:
   name: docker.io/aquasec/trivy:latest
   entrypoint: [""]
  allow_failure: false
  only:
    - main
    - develop
    - integration
    - tags

  stage: trivy_scan_build
  variables:
    # No need to clone the repo, we exclusively work on artifacts.  See
    # https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
    GIT_STRATEGY: none
    # TRIVY_USERNAME: "$CI_REGISTRY_USER"
    # TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
    # TRIVY_AUTH_URL: "$CI_REGISTRY"
    #FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

  before_script:
    - mkdir -p ~/.docker && GCR_AUTH=$(echo "_json_key:$GCP_REPO_KEY" | base64 | tr -d '[:space:]')
    - |
      cat << EOF > ~/.docker/config.json
      {"auths":
        {
          "europe-west3-docker.pkg.dev":{"auth":"$GCR_AUTH"}
        }
      }
      EOF
  script:
    - trivy --version
    # cache cleanup is needed when scanning images with the same tags, it does not remove the database
    - trivy image --clear-cache
    # update vulnerabilities db
    - trivy image --download-db-only --no-progress --cache-dir .trivycache/
    # Builds report and puts it in the default workdir $CI_PROJECT_DIR, so `artifacts:` can take it from there
    - trivy image --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "@/contrib/gitlab.tpl" --output "$CI_PROJECT_DIR/gl-container-scanning-report.json" "$MICROSERVICE_BRANCH_OR_TAG_IMAGE"
    # Prints full report
    - trivy image --exit-code 0 --cache-dir .trivycache/ --no-progress $MICROSERVICE_BRANCH_OR_TAG_IMAGE > "${CI_PROJECT_TITLE}"-microservice-vulnerabilities-trivy.txt
    # to fail on critical vulnerabilities change to --exit-code 1
    - trivy image --exit-code 0 --cache-dir .trivycache/ --severity CRITICAL --no-progress $MICROSERVICE_BRANCH_OR_TAG_IMAGE   > "${CI_PROJECT_TITLE}"-microservice-vulnerabilities-critical-failed-trivy.txt
  cache:
    paths:
      - .trivycache/
  # Enables https://docs.gitlab.com/ee/user/application_security/container_scanning/ (Container Scanning report is available on GitLab EE Ultimate or GitLab.com Gold)
  artifacts:
    when:                          always
    reports:
      container_scanning:          gl-container-scanning-report.json
    paths:
      - ${CI_PROJECT_TITLE}-microservice-vulnerabilities-trivy.txt
      - ${CI_PROJECT_TITLE}-microservice-vulnerabilities-critical-failed-trivy.txt

mykter · 2022-11-18T16:53:16Z

@ahsan-raza what line triggers the error message? It wouldn't surprise me if the trivy container has changed in some way since I used this workaround.

ahsan-raza · 2022-11-18T22:05:26Z

It breaks on trivy image command. I tried to run the previous docker image but all in vain.

- trivy image --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "@/contrib/gitlab.tpl" --output "$CI_PROJECT_DIR/gl-container-scanning-report.json" "$MICROSERVICE_BRANCH_OR_TAG_IMAGE"

manishjagtap added the kind/bug Categorizes issue or PR as related to a bug. label Dec 22, 2020

manishjagtap mentioned this issue Dec 25, 2020

Issue with Google Cloud Container registry #265

Open

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. label Mar 23, 2021

krol3 added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and will be auto-closed. labels Mar 26, 2021

nobletrout mentioned this issue Nov 30, 2021

docs: provide more information on scanning Google's GCR #1426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to scan image in Google Artifact Registry #783

Not able to scan image in Google Artifact Registry #783

manishjagtap commented Dec 22, 2020 •

edited

Loading

knqyf263 commented Dec 23, 2020

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

tisc0 commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 25, 2020 •

edited

Loading

manishjagtap commented Jan 5, 2021 •

edited

Loading

dinvlad commented Jan 13, 2021 •

edited

Loading

manishjagtap commented Jan 21, 2021

github-actions bot commented Mar 23, 2021

materemias commented Jul 8, 2021

mykter commented Aug 10, 2021

krol3 commented Nov 24, 2021

ahsan-raza commented Nov 18, 2022

mykter commented Nov 18, 2022

ahsan-raza commented Nov 18, 2022

Not able to scan image in Google Artifact Registry #783

Not able to scan image in Google Artifact Registry #783

Comments

manishjagtap commented Dec 22, 2020 • edited Loading

Description

Case#1:

What did you expect to happen?

What happened instead?

Output of run with -debug:

Output of trivy -v:

Additional details (base image name, container registry info...):

knqyf263 commented Dec 23, 2020

manishjagtap commented Dec 24, 2020 • edited Loading

manishjagtap commented Dec 24, 2020 • edited Loading

Case#2:

tisc0 commented Dec 24, 2020 • edited Loading

manishjagtap commented Dec 24, 2020 • edited Loading

manishjagtap commented Dec 24, 2020 • edited Loading

manishjagtap commented Dec 25, 2020 • edited Loading

manishjagtap commented Jan 5, 2021 • edited Loading

dinvlad commented Jan 13, 2021 • edited Loading

manishjagtap commented Jan 21, 2021

github-actions bot commented Mar 23, 2021

materemias commented Jul 8, 2021

mykter commented Aug 10, 2021

krol3 commented Nov 24, 2021

ahsan-raza commented Nov 18, 2022

mykter commented Nov 18, 2022

ahsan-raza commented Nov 18, 2022

manishjagtap commented Dec 22, 2020 •

edited

Loading

Output of run with `-debug`:

Output of `trivy -v`:

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

tisc0 commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 24, 2020 •

edited

Loading

manishjagtap commented Dec 25, 2020 •

edited

Loading

manishjagtap commented Jan 5, 2021 •

edited

Loading

dinvlad commented Jan 13, 2021 •

edited

Loading