With 0.43.2 push to Docker Hub is not working any more

[ilgrosso]

Description

The Jenkins job which pushes Apache Syncope images to Docker Hub started failing after upgrading the docker-maven-plugin from 0.43.0 to 0.43.2.

Sample job output: https://ci-builds.apache.org./job/Syncope/job/Syncope-3_0_X-deploy/203/console

DockerHub credentials are stored under ~/.m2/settings.xml

Info

• docker-maven-plugin version : 0.43.2
• Maven version (mvn -v) : 3.6.3

• Docker version :
• If it's a bug, how to reproduce :
• If it's a feature request, what is your use case :
• Sample project : [GitHub Clone URL]

[rohanKanojia]

@ilgrosso : Hello, Could you please provide more details? What error are you getting? What is your plugin configuration?

[ilgrosso]

@rohanKanojia the reported error is

DOCKER> #44 ERROR: failed to push apache/syncope:3.0.5-SNAPSHOT: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

The plugin configuration can be found at https://github.com/apache/syncope/blob/3_0_X/docker/core/pom.xml#L237-L273

[rohanKanojia]

@ilgrosso : Do you have docker credentials in repository in this format?

  <servers>
    <server>
      <id>docker.io</id>
      <username>username</username>
      <password>secret</password>  
    </server>
  </servers>

[ilgrosso]

@rohanKanojia I've asked the Jenkins admin and will report here, thank you

[rohanKanojia]

@ilgrosso : Thanks, I'm just trying to check what could have broken in previous release. We did introduce one change related to auth #1693

[ilgrosso]

@rohanKanojia this is their response:

Yes they are exactly like that
Or it would not have worked all this time I imagine

[rohanKanojia]

Thanks for clarifying! Is this bug seen only on v0.43.2 or also on v0.43.1?

[ilgrosso]

Unfortunately I haven't tried with 0.43.1, just 0.43.0 (working) and 0.43.2 (not working)

[mprins]

Also running into this problem with ghcr.io and 0.43.2, this setup was working with 0.43.0 and earlier

Maven settings: https://github.com/B3Partners/tailormap-api/blob/7126ebe39a2c3470a7b247cbf91ef4a01a29c675/.github/maven-settings.xml#L7-L17

pom/config: https://github.com/B3Partners/tailormap-api/blob/7126ebe39a2c3470a7b247cbf91ef4a01a29c675/pom.xml#L1148-L1198

credentials are set from secrets as env variables in: https://github.com/B3Partners/tailormap-api/blob/7126ebe39a2c3470a7b247cbf91ef4a01a29c675/.github/workflows/ubuntu-maven.yml#L168-L178

typical build log:

[INFO] DOCKER> #14 exporting to image
[INFO] DOCKER> #14 exporting layers
[INFO] DOCKER> #14 exporting layers 4.4s done
[INFO] DOCKER> #14 exporting manifest sha256:685cf37b6a2bcbf52f63925608334cf2a59a1a9346f8957ba15c5026944e13c0 done
[INFO] DOCKER> #14 exporting config sha256:0482d55d2176e7e51e1378220a96626bacdcb62be88744dc2130f7a8d2ce06df done
[INFO] DOCKER> #14 exporting manifest sha256:5b17d756175ab7e8136cbcf5f83d03bd3b2b49633630fb3ba9ba3f9bc19e5937 done
[INFO] DOCKER> #14 exporting config sha256:34af804fd992fdb8d0360bcb5cc76f9bade06f7ea450d13730055c4726d93c16 done
[INFO] DOCKER> #14 exporting manifest list sha256:54e915e59920901da68737bd99698f21df4f484f5cc3759a16551084bea035dd done
[INFO] DOCKER> #14 pushing layers
[INFO] DOCKER> #14 pushing layers 0.3s done
[INFO] DOCKER> #14 ERROR: failed to push ghcr.io/b3partners/tailormap-api:snapshot: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Ab3partners%2Ftailormap-api%3Apull%2Cpush&service=ghcr.io: 403 Forbidden
[INFO] DOCKER> ------
[INFO] DOCKER>  > exporting to image:
[INFO] DOCKER> ------
[INFO] DOCKER> ERROR: failed to solve: failed to push ghcr.io/b3partners/tailormap-api:snapshot: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Ab3partners%2Ftailormap-api%3Apull%2Cpush&service=ghcr.io: 403 Forbidden
Error:  DOCKER> Error status (1) when building
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:04 min (Wall Clock)
[INFO] Finished at: 2023-08-08T09:50:24Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal io.fabric8:docker-maven-plugin:0.43.2:push (docker-push) on project tailormap-api: Error status (1) when building -> [Help 1]

[rombert]

Same here for Apache Sling, Docker pushes have started failing for us since we moved from 0.43.0 to 0.43.2 . See for instance https://github.com/apache/sling-org-apache-sling-starter/actions/runs/5785078688/job/15677027953#step:7:3989

Failed to execute goal io.fabric8:docker-maven-plugin:0.43.2:build (build-docker-image) on project org.apache.sling.starter: Execution build-docker-image of goal io.fabric8:docker-maven-plugin:0.43.2:build failed: Cannot invoke "String.equalsIgnoreCase(String)" because "configuredRegistry" is null

You can see the pom.xml at https://github.com/apache/sling-org-apache-sling-starter/blob/37a32dde3a3fe9f6b1bad206973909da44ccd875/pom.xml#L267-L357 and the way we trigger the build from GitHub actions at https://github.com/apache/sling-org-apache-sling-starter/blob/37a32dde3a3fe9f6b1bad206973909da44ccd875/.github/workflows/docker-push.yml#L37-L48

Hope this helps.

[rohanKanojia]

@ilgrosso @mprins: Hello, I'm trying to debug this issue. docker:push goal prints the finally executed buildx command in maven logs. Could you guys please provide that command ran with v0.43.0 (working) and v0.43.2 (not working)? It can help in debugging what's portion of buildx command is generated differently in this version.

[ilgrosso]

With 0.43.2:

docker buildx build --progress=plain --builder maven --platform linux/amd64,linux/arm64 --tag apache/syncope:3.0.5-SNAPSHOT --file=/home/jenkins/jenkins-agent/workspace/Syncope/Syncope-3_0_X-deploy/docker/core/target/docker/apache/syncope/3.0.5-SNAPSHOT/tmp/docker-build/Dockerfile /home/jenkins/jenkins-agent/workspace/Syncope/Syncope-3_0_X-deploy/docker/core/target/docker/apache/syncope/3.0.5-SNAPSHOT/tmp/docker-build --push

With 0.43.0:

docker --config /home/jenkins/workspace/Syncope/Syncope-3_0_X-deploy/docker/core/target/docker/apache/syncope/3.0.5-SNAPSHOT/docker buildx build --progress=plain --builder maven --platform linux/amd64,linux/arm64 --tag apache/syncope:3.0.5-SNAPSHOT --file=/home/jenkins/workspace/Syncope/Syncope-3_0_X-deploy/docker/core/target/docker/apache/syncope/3.0.5-SNAPSHOT/tmp/docker-build/Dockerfile /home/jenkins/workspace/Syncope/Syncope-3_0_X-deploy/docker/core/target/docker/apache/syncope/3.0.5-SNAPSHOT/tmp/docker-build --push

[rohanKanojia]

@ilgrosso @mprins @rombert : I've released 0.43.3 . Could you please try again on this version and see if you're still facing this issue?

[rombert]

@rohanKanojia - worked just fine for me, thanks! https://github.com/apache/sling-org-apache-sling-starter/actions/runs/5849388029/job/15857660731

[ilgrosso]

I confirm that 0.43.3 is working fine for me as well, thank you very much @rohanKanojia

[mprins]

0.43.3 works for me