Trying to push Docker image to AWS ECR from within Kaniko fails with 'Invalid JSON syntax' on ImageManifest

[SCaveAtWork]

Actual behavior
Attempting to push to AWS ECR from within Kaniko results in the following error:

error pushing image: failed to push to destination <OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/authoring:ryan: PUT https://<OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/v2/authoring/manifests/ryan: UNSUPPORTED: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

Expected behavior
The push should succeed without throwing an invalid JSON error.

To Reproduce
Execute the following to start running inside of Kaniko:

docker run --rm -it -v $PWD:/build -w /build --entrypoint /bin/sh gcr.io/kaniko-project/executor:v1.16.0-debug

Use the following Dockerfile:

FROM public.ecr.aws/debian/debian:10
RUN echo "Hello, World" > hello
ENTRYPOINT ["cat", "hello"]

And execute the following script, filling in the missing secret values:

#!/busybox/env sh

set -ex

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_DEFAULT_REGION="us-gov-west-1"

echo "{\"credHelpers\":{\"$REGISTRY_URL\":\"ecr-login\"}}" > $DOCKER_CONFIG/config.json

/kaniko/executor --dockerfile /build/Dockerfile --destination <OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/authoring:ryan

The output we are receiving, from this simple example as well as more complex examples, is the following:

+ /kaniko/executor --dockerfile /build/Dockerfile --destination <OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/authoring:ryan
INFO[0002] Retrieving image manifest public.ecr.aws/debian/debian:10 
INFO[0002] Retrieving image public.ecr.aws/debian/debian:10 from registry public.ecr.aws 
INFO[0003] Built cross stage deps: map[]                
INFO[0003] Retrieving image manifest public.ecr.aws/debian/debian:10 
INFO[0003] Returning cached image manifest              
INFO[0003] Executing 0 build triggers                   
INFO[0003] Building stage 'public.ecr.aws/debian/debian:10' [idx: '0', base-idx: '-1'] 
INFO[0003] Unpacking rootfs as cmd RUN echo "Hello, World" > hello requires it. 
INFO[0026] RUN echo "Hello, World" > hello              
INFO[0026] Initializing snapshotter ...                 
INFO[0026] Taking snapshot of full filesystem...        
INFO[0027] Cmd: /bin/sh                                 
INFO[0027] Args: [-c echo "Hello, World" > hello]       
INFO[0027] Running: [/bin/sh -c echo "Hello, World" > hello] 
INFO[0027] Taking snapshot of full filesystem...        
INFO[0028] ENTRYPOINT ["cat", "hello"]                  
INFO[0028] Pushing image to <OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/authoring:ryan 
error pushing image: failed to push to destination <OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/authoring:ryan: PUT https://<OUR_CUSTOMER_ID>.dkr.ecr.us-gov-west-1.amazonaws.com/v2/authoring/manifests/ryan: UNSUPPORTED: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

Additional Information

• Files
  kaniko_invalidJSONSyntax.zip
• Dockerfile
  Dockerfile provided above.
• Build Context
  Build files provided above.
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:v1.16.0-debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[JeromeJu]

Thanks for filing this @SCaveAtWork , could you provide the output when running with docker that succeeds with this Dockerfile? It looks like that the ImageManifest is specifically unsupported by the registry, I am curious in which way kaniko could be different handling this compared with docker.

[RLovelett]

@JeromeJu is there a way to extract the ImageManifest? If so we could provide the one that Kaniko generates and Docker. It is beyond my skills (and I assume @SCaveAtWork) to know where to look to find the ImageManifest file.

[SCaveAtWork]

...could you provide the output when running with docker that succeeds with this Dockerfile?

[+] Building 2.4s (6/6) FINISHED                                                                                                                                                                                                                            
 => [internal] load .dockerignore                                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.83kB                                                                                                                                                                                                                 0.0s
 => [internal] load metadata for public.ecr.aws/debian/debian:10                                                                                                                                                                                       1.9s
 => CACHED [1/2] FROM public.ecr.aws/debian/debian:10@sha256:c0191cd2dd75b6a1165a1777baf6404b684ca86d976bcacf457c0113ac2b8780                                                                                                                          0.0s
 => [2/2] RUN echo "Hello, World" > hello                                                                                                                                                                                                              0.4s
 => exporting to image                                                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                                                0.0s
 => => writing image sha256:a41d6a55f309e40e846f81ee71e53a95679be8c91f87975f275d4d7e1ab3985b           

[JeromeJu]

Thanks @SCaveAtWork for the inputs and @RLovelett .

There doesn't seem to be a nice way to extract the manifest at the moment for kaniko. But we could add debug logs at https://github.com/GoogleContainerTools/kaniko/blob/main/pkg/image/image_util.go if we would like. Would you like to spin up PR for us to make this a feature that goes into the next release?

[dirk39]

Hello @SCaveAtWork !
We had the same issue with the same ECR Gallery repo.
Looking at the gallery page, they suggest to use temporarily not this repo but this one: https://gallery.ecr.aws/docker/library/debian.
We have replaced the image source from public.ecr.aws/debian/debian:bookworm-slim to public.ecr.aws/docker/library/debian:bookworm-slim, which works.

[SCaveAtWork]

We have replaced the image source from public.ecr.aws/debian/debian:bookworm-slim to public.ecr.aws/docker/library/debian:bookworm-slim, which works.

Thank you, this worked!! All we had to do is change the first line of our Dockerfile as described and we are off to the races!

[RLovelett]

As @SCaveAtWork said, this is working for us. On that level, we can just accept the new base container and move on. Should we close this issue?

I would add, that the whole thing is a bit of a head scratcher and something does not seem quite right to me.

The gallery page says to stop using it and to use the library link instead and cites aws/containers-roadmap#2023 as the reason.

However, the GitHub issue is resolved. Doesn't that imply that the image should be good now? Further more, we can push using Docker if we use the "broken" base image but cannot if we use Kaniko. I feel like there is still something different here.

In the end, all of this is way beyond my understanding level of containers and container registries (not to mention Docker/Kaniko). There is just something about this that does not seem "correct".