Renovate fails to get Docker tags from AWS ECR

[erikkrieg]

What Renovate type are you using?
Renovate CLI

Describe the bug
When trying to get Docker image tags from AWS ECR, the request is rejected because the maxResults exceeds the limits for ECR.

Here you can see the query parameter, n, being set to 10000:

renovate/lib/datasource/docker/index.js

Line 263 in d501790

let url = `${registry}/v2/${repository}/tags/list?n=10000`;

AWS ECR will reject the request with the following:

{
  "errors": [
    {
      "code": "UNSUPPORTED",
      "message": "Invalid parameter at 'maxResults' failed to satisfy constraint: 'Member must have value less than or equal to 1000'"
    }
  ]
}

I found reference to maxResults in AWS docs for ECR that state they accept values less than or equal to 1000.

Expected behaviour
Renovate should be able to integrate with AWS ECR. This value should either be:

1. Configurable.
2. Conditionally set to an appropriate value if AWS ECR is detected as the registry.
3. Hardcoded to a lower value.

Probably the first two options are safer, since behaviour would not change for existing implementations.

[rarkins]

How would we detect? Eg consistent domain suffix?

[erikkrieg]

Hey, @rarkins. Thanks for taking interest :D

Looking at the docs for AWS ECR registries, they say:

The URL for your default registry is https://aws_account_id.dkr.ecr.region.amazonaws.com.

In their example, aws_account_id and region are placeholders. Account ID is an integer, while region is a string.

This suggests the URL can be changed, but I am not sure how common changing the URL is.

If detection is still a desirable approach, then I think the following regex might work:

const exampleRegistry = '123456789.dkr.ecr.us-east-1.amazonaws.com';
const AWS_ECR_REGEX = /\d+\.dkr\.ecr\.[-a-z0-9]+\.amazonaws\.com/;
AWS_ECR_REGEX.test(exampleRegistry); // true

[rarkins]

@erikkrieg thanks for the additional info. Are you able to paste the full error message from the log? i.e. that you already copy pasted from earlier? One other possibility to consider is that we catch that error and retry with 1000 instead of 10000. I'd also like to see the headers that are returned in such a case.

Also, are you able to manually query the URL with a low n value to see what type of pagination response is returned?

[faceleg]

I've run into the same thing, here is the full error I'm getting:

 WARN: Error getting docker image tags (repository=namespace/repo)
       "registry": "https://REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com",
       "dockerRepository": "namespace/private",
       "err": {
         "name": "HTTPError",
         "host": "REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com",
         "hostname": "REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com",
         "method": "GET",
         "path": "/v2/namespace/private/tags/list?n=10000",
         "protocol": "https:",
         "url": "https://REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com/v2/namespace/private/tags/list?n=10000",
         "gotOptions": {
           "path": "/v2/namespace/private/tags/list?n=10000",
           "protocol": "https:",
           "slashes": true,
           "auth": "USER:PASSWORD",
           "host": "REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com",
           "port": null,
           "hostname": "REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com",
           "hash": null,
           "search": "?n=10000",
           "pathname": "/v2/namespace/private/tags/list",
           "href": "https://REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com/v2/namespace/private/tags/list?n=10000",
           "headers": {
             "user-agent": "https://github.com/renovatebot/renovate",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate"
           },
           "hooks": {
             "beforeError": [],
             "init": [],
             "beforeRequest": [],
             "beforeRedirect": [],
             "beforeRetry": [],
             "afterResponse": []
           },
           "retry": {"methods": {}, "statusCodes": {}, "errorCodes": {}},
           "decompress": true,
           "throwHttpErrors": true,
           "followRedirect": true,
           "stream": false,
           "form": false,
           "json": true,
           "cache": false,
           "useElectronNet": false,
           "method": "GET",
           "gotTimeout": {"request": 60000}
         },
         "statusCode": 405,
         "statusMessage": "Method Not Allowed",
         "headers": {
           "content-type": "application/json; charset=utf-8",
           "date": "Wed, 12 Jun 2019 03:48:40 GMT",
           "docker-distribution-api-version": "registry/2.0",
           "content-length": "164",
           "connection": "Close"
         },
         "body": {
           "errors": [
             {
               "code": "UNSUPPORTED",
               "message": "Invalid parameter at 'maxResults' failed to satisfy constraint: 'Member must have value less than or equal to 1000'"
             }
           ]
         },
         "message": "Response code 405 (Method Not Allowed)",
         "stack": "HTTPError: Response code 405 (Method Not Allowed)\n    at EventEmitter.emitter.on (/builds/user/repo/renovate-gitlab/node_modules/got/source/as-promise.js:74:19)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }
¨

[rarkins]

It sounds like we can either:

1. Look for hostname matching dkr.ecr.*.amazonaws.com and if found then change n=10000 to n=1000, or
2. Look for 405 responses, and check if `err.body && err.body.errors && err.body.errors.length && err.body.errors[0].message && err.body.errors[0].message.includes('Member must have value less than or equal to 1000')

[renovate-bot]

🎉 This issue has been resolved in version 19.22.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀