[Docker] Self-Hosted quay enterprise support

[KnisterPeter]

Hej there,

we would like to enable renovate to update our Dockerfiles. We have a self hosted quay enterprise instance.

I've already tried to add hostRules like this:

  hostRules.add({
    hostType: 'docker',
    matchHost: 'quay.company.com',
    token: 'xxxx',
  });
  // or
  hostRules.add({
    hostType: 'docker',
    matchHost: 'quay.company.com',
    username: 'xxxx',
    password: 'xxxx',
  });

Both are not working and only showing unauthorized access.

The token configuration just prints these lines from debug log:

DEBUG: Failed to look up dependency quay.company.com/org/repo (repository=org/repo, packageFile=Dockerfile, dependency=quay.company.com/org/repo)

The username/password variant tries to make a request against:

https://quay.company.com/v2/auth?service=quay.company.com&scope=repository:repo/org:pull

According to the quay docs, quay enterprise does only support v1 APIs:
https://docs.quay.io/api/

There is special handling for public quay, but that excludes enterprise servers.
I think it might work if the check for quay is not only looking for quay.io.

renovate/lib/datasource/docker/index.ts

Lines 106 to 114 in a133bb9

	const isQuay = regEx(/^https:\/\/quay\.io(?::[1-9][0-9]{0,4})?$/i).test(
	registryHost
	);
	let tags: string[] | null;
	if (isQuay) {
	tags = await getTagsQuayRegistry(registryHost, dockerRepository);
	} else {
	tags = await getDockerApiTags(registryHost, dockerRepository);
	}

Would this be worth an issue and a PR?
If so, what is the starting point to extend the isQuay check based on host rules?

[viceice]

It looks like your credentials are not working as expected. We are using the same calls the docker cli would make. So please validate the credentials with the docker cli.

You can also try a curl -sv https://quay.company.com/v2/auth?service=quay.company.com&scope=repository:repo/org:pull and post reponse here, so we can check whats going wrong

[KnisterPeter]

Hmm, the response of that query is this one:

xxx:~/source/ (main)$ curl -sv --user "user:pass" "https://quay.company.com/v2/auth?service=quay.company.com&scope=repository:repo/org:pull"
*   Trying 0.0.0.0:443...
* Connected to quay.company.com (0.0.0.0) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.company.com
*  start date: Jan  4 14:28:40 2021 GMT
*  expire date: Feb  5 14:28:40 2022 GMT
*  subjectAltName: host "quay.company.com" matched cert's "*.company.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign GCC R3 DV TLS CA 2020
*  SSL certificate verify ok.
* Server auth using Basic with user 'user'
> GET /v2/auth?service=quay.company.com&scope=repository:repo/org:pull HTTP/1.1
> Host: quay.company.com
> Authorization: Basic base64pass
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.12.1
< Date: Fri, 21 Jan 2022 09:21:58 GMT
< Content-Type: application/json
< Content-Length: 994
< Connection: close
< Cache-Control: no-cache, no-store, must-revalidate
< X-Frame-Options: DENY
< 
{"token":"valid-token"}
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):

I will re-verify my credentials setup.

[KnisterPeter]

With exactly the same credentials in my hostRules I'll get this response in the logfile:

DEBUG: Unauthorized docker lookup (repository=repo/org)
       "registryHost": "https://quay.company.com",
       "dockerRepository": "repo/base-image"
DEBUG: Response code 401 (UNAUTHORIZED) (repository=repo/org)
       "err": {
         "name": "HTTPError",
         "code": "ERR_NON_2XX_3XX_RESPONSE",
         "timings": {
           "start": 1642757365657,
           "socket": 1642757365658,
           "lookup": 1642757365661,
           "connect": 1642757365720,
           "secureConnect": 1642757365748,
           "upload": 1642757365748,
           "response": 1642757365783,
           "end": 1642757365784,
           "phases": {
             "wait": 1,
             "dns": 3,
             "tcp": 59,
             "tls": 28,
             "request": 0,
             "firstByte": 35,
             "download": 1,
             "total": 127
           }
         },
         "message": "Response code 401 (UNAUTHORIZED)",
         "stack": "HTTPError: Response code 401 (UNAUTHORIZED)\n    at Request.<anonymous> (xxx)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/31.28.3 (https://github.com/renovatebot/renovate)",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://quay.company.com/v2/auth?service=quay.company.com&scope=repository:repo/base-image:pull",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         },
         "response": {
           "statusCode": 401,
           "statusMessage": "UNAUTHORIZED",
           "body": {"error": "Anonymous access is not allowed"},
           "headers": {
             "server": "nginx/1.12.1",
             "date": "Fri, 21 Jan 2022 09:29:25 GMT",
             "content-type": "text/html; charset=utf-8",
             "content-length": "44",
             "connection": "close",
             "access-control-allow-origin": "*",
             "access-control-allow-methods": "HEAD, OPTIONS, GET",
             "access-control-allow-headers": "Authorization, Content-Type",
             "access-control-max-age": "21600"
           },
           "httpVersion": "1.1"
         }
       }
DEBUG: Failed to get authHeaders for getTags lookup (repository=repo/org)
DEBUG: Failed to look up dependency quay.company.com/repo/base-image (repository=repo/org, packageFile=Dockerfile, dependency=quay.company.com/repo/base-image)

[viceice]

maybe you should enable trace loglevel, so you see some more docker auth related log messages. Use a small repo with a single dep or disable all deps and only allow one to reduce the log size

[viceice]

You remove all hostrules here;

  if (repoConfig.hostRules) {
      hostRules.clear();
      repoConfig.hostRules.forEach((rule) => hostRules.add(rule));
      repoConfig.hostRules = [];
    }

hostRules will probably always an empty array.

[KnisterPeter]

Hmm, I'll give it a try without clearing the host rules there.
Not sure why we are doing this... 🙄 😄

[viceice]

You probably need so safe the initial host rules, clear before each repo, add saved rules, add repo rules.

[KnisterPeter]

We do start it with just one repo, we should be fine without clear and restore at all I think.
That code evolved a bit. Maybe that's already the issue.

[KnisterPeter]

Now with trace (and without removed host rules) I'll get these lines:

TRACE: Cache miss (repository=org/repo)
       "namespace": "datasource-docker-tags",
       "key": "https://quay.company.com:org/base"
TRACE: Authorization disabled (repository=org/repo)
       "url": "https://quay.company.com/v2/"
TRACE: got request (repository=org/repo)
       "url": "https://quay.company.com/v2/",
       "options": {
         "method": "get",
         "context": {"hostType": "docker"},
         "hostType": "docker",
         "throwHttpErrors": false,
         "noAuth": true,
         "hooks": {"beforeRedirect": ["[function]"]},
         "headers": {
           "user-agent": "RenovateBot/31.28.3 (https://github.com/renovatebot/renovate)"
         },
         "timeout": 60000
       }

Not sure why the auth is disabled, since I've added a host-rule to state that for that host username and password are given.

[KnisterPeter]

Oh, I do test with dry-run maybe that is disabling the auth?

[viceice]

The /v2 request is expected to be without auth, we do this to detect auth header. You should see another request to your auth provider after this with auth header.

[KnisterPeter]

Okay, I think the auth is removed intentionally:

renovate/lib/datasource/docker/common.ts

Lines 65 to 68 in 3b65140

	const apiCheckResponse = await http.get(apiCheckUrl, {
	throwHttpErrors: false,
	noAuth: true,
	});

But that leads to my problem I guess.

[viceice]

Nope, there should be more requests after that. It's expected that this returns a www-authenticate header and a 401, which should be catched silently

[KnisterPeter]

Forget my answer. I'll dig a bit deeper. That line is just checking if auth is required.

[KnisterPeter]

So, after some more debugging we are ending here:

renovate/lib/datasource/docker/common.ts

Lines 170 to 177 in 3b65140

	if (err.statusCode === 401) {
	logger.debug(
	{ registryHost, dockerRepository },
	'Unauthorized docker lookup'
	);
	logger.debug({ err });
	return null;
	}

The auth-check without auth above does throw an error. This in fact disables support for authenticated docker access, right?

[viceice]

did you get this message?

renovate/lib/datasource/docker/common.ts

Lines 104 to 107 in 11dd27f

	logger.trace(
	{ registryHost, dockerRepository },
	`Using basic auth for Docker registry`
	);

[KnisterPeter]

No, I was thrown into the catch block below.

[viceice]

If you don't see that line, your hostrule don't work as expected. you need to debug yourself, as you don't use a official supported way to run renovate. 🤷‍♂️

[KnisterPeter]

Okay, compiling my changes is a requirement... 🤦
Thanks for your help and patience. 😃