Ability to use private registry #99

Prodian0013 · 2019-02-28T23:11:31Z

Is your feature request related to a problem? Please describe.
The current installation method requires internet access.

Describe the solution you'd like
Provide the capability to use a private registry when initializing the k3s server and agent.

Describe alternatives you've considered
Use crictl to pull the images from a private registry (unproven at this point).

Additional context
Add any other context or screenshots about the feature request here.

The text was updated successfully, but these errors were encountered:

ibuildthecloud · 2019-03-01T18:21:22Z

@Prodian0013 in general k8s can support private repos through pull secrets. Are you looking for a way to either replicate the images need for k3s in your private repo or will #92 suffice?

Prodian0013 · 2019-03-02T02:04:35Z

@Prodian0013 in general k8s can support private repos through pull secrets. Are you looking for a way to either replicate the images need for k3s in your private repo or will #92 suffice?

@ibuildthecloud our current implementation uses docker and kubeadm to init the cluster. Prior to the kube init each image is preloaded on each node from a private registry and tagged to the original image name. Of course this is all automated.

rcarmo · 2019-05-18T14:33:53Z

It would be great to have this working with internal private registries in general (not just for setup), both with and without auth, for CI pipelines and such. I use registry:2 for these scenarios.

galal-hussein · 2019-09-27T19:52:59Z

The support for private registry can be done from containerd template but its complicated so proposing these designs to this feature:

Design 1

Adding couple of flags to support supplying of private registry address/auth info
k3s will use these flag to set up containerd template to support the private registry
k3s also should support docker mode and configure private registry for docker

Design 2

Adding a flag that points to config file to support multiple private registries with their authentication, for example:

- address: x.x.x.x
  username: xxxxx
  password: xxxxxx
- address: y.y.y.y
  auth: xxxxxxxxx

k3s will use parse this config file and modify the containerd template accordingly
k3s also should support docker mode and configure private registry for docker

ibuildthecloud · 2019-10-03T00:47:17Z

@galal-hussein Design 2 sounds better given that there are password/secrets involved. I would just default the file to /etc/rancher/k3s/registries.yaml.

ShylajaDevadiga · 2019-10-17T20:07:33Z

Verified using master.

morsik · 2019-11-22T16:31:53Z

So… how can we use this? there's no documentation for this feature and file format of that registries.yaml is unknown :(

rcarmo · 2019-11-23T12:23:22Z

What sort of mechanisms do we have to easily and automatically provision a private registry across all nodes? I’m asking this because I was previously using an insecure private registry running on he master node, and all I had to do was copy the old configuration file across. Now to have an air-gapped cluster I have to do the entire CA rigmarole and copy across certificate files as well. Are we sure that containerd does not support HTTP-only registries at all?

…

On 22 Nov 2019, at 17:05, David Nuzik ***@***.***> wrote: This is documented here: https://rancher.com/docs/k3s/latest/en/installation/airgap/#create-registry-yaml We'll look into documenting separately from an air-gap scenario. Registry cannot be insecure (containerd requires this). — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

morsik · 2019-11-27T09:49:03Z

@davidnuzik thanks! I was able to recreate config from source code ;)

But yeah… I didn't know about that air-gap, still don't know what's that and separating is good idea!

ramukima · 2019-11-30T02:48:15Z

@morsik Here is a sample registries.yaml content that works for me -

mirrors:
  private.registry.com:
    endpoint:
    - https://private.registry.com
configs:
  private.registry.com:
    auth:
      username: <username>
      password: <password>

davidnuzik · 2020-01-20T21:42:12Z

FYI PR for adding documentation for Private Registry rancher/docs#2194

jaredallard · 2020-02-11T00:36:24Z

It looks like this will break when you supply a registry password that is a json key, i.e _json_key for gcr. Any good options there?

ghost · 2020-11-18T22:18:10Z

Yeah, I'm trying to figure out how to give agent nodes access to a private GCR with a service account using docker as the container manager. Ideally it would there would just be a secret in the cluster that could be referenced via imagePullSecrets:. Otherwise, how does one add a GCP service account json file to a registries.yaml file?

armourshield · 2021-01-25T14:29:26Z

Hi all,

I am trying to use docker registry with my k3s. I have docker registry running in a different server (dockerregistryip) and the k3s master in another (k3smasterip)

I have followed the instruction of setting up the docker login and certs as in the documentation and it worked.

I can push and pull the images. I did the following setup with docker login myregistrydomain.com:443.

This was done in k3smasterip

vi /etc/hosts
(dockerregistryip) myregistrydomain.com

vi /etc/docker/daemon.json
{
  "insecure-registries" : ["https://myregistrydomain.com:443"]
}

mkdir /etc/docker/certs.d
mkdir /etc/docker/certs.d/myregistrydomain.com

cp certs/domain.crt /etc/docker/certs.d/myregistrydomain.com/ca.crt
cp certs/domain.key /etc/docker/certs.d/myregistrydomain.com/
cp certs/domain.crt /etc/docker/certs.d/myregistrydomain.com/domain.cert

cp certs/domain.crt /usr/local/share/ca-certificates/myregistrydomain.com.crt
update-ca-certificates

sudo systemctl restart docker
sudo docker info

For k3s, I followed the instruction for WITH TLS and AUTHENTICATION

mirrors:
  myregistrydomain.com:443:
    endpoint:
      - "https://myregistrydomain.com:443"
configs:
  "myregistrydomain.com:443":
     auth:
       username: username
       password: password
     tls:
       cert_file: "/etc/docker/certs.d/myregistrydomain.com/domain.cert"
       key_file: "/etc/docker/certs.d/myregistrydomain.com/domain.key"
       ca_file: "/etc/docker/certs.d/myregistrydomain.com/ca.crt"

When I do a deployment with the image which is there in repository I get the following error

apiVersion: apps/v1
kind: Deployment
...
spec:
      containers:
      - name: imagename
        image: myregistrydomain.com:443/repo/imagename:tag

ERROR:
ImagePullBackOff

DETAILS:
Failed to pull image "myregistrydomain.com:443/repo/imagename:tag": rpc error: code = Unknown desc = failed to pull and unpack image "myregistrydomain.com:443/repo/imagename:tag": failed to resolve reference "myregistrydomain.com:443/repo/imagename:tag": failed to do request: Head "https://myregistrydomain.com:443/v2/repo/imagename/manifests/tag": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

Have I misconfigured k3s? because docker in k3master server can push and pull images from dockerregistryip. Your help will be really appreciated.

armourshield · 2021-01-25T14:39:24Z

Hi all,

I am trying to use docker registry with my k3s. I have docker registry running in a different server (dockerregistryip) and the k3s master in another (k3smasterip)

I have followed the instruction of setting up the docker login and certs as in the documentation and it worked.

I can push and pull the images. I did the following setup with docker login myregistrydomain.com:443.

This was done in k3smasterip
vi /etc/hosts
(dockerregistryip) myregistrydomain.com
vi /etc/docker/daemon.json
{
  "insecure-registries" : ["https://myregistrydomain.com:443"]
}
mkdir /etc/docker/certs.d
mkdir /etc/docker/certs.d/myregistrydomain.com
cp certs/domain.crt /etc/docker/certs.d/myregistrydomain.com/ca.crt
cp certs/domain.key /etc/docker/certs.d/myregistrydomain.com/
cp certs/domain.crt /etc/docker/certs.d/myregistrydomain.com/domain.cert
cp certs/domain.crt /usr/local/share/ca-certificates/myregistrydomain.com.crt
update-ca-certificates
sudo systemctl restart docker
sudo docker info
For k3s, I followed the instruction for WITH TLS and AUTHENTICATION
mirrors:
  myregistrydomain.com:443:
    endpoint:
      - "https://myregistrydomain.com:443"
configs:
  "myregistrydomain.com:443":
     auth:
       username: username
       password: password
     tls:
       cert_file: "/etc/docker/certs.d/myregistrydomain.com/domain.cert"
       key_file: "/etc/docker/certs.d/myregistrydomain.com/domain.key"
       ca_file: "/etc/docker/certs.d/myregistrydomain.com/ca.crt"
When I do a deployment with the image which is there in repository I get the following error
apiVersion: apps/v1
kind: Deployment
...
spec:
      containers:
      - name: imagename
        image: myregistrydomain.com:443/repo/imagename:tag
ERROR:
ImagePullBackOff

DETAILS:
Failed to pull image "myregistrydomain.com:443/repo/imagename:tag": rpc error: code = Unknown desc = failed to pull and unpack image "myregistrydomain.com:443/repo/imagename:tag": failed to resolve reference "myregistrydomain.com:443/repo/imagename:tag": failed to do request: Head "https://myregistrydomain.com:443/v2/repo/imagename/manifests/tag": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

Have I misconfigured k3s? because docker in k3master server can push and pull images from dockerregistryip. Your help will be really appreciated.

Solved it by adding insecure_skip_verify flag to true. I think this is needed for self-signed certs.

brandond · 2021-01-25T23:30:38Z

Solved it by adding insecure_skip_verify flag to true. I think this is needed for self-signed certs.

No @armourshield - the error message tells you what the problem is: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

You need to update the cert on your registry to include the hostname in its Subject Alternative Names. Go 1.15 no longer supports certs without SANs unless you set the environment variable mentioned in the error.

bvhoang2003 · 2021-02-19T14:42:30Z

Where do we add the insecure_skip_verify flag? Thanks!

brandond · 2021-02-19T18:10:03Z

@bvhoang2003 https://rancher.com/docs/k3s/latest/en/installation/private-registry/#configs

This was referenced Mar 5, 2019

Allows to preload a docker image on the k3s node agents #92

Closed

Running k3s on an airgap environment #172

Closed

erikwilson added the kind/enhancement An improvement to existing functionality label Mar 25, 2019

simmessa mentioned this issue May 28, 2019

imagePullSecrets support #502

Closed

davidnuzik added this to the v1.0 - Backlog milestone Sep 24, 2019

davidnuzik assigned galal-hussein Sep 26, 2019

davidnuzik added the [zube]: Working label Oct 2, 2019

galal-hussein mentioned this issue Oct 7, 2019

Add private registry support to containerd #867

Merged

galal-hussein added [zube]: Peer Review and removed [zube]: Working labels Oct 10, 2019

davidnuzik added [zube]: To Test and removed [zube]: Peer Review labels Oct 16, 2019

davidnuzik assigned ShylajaDevadiga Oct 16, 2019

ShylajaDevadiga closed this as completed Oct 17, 2019

ShylajaDevadiga added [zube]: Done and removed [zube]: To Test labels Oct 17, 2019

j0holo mentioned this issue Dec 21, 2019

[Documentation] Private Registry #1121

Closed

dweomer mentioned this issue May 16, 2020

k3s with private registry #1802

Closed

davidnuzik removed the [zube]: Done label Sep 23, 2020

hilookas mentioned this issue Oct 7, 2021

miss files while copying #4175

Closed

1 task

VladimirKorzh mentioned this issue Nov 26, 2021

CoreDNS name resolution fails, NodePort is not available on nodes without the pod #4588

Closed

domvo mentioned this issue Dec 16, 2021

Plain k3s (traefik) doesn't work with cert-manager #4765

Closed

stintel mentioned this issue Jun 16, 2022

IPv6-only setup fails: cannot configure IPv4 cluster-cidr: no IPv4 CIDRs found #5737

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to use private registry #99

Ability to use private registry #99

Prodian0013 commented Feb 28, 2019

ibuildthecloud commented Mar 1, 2019

Prodian0013 commented Mar 2, 2019

rcarmo commented May 18, 2019 •

edited

Loading

galal-hussein commented Sep 27, 2019 •

edited

Loading

ibuildthecloud commented Oct 3, 2019

ShylajaDevadiga commented Oct 17, 2019

morsik commented Nov 22, 2019

rcarmo commented Nov 23, 2019 via email

morsik commented Nov 27, 2019

ramukima commented Nov 30, 2019 •

edited

Loading

davidnuzik commented Jan 20, 2020

jaredallard commented Feb 11, 2020

ghost commented Nov 18, 2020

armourshield commented Jan 25, 2021

armourshield commented Jan 25, 2021

brandond commented Jan 25, 2021 •

edited

Loading

bvhoang2003 commented Feb 19, 2021

brandond commented Feb 19, 2021

Ability to use private registry #99

Ability to use private registry #99

Comments

Prodian0013 commented Feb 28, 2019

ibuildthecloud commented Mar 1, 2019

Prodian0013 commented Mar 2, 2019

rcarmo commented May 18, 2019 • edited Loading

galal-hussein commented Sep 27, 2019 • edited Loading

Design 1

Design 2

ibuildthecloud commented Oct 3, 2019

ShylajaDevadiga commented Oct 17, 2019

morsik commented Nov 22, 2019

rcarmo commented Nov 23, 2019 via email

morsik commented Nov 27, 2019

ramukima commented Nov 30, 2019 • edited Loading

davidnuzik commented Jan 20, 2020

jaredallard commented Feb 11, 2020

ghost commented Nov 18, 2020

armourshield commented Jan 25, 2021

armourshield commented Jan 25, 2021

brandond commented Jan 25, 2021 • edited Loading

bvhoang2003 commented Feb 19, 2021

brandond commented Feb 19, 2021

rcarmo commented May 18, 2019 •

edited

Loading

galal-hussein commented Sep 27, 2019 •

edited

Loading

ramukima commented Nov 30, 2019 •

edited

Loading

brandond commented Jan 25, 2021 •

edited

Loading