Can't connect to k3s container

[rpocase]

Thanks for putting this together! I'm looking at evaluating this in place of rolling my own or extending this to try and be usable with k3d. I'm having trouble getting the service to actually launch, though. Any suggestions?

Steps to reproduce:

1. Run initialization as indicated in the README:

docker network create k3s
docker run -d --privileged --name k3s --hostname k3s --network k3s unboundedsystems/k3s-dind
docker exec k3s get-kubeconfig.sh > ./k3sconfig
export KUBECONFIG=./k3sconfig

2. Change localhost to container name:

kubectl get nodes
The connection to the server localhost:8443 was refused - did you specify the right host or port?
sed -i 's/localhost/k3s/' ./k3sconfig

kubectl get nodes
Unable to connect to the server: dial tcp: lookup k3s on 127.0.0.53:53: server misbehaving

Getting a lot of errors in the k3s container logs, but they all seem to be the same root cause.

E1001 13:34:05.282776     177 summary_sys_containers.go:47] Failed to get system container stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get cgroup stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get container info for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": unknown container "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13"
E1001 13:34:15.286885     177 summary_sys_containers.go:47] Failed to get system container stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": failed to get cgroup stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": failed to get container info for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": unknown container "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy"
E1001 13:34:15.286915     177 summary_sys_containers.go:47] Failed to get system container stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get cgroup stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get container info for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": unknown container "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13"
E1001 13:34:25.290442     177 summary_sys_containers.go:47] Failed to get system container stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": failed to get cgroup stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": failed to get container info for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy": unknown container "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13/kube-proxy"
E1001 13:34:25.290471     177 summary_sys_containers.go:47] Failed to get system container stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get cgroup stats for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": failed to get container info for "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13": unknown container "/docker/92777451d41fa70a762b16b3c061c0b23858421cb3805152544c48ceb16a9d13"

[mterrel]

Hi @rpocase!

Sorry you're having issues with this. It looks like the README is wrong. We typically use k3s on its own Docker network, so the instructions have some remnants of that usage. See if this set of steps works for you:

First, make sure any previous k3s containers are not running:

docker stop k3s
docker rm k3s

Then run without putting k3s on its own network and instead, expose the API port:

docker run -d --privileged --name k3s --hostname k3s -p 8443:8443 unboundedsystems/k3s-dind

Then the final couple of steps should work:

docker exec k3s get-kubeconfig.sh > ./k3sconfig
export KUBECONFIG=./k3sconfig
kubectl get nodes

Note that this form uses Docker's host networking. So if you plan on running Kubernetes network services, you'll need to also expose those ports (using -p) when you start the k3s container. If you're doing more dynamic services in Kubernetes, you'll want to switch to using non-host Docker networks due to Docker's limitations there.

Please confirm that this works for you and I'll update the README. Or if it doesn't, let me know and I'll help troubleshoot.

[rpocase]

Thanks for the fast feedback. That does get me going on from an API perspective.

kubectl get nodes                
NAME   STATUS   ROLES    AGE    VERSION
k3s    Ready    <none>   2m1s   v1.14.1-k3s.4

I am still seeing a lot of E1001 in the log. Is this expected?

E1001 17:34:18.755606     171 summary_sys_containers.go:47] Failed to get system container stats for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d/kube-proxy": failed to get cgroup stats for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d/kube-proxy": failed to get container info for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d/kube-proxy": unknown container "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d/kube-proxy"
E1001 17:34:18.755698     171 summary_sys_containers.go:47] Failed to get system container stats for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d": failed to get cgroup stats for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d": failed to get container info for "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d": unknown container "/docker/c12267d82984437eabbbd7bf63a317db8466b07822fa9003719754beba45cd4d"

[mterrel]

We also see that in our logs, but so far haven't noticed any other effect, so it hasn't been worthwhile to track down what's going on.