make proxy work with kubernetes

[errordeveloper]

Kubernetes wants to look at container IP, and the proxy doesn't support this right now.

[errordeveloper]

Kubernetes plugins are an ongoing effort and are covered by kubernetes/kubernetes#3350 and kubernetes/kubernetes#5069 as well as kubernetes/kubernetes#9646 (on which Weave plugin would depend on).

[errordeveloper]

Additionally, a plugin will get information about Kubernetes namespace a pod runs in, which we could map to a subnet and provide isolation.

[rade]

Kubernetes wants to look at container IP, and the proxy doesn't support this right now.

You mean kubernetes is doing the (equivalent of) docker inspect on a container in order find its IP address? What does it use that address for? What goes wrong due to that address being the docker network IP rather than the weave IP?

[errordeveloper]

You mean kubernetes is doing the (equivalent of) docker inspect on a container in order find its IP address?

Yes.

What does it use that address for?

It stores it in etcd, for the start (10.2.0.0/16 is the big cluster subnet on weave).

// core@etcd-02 ~ $ etcdctl get /registry/pods/default/frontend-brec9 | jq .status
{
  "phase": "Running",
  "conditions": [
    {
      "type": "Ready",
      "status": "True"
    }
  ],
  "hostIP": "172.18.0.14",
  "podIP": "10.2.2.7",
  "startTime": "2015-07-22T10:41:31Z",
  "containerStatuses": [
    {
      "name": "php-redis",
      "state": {
        "running": {
          "startedAt": "2015-07-22T10:44:46Z"
        }
      },
      "lastState": {},
      "ready": true,
      "restartCount": 0,
      "image": "kubernetes/example-guestbook-php-redis:v2",
      "imageID": "docker://dac65295b08eabdd483c787bbd765e1b244e4f25ca1474329dc5b125e6a0f692",
      "containerID": "docker://862c74842e8b56d784516bc05d080040c2880cddb6d1ef533513b188425252bc"
    }
  ]
}
// core@etcd-02 ~ $ etcdctl get /registry/pods/default/redis-master-va7q8 | jq .status
{
  "phase": "Running",
  "conditions": [
    {
      "type": "Ready",
      "status": "True"
    }
  ],
  "hostIP": "172.18.0.14",
  "podIP": "10.2.2.6",
  "startTime": "2015-07-22T10:41:31Z",
  "containerStatuses": [
    {
      "name": "master",
      "state": {
        "running": {
          "startedAt": "2015-07-22T10:42:25Z"
        }
      },
      "lastState": {},
      "ready": true,
      "restartCount": 0,
      "image": "redis",
      "imageID": "docker://0ff407d5a7d9ed36acdf3e75de8cc127afecc9af234d05486be2981cdc01a38c",
      "containerID": "docker://5055d92b6cc955197b20c50b7ef555bee9ea5983a64f3aae96ea50338b3e8f2d"
    }
  ]
}

It then gets used at least by the kube-proxy service, which sets up some NAT rules to map virtual addresses into itself and then proxies to the container using the IP address it knows for it. Here is an example with 10.2.2.1 being the address on weave bridge and 10.2.2.6 being our Redis container:

tcp        0      0 10.2.2.1:41229          10.2.2.6:6379           ESTABLISHED 1093/kube-proxy     

What goes wrong due to that address being the docker network IP rather than the weave IP?

The kube-proxy will misbehave in one way or another, not sure how many on what kind of error messages it will give...

[paulbellamy]

From discussion:

We should optionally have the proxy overwrite the IP returned by docker inspect with the weave ip for the container.

[rade]

So this is really issue #117 then. Or, rather, some ways of solving #117 would address this, e.g. getting the proxy to (optionally, not by default!) rewrite the IP address returned by docker inspect to the first address returned by weave ps <container_id>.

[rade]

https://github.com/jongpieter/coreos-guide/blob/master/coreos-weave-kubernetes/coreos-weave-kubernetes.md appears to work w/o requiring this change. How is that possible?

[errordeveloper]

https://github.com/jongpieter/coreos-guide/blob/master/coreos-weave-kubernetes/coreos-weave-kubernetes.md appears to work w/o requiring this change. How is that possible?

I don't see kubelet being configured to use the proxy. I am checking with author via email, to see whether whether I am missing something, or if there is something that is undocumented etc.

[errordeveloper]

I have tested the proxy built from revision 0ddba62 with and without --rewrite-inspect flag. Running the proxy without this flag results in services unreachable between hosts.