RFC: Docker-based development support

[ironcladlou]

Why build or run OpenShift in Docker?

Building and running on bare metal and on VMs each have advantages and disadvantages. Using Docker for builds and development is useful when bare metal is too limited and VMs are too cumbersome. Here are some of the reasons to use Docker when developing OpenShift:

• Many of the scheduling, load balancing, replication, and routing features typically only available to multi-machine VM setups become available but without the need for VMs. (Each component is bound to its own routable IP in a flat subnet topology that more accurately resembles a prod clustered deployment.)
• It's easy to compose and share portable runnable images to implement arbitrary topologies for testing, demos, and development.
• With warm caches, the build and launch cycle is very close in turnaround speed compared to bare metal development with hack/build-go.sh.
• Docker makes it possible to create integrated, shared development toolkits which are more portable than native scripts and provide a more native user experience than VM tooling. (For example, origin-build provides httpie to developers with no host requirements beyond Docker, and in a way which can easily and closely integrate with both the cluster components and the host OS)

[ironcladlou]

@smarterclayton @pmorie @ncdc @csrwng @derekwaynecarr @danmcp

[smarterclayton]

Articulate why you broke up the all in one into a ton of things? What's the value to real use cases to have ten containers?

[ironcladlou]

Articulate why you broke up the all in one into a ton of things? What's the value to real use cases to have ten containers?

First, a question from your question: I want to do some development or integration-like testing of the scheduler and load balancer against 5 minions/kubelets and pods associated with them. What are my options today other than a cluster of VMs?

[ironcladlou]

There's nothing preventing an "all-in-one" image as well. The point is having a way to compose clusters locally (multi-master tests, multi-minion, multi-anything) without VMs or host port juggling.

[smarterclayton]

Testing multiple containers in docker on a single vm is useful. I don't see the carry over to testing Openshift as multiple containers adding anything to that.

[smarterclayton]

Host port problem is going away soon in favor of testing real networks. I don't think that requires breaking up the all in one for that test env.

[ironcladlou]

Host port is going away in favor of testing real networks. I don't think that requires breaking up the all in one for that test env.

How do you do live testing of the scheduler, load balancer, replication controller, multi-anything support today without multiple VMs?

[smarterclayton]

Discussed via phone, three concrete use cases:

• from a system with docker, easily build the source of OpenShift and launch containers that act as individual minions on a single system (q: won't e kubelets fight over the docker namespace?)
• get a build / compile env in a docker container that works on a linux system allowing you to not have to install Go or other dependencies on your system
• eventually allow anyone with boot2docker (windows, mac users) to have a runtime environment they can deploy and test kubernetes with

The docker image is best for system level integration testing, spin up a container you can throw away to get back to a clean env.

[ironcladlou]

For comparison purposes, here's a branch which collapses all the builder images into a single all-in-one builder image which is reused to launch the discrete components in their own containers: https://github.com/ironcladlou/origin/compare/docker-build-support-collapsed?expand=1

Functionally equivalent, less stuff overall. All linking setup very explicitly expressed in one place now rather than being spread across build-docker.sh and a Dockerfile per component.

EDIT: note that the linked branch is a rough POC to demonstrate the concept.

[pmorie]

I really like this approach because it allows you to have a clean,
[relatively] self-contained deployment of origin without dependencies on:

1. A VM infra and orchestration tool for creating said VM
2. A particular linux distribution for your development host

I think this will be much easier to get into than a Vagrant-created
development VM (not to mention just being 'cooler') and will be much easier
for random folks to get started with.

On Sat, Sep 6, 2014 at 8:35 PM, Dan Mace [email protected] wrote:

For comparison purposes, here's a branch which collapses all the builder
images into a single all-in-one builder image which is reused to launch the
discrete components in their own containers:
https://github.com/ironcladlou/origin/compare/docker-build-support-collapsed?expand=1

Functionally equivalent, less stuff overall. All linking setup very
explicitly expressed in one place now rather than being spread across
build-docker.sh and a Dockerfile per component.

—
Reply to this email directly or view it on GitHub
#52 (comment).

[derekwaynecarr]

I like this too. Guessing the major thing we would be missing is per pod IP addresses?

I can take a stab Monday at getting that working in upstream kubernetes by reusing the provision networking pieces after I get a chance to play with this some more.

On Sep 6, 2014, at 8:39 PM, Paul Morie [email protected] wrote:

I really like this approach because it allows you to have a clean,
[relatively] self-contained deployment of origin without dependencies on:

1. A VM infra and orchestration tool for creating said VM
2. A particular linux distribution for your development host

I think this will be much easier to get into than a Vagrant-created
development VM (not to mention just being 'cooler') and will be much easier
for random folks to get started with.

On Sat, Sep 6, 2014 at 8:35 PM, Dan Mace [email protected] wrote:

For comparison purposes, here's a branch which collapses all the builder
images into a single all-in-one builder image which is reused to launch the
discrete components in their own containers:
https://github.com/ironcladlou/origin/compare/docker-build-support-collapsed?expand=1

Functionally equivalent, less stuff overall. All linking setup very
explicitly expressed in one place now rather than being spread across
build-docker.sh and a Dockerfile per component.

—
Reply to this email directly or view it on GitHub
#52 (comment).

—
Reply to this email directly or view it on GitHub.

[ironcladlou]

Per-pod IP addresses are implemented; containers in a pod still share their networking stack with a networking container as usual. All component and pod containers have IPs on the same subnet by virtue of living on the same docker bridge, which combined with links makes everything flatly routable. From an etcd perspective, there's a multi-machine cluster in action, and the kubelet doesn't care that its containers are colocated on the same physical host; docker abstracts the details of where the kubelet's containers actually live.

[smarterclayton]

On Sep 6, 2014, at 10:39 PM, Dan Mace [email protected] wrote:

Per-pod IP addresses are implemented; containers in a pod still share their networking stack with a networking container as usual. All component and pod containers have IPs on the same subnet by virtue of living on the same docker bridge (assuming a default Docker configuration). From an etcd perspective, there's a multi-machine cluster in action, and the kubelet doesn't care that its containers are colocated on the same physical host; docker abstracts the details of where the kubelet's containers actually live.

The kubelet will nuke containers from other machines unless you did something to the kubelet to support that

—
Reply to this email directly or view it on GitHub.

[ironcladlou]

The kubelet will nuke containers from other machines unless you did something to the kubelet to support that

Hm, I think you're right. Looks like the kubelet assumes that any container with a name prefixed with k8s is managed by the kubelet. That explains the guestbook weirdness I saw but didn't yet investigate (with some containers being created and then killed rapidly; the kubelets must have been deleting each others' pods).

Need to think about that some more. Random musings: Is the core assumption flawed? (e.g. Should container labels express a host affinity, like the kubelet does already? That would allow logical partitioning of containers by kubelet within Docker but I have no use case beyond setting up test clusters.) Is there any other way to accommodate the existing behavior without using docker-in-docker?

[smarterclayton]

On Sep 7, 2014, at 7:59 AM, Dan Mace [email protected] wrote:

The kubelet will nuke containers from other machines unless you did something to the kubelet to support that

Hm, I think you're right. Looks like the kubelet assumes that any container with a name prefixed with k8s is managed by the kubelet. That explains the guestbook weirdness I saw but didn't yet investigate (with some containers being created and then killed rapidly; the kubelets must have been deleting each others' pods).

Need to think about that some more. Random musings: Is the core assumption flawed? (e.g. Should container labels express a host affinity, like the kubelet does already? That would allow logical partitioning of containers by kubelet within Docker but I have no use case beyond setting up test clusters.) Is there any other way to accommodate the existing behavior without using docker-in-docker?

Testing might be enough of a use case to make it desirable.
—
Reply to this email directly or view it on GitHub.

[pmorie]

Testing is another big value add. We should prototype e2e tests against
this setup.

On Sunday, September 7, 2014, Clayton Coleman [email protected]
wrote:

On Sep 7, 2014, at 7:59 AM, Dan Mace <[email protected]
javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

The kubelet will nuke containers from other machines unless you did
something to the kubelet to support that

Hm, I think you're right. Looks like the kubelet assumes that any
container with a name prefixed with k8s is managed by the kubelet. That
explains the guestbook weirdness I saw but didn't yet investigate (with
some containers being created and then killed rapidly; the kubelets must
have been deleting each others' pods).

Need to think about that some more. Random musings: Is the core
assumption flawed? (e.g. Should container labels express a host affinity,
like the kubelet does already? That would allow logical partitioning of
containers by kubelet within Docker but I have no use case beyond setting
up test clusters.) Is there any other way to accommodate the existing
behavior without using docker-in-docker?

Testing might be enough of a use case to make it desirable.
—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#52 (comment).

[ironcladlou]

The latest branch ditches the multiple images in favor of a single build image which also serves as a runner. It also has some really, really rough code to:

1. Stop using Docker links (to allow routing without knowing in advance what ports will be allocated later, i.e. in pod containers)
2. Facilitate developing both k8s and origin in tandem by allowing local k8s (or other) source to override the Godeps dependencies during the build (for my own sanity)

Something's still wrong with proxying.

To even try this branch, you need to apply this patch to a local clone of the openshfit/kubernetes repo using the stable branch:

ironcladlou/kubernetes@552c68a

[ironcladlou]

Talking and thinking about all this more, docker-in-docker sidesteps all the kubelet and proxy issues entirely, at the expense of some UX complexity: to get at the pod containers, we'd have to try and transparently expose the inner dockers (random idea: docker run ... docker ps using bind mounts to inner docker sockets to see a minion's containers). This complexity compounds the deeper you go, as with docker-in-docker in container pods executing builds.

Docker-in-docker also means a loss of image caching on the host, so spinning up tests is going to be drastically slower unless host level caching is in place.

[ironcladlou]

The latest commit demonstrates how this could work using docker-in-docker. It requires no patching to kubernetes. Here's how I inspect the docker daemon within the minions:

DOCKER_HOST=tcp://$(dr-ip origin-minion-1):2375 docker ps

I use a little helper script called dr-ip to get the IPs of docker containers:

docker inspect -f "{{ .NetworkSettings.IPAddress }}" $1 | tr -d '\n'

The guestbook example (with all hostPort refs stripped out) works out of the box with this setup. It does require --privileged containers for the kubelets.

[ncdc]

launch

[smarterclayton]

Is there any benefit for an end user to try this path? If so, have a section for it in CONTRIBUTING.adoc

[openshift-bot]

Origin Action Required: Pull request cannot be automatically merged, please rebase your branch from latest HEAD and push again