Running Torus on a kubernetes cluster is as easy as running the included torus-k8s-oneshot.yaml
. However, this will not install the FlexVolume plugin to allow you to use Torus as a volume provider for other Kubernetes pods.
git clone http://github.com/coreos/coreos-kubernetes
cd coreos-kubernetes
git remote add barakmich https://github.com/barakmich/coreos-kubernetes.git
git fetch barakmich
git checkout barakmich/torus-k8s
When this sets up a multinode cluster (using the generic
scripts) it will pull the mount tool from S3
On Vagrant, use the kube-config. The 10m-k8s scripts are helpful for other cloud providers.
kubectl create -f torus-k8s-oneshot.yaml
And you're done.
Grab the latest release and use the included torusctl
tool. Or just build it from source.
torusctl -C $IP_IN_CLUSTER:32379 list-peers
Which should tell you everything about the cluster.
Create a volume, eg:
torusctl -C $IP_IN_CLUSTER:32379 volume create-block pg1 2GiB
And now use this volume in any other kubernetes pods, for example:
kubectl create -f postgres-oneshot.yaml
TORUSPOD=$(kubectl get pods -l app=postgres-torus -o name | cut -d/ -f2)
kubectl exec -i $TORUSPOD -- psql postgres -U postgres < test-data.sql
kubectl exec $TORUSPOD -- psql postgres -U postgres -c 'select * from films'
First lets cordon off the node postgres is currently on, so that when we kill it, it doesn't go to the same node.
PGNODE=$(kubectl get pods -l app=postgres-torus -o jsonpath='{.items[0].spec.nodeName}')
kubectl cordon $PGNODE
kubectl get nodes
Node we will delete the existing postgres pod, and then watch for a new one to come up and replace it
kubectl delete pod -l app=postgres-torus
kubectl get pod -l app=postgres-torus -w
You should see some output similar to
$ kubectl get pods -w -l app=postgres-torus
NAME READY STATUS RESTARTS AGE
postgres-torus-1844296455-6z492 1/1 Terminating 1 8m
postgres-torus-1844296455-mv6v9 0/1 ContainerCreating 0 13s
NAME READY STATUS RESTARTS AGE
postgres-torus-1844296455-6z492 0/1 Terminating 1 9m
postgres-torus-1844296455-6z492 0/1 Terminating 1 9m
postgres-torus-1844296455-6z492 0/1 Terminating 1 9m
postgres-torus-1844296455-mv6v9 1/1 Running 0 1m
Finally we can verify that the data is still there:
TORUSPOD=$(kubectl get pods -l app=postgres-torus -o name | cut -d/ -f2)
kubectl exec $TORUSPOD -- psql postgres -U postgres -c 'select * from films'
Lastly, let's uncordon that node we cordoned in the beginning:
kubectl uncordon $PGNODE
You can cleanup resources you deployed:
kubectl delete pod etcd-torus
kubectl delete svc etcd-torus etcd-torus-internal
kubectl delete daemonset torus
kubectl delete svc postgres-torus
kubectl delete deployment postgres-torus
NOTICE: The FlexVolume functionality currently uses systemd to manage its lifecycle. Running as a FlexVolume on non-systemd systems is TBD
Kubernetes v1.2 supports FlexVolumes by placing a plugin binary in a specific location. By default, that is
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/
The torusblk
tool already conforms to this interface, so it's a simple matter of naming it correctly and placing it correctly.
mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/coreos.com~torus/
cp torusblk /usr/libexec/kubernetes/kubelet-plugins/volume/exec/coreos.com~torus/torus
Notice that the cp
command renames torusblk
as torus
in the target directory.
After that, restart the kubelet (ie, systemctl restart kubelet
, or /etc/init.d/kubelet restart
) -- and the plugin is ready.
Currently the FlexVolume uses nbd kernel module. You have to enable nbd
kernel module on Node hosts.
modprobe nbd nbds_max=32