Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

clear containers #1945

Merged
merged 13 commits into from
Jan 17, 2018
1 change: 1 addition & 0 deletions docs/clusterdefinition.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Here are the valid values for the orchestrator types:
|kubernetesImageBase|no|This specifies the base URL (everything preceding the actual image filename) of the kubernetes hyperkube image to use for cluster deployment, e.g., `k8s-gcrio.azureedge.net/`.|
|dockerEngineVersion|no|Which version of docker-engine to use in your cluster, e.g.. "17.03.*"|
|networkPolicy|no|Specifies the network policy tool for the cluster. Valid values are:<br>`"azure"` (default), which provides an Azure native networking experience,<br>`none` for not enforcing any network policy,<br>`calico` for Calico network policy (clusters with Linux agents only).<br>See [network policy examples](../examples/networkpolicy) for more information.|
|containerRuntime|no|The container runtime to use as a backend. The default is `docker`. The only other option is `clear-containers`.|
|clusterSubnet|no|The IP subnet used for allocating IP addresses for pod network interfaces. The subnet must be in the VNET address space. Default value is 10.244.0.0/16.|
|dnsServiceIP|no|IP address for kube-dns to listen on. If specified must be in the range of `serviceCidr`.|
|dockerBridgeSubnet|no|The specific IP and subnet used for allocating IP addresses for the docker bridge network created on the kubernetes master and agents. Default value is 172.17.0.1/16. This value is used to configure the docker daemon using the [--bip flag](https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0).|
Expand Down
35 changes: 35 additions & 0 deletions docs/kubernetes/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
|Managed Disks|Beta|`vlabs`|[kubernetes-vmas.json](../../examples/disks-managed/kubernetes-vmss.json)|[Description](#feat-managed-disks)|
|Calico Network Policy|Alpha|`vlabs`|[kubernetes-calico.json](../../examples/networkpolicy/kubernetes-calico.json)|[Description](#feat-calico)|
|Custom VNET|Beta|`vlabs`|[kubernetesvnet-azure-cni.json](../../examples/vnet/kubernetesvnet-azure-cni.json)|[Description](#feat-custom-vnet)|
|Clear Containers Runtime|Alpha|`vlabs`|[kubernetes-clear-containers.json](../../examples/kubernetes-clear-containers.json)|[Description](#feat-clear-containers)|

<a name="feat-kubernetes-msi"></a>

Expand Down Expand Up @@ -236,3 +237,37 @@ E.g.:
}
]
```

<a name="feat-clear-containers"></a>

## Clear Containers

You can designate kubernetes agents to use Intel's Clear Containers as the
container runtime by setting:

```
"kubernetesConfig": {
"containerRuntime": "clear-containers"
}
```

You will need to make sure your agents are using a `vmSize` that [supports
nested
virtualization](https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/).
These are the `Dv3` or `Ev3` series nodes.

You will also need to attach a disk to those nodes for the device-mapper disk that clear containers will use.
This should look like:

```
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D4s_v3",
"availabilityProfile": "AvailabilitySet",
"storageProfile": "ManagedDisks",
"diskSizesGB": [1023]
}
],
```
53 changes: 53 additions & 0 deletions examples/kubernetes-clear-containers.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.8",
"kubernetesConfig": {
"networkPolicy": "azure",
"containerRuntime": "clear-containers",
"etcdVersion": "3.1.10",
"addons": [
{
"name": "tiller",
"enabled" : false
},
{
"name": "kubernetes-dashboard",
"enabled" : false
}
]
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D4s_v3",
"availabilityProfile": "AvailabilitySet",
"storageProfile": "ManagedDisks",
"diskSizesGB": [1023]
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}
3 changes: 2 additions & 1 deletion parts/k8s/artifacts/kuberneteskubelet.service
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ ExecStart=/usr/bin/docker run \
--volume=/sys:/sys:ro \
--volume=/var/run:/var/run:rw \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/containers/:/var/lib/containers:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:shared \
--volume=/var/log:/var/log:rw \
--volume=/etc/kubernetes/:/etc/kubernetes:ro \
Expand All @@ -39,7 +40,7 @@ ExecStart=/usr/bin/docker run \
--v=2 ${KUBELET_FEATURE_GATES} \
--non-masquerade-cidr=${KUBELET_NON_MASQUERADE_CIDR} \
--volume-plugin-dir=/etc/kubernetes/volumeplugins \
$KUBELET_CONFIG \
$KUBELET_CONFIG $KUBELET_OPTS \
${KUBELET_REGISTER_NODE} ${KUBELET_REGISTER_WITH_TAINTS}

[Install]
Expand Down
3 changes: 2 additions & 1 deletion parts/k8s/kubernetesagentcustomdata.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ write_files:
KUBELET_CONFIG={{GetKubeletConfigKeyVals .KubernetesConfig }}
KUBELET_IMAGE={{WrapAsVariable "kubernetesHyperkubeSpec"}}
DOCKER_OPTS=
KUBELET_OPTS=
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is KUBELET_OPTS doing anything here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is... it is being set when the runtime is set, unfortunately I was going to do this via the config file but the behavior of clear containers for how I usually setup clusters is the system nodes use docker and don't have nested virt enabled, and the agents all have nested virt so it sets --container-runtime=remote for only those nodes and not all of them

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, didn't see it because gh was truncating the diff with all the whitespace fun! Whitespace has been made less fun, now I can see the setKubeletOpts function without clicking things.

KUBELET_REGISTER_SCHEDULABLE=true
KUBELET_NODE_LABELS={{GetAgentKubernetesLabels . "',variables('labelResourceGroup'),'"}}
{{if IsKubernetesVersionGe "1.6.0"}}
Expand Down Expand Up @@ -194,4 +195,4 @@ runcmd:
- apt-mark unhold walinuxagent
- mkdir -p /opt/azure/containers && touch /opt/azure/containers/runcmd.complete
- echo `date`,`hostname`, endruncmd>>/opt/m
{{end}}
{{end}}
1 change: 1 addition & 0 deletions parts/k8s/kubernetesmastercustomdata.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ MASTER_ADDONS_CONFIG_PLACEHOLDER
KUBELET_CONFIG={{GetKubeletConfigKeyVals .MasterProfile.KubernetesConfig}}
KUBELET_IMAGE={{WrapAsVariable "kubernetesHyperkubeSpec"}}
DOCKER_OPTS=
KUBELET_OPTS=
KUBELET_NODE_LABELS={{GetMasterKubernetesLabels "',variables('labelResourceGroup'),'"}}
{{if IsKubernetesVersionGe "1.6.0"}}
{{if HasLinuxAgents}}
Expand Down
Loading