Skip to content

Latest commit

 

History

History
456 lines (350 loc) · 21.1 KB

windows.md

File metadata and controls

456 lines (350 loc) · 21.1 KB
Raw

Deploying Antrea on Windows

Table of Contents

Overview

Antrea supports Windows worker Nodes. On Windows Nodes, Antrea sets up an overlay network to forward packets between Nodes and implements NetworkPolicies. Currently Geneve, VXLAN, and STT tunnels are supported.

This page shows how to install antrea-agent on Windows Nodes and register the Node to an existing Kubernetes cluster.

For the detailed design of how antrea-agent works on Windows, please refer to the design doc.

Note: Docker support on Windows Nodes was dropped completely in Antrea v2.0, making containerd the only supported container runtime. As part of this change, we renamed the antrea-windows-containerd.yml manifest to antrea-windows.yml, and the antrea-windows-containerd-with-ovs.yml manifest to antrea-windows-with-ovs.yml. Prior to the Antrea v2.0 release, the antrea-windows.yml manifest was used to support Windows Nodes with Docker. For the best experience, make sure that you refer to the version of the documentation that matches the Antrea version you are deploying.

Components that run on Windows

The following components should be configured and run on the Windows Node.

antrea-agent and the OVS daemons can either run as Pods (containerized) or as Windows services, and the following configurations are supported:

OVS daemons antrea-agent Supported Refer to
Containerized Containerized Yes (recommended) Installation as a Pod
Containerized Windows Service No N/A
Windows Services Containerized Yes Installation as a Pod
Windows Services Windows Services Yes Installation as a Service

Antrea Windows demo

Watch this demo video of running Antrea in a Kubernetes cluster with both Linux and Windows Nodes. The demo also shows the Antrea OVS bridge configuration on a Windows Node, and NetworkPolicy enforcement for Windows Pods. Note, OVS driver and daemons are pre-installed on the Windows Nodes in the demo.

Deploying Antrea on Windows worker Nodes

Running Antrea on Windows Nodes requires the containerd container runtime. The recommended installation method is Installation as a Pod, and it requires containerd 1.6 or higher. If you prefer running the Antrea Agent as a Windows service, or if you are using containerd 1.5, you can use the Installation as a Service method.

Starting from v2.1, Antrea Windows image is built on Linux host with docker buildx and uses hpc as the base image.

Prerequisites

  • Create a Kubernetes cluster.
  • Obtain a Windows Server 2019 license (or higher) in order to configure the Windows Nodes that will host Windows containers. And install the latest Windows updates.
  • On each Windows Node, install the following:
    • Hyper-V with management tools. If your Nodes do not have the virtualization capabilities required by Hyper-V, use the workaround described in the Known issues section.
    • containerd.

Installation as a Pod

This installation method requires Antrea v1.10 or higher, and containerd 1.6 or higher (containerd 1.7 or higher is recommended). It relies on support for Windows HostProcess Pods, which is generally available starting with K8s 1.26.

More detailed containerd Version requirements are outlined below:

Kubernetes Version Recommended containerd Version
1.26 1.7.0+, 1.6.18+
1.27 1.7.0+, 1.6.18+
1.28 1.7.0+, 1.6.18+
1.29 1.7.11+, 1.6.27+
1.30 1.7.13+, 1.6.28+

Note: Starting from Antrea v2.1, Antrea Windows image is built based on the HPC (Host Process Containers) image, containerd version 1.6.18 or higher is required because versions earlier than 1.6.18 do not support importing HPC images on Windows.

For more detailed information on Kubernetes-supported containerd versions, refer to the Containerd releases page

Starting with Antrea v1.13, Antrea takes over all the responsibilities of kube-proxy for Windows Nodes by default, and kube-proxy should not be deployed on Windows Nodes with Antrea.

Download & Configure Antrea for Linux

Deploy Antrea for Linux on the control-plane Node following Getting started document. The following command deploys Antrea with the version specified by <TAG>:

kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea.yml

Add Windows antrea-agent DaemonSet

You need to manually set the kubeAPIServerOverride field in the YAML configuration file as the Antrea Proxy proxyAll mode is enabled by default.

    # Provide the address of Kubernetes apiserver, to override any value provided in kubeconfig or InClusterConfig.
    # Defaults to "". It must be a host string, a host:port pair, or a URL to the base of the apiserver.
    kubeAPIServerOverride: "10.10.1.1:6443"

    # Option antreaProxy contains Antrea Proxy related configuration options.
    antreaProxy:
      # Option proxyAll tells antrea-agent to proxy ClusterIP Service traffic, regardless of where they come from.
      # Therefore, running kube-proxy is no longer required. This requires the Antrea Proxy feature to be enabled.
      # Note that this option is experimental. If kube-proxy is removed, option kubeAPIServerOverride must be used to access
      # apiserver directly.
      proxyAll: true

You can run both the Antrea Agent and the OVS daemons on Windows Nodes using a single DaemonSet, by applying the file antrea-windows-with-ovs.yml. This is the recommended installation method. The following commands download the manifest, set kubeAPIServerOverride, and create the DaemonSet:

KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \
curl -sL https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-windows-with-ovs.yml | \
sed "s|.*kubeAPIServerOverride: \"\"|    kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \
kubectl apply -f -

Alternatively, to deploy the antrea-agent Windows DaemonSet without the OVS daemons, apply the file antrea-windows.yml with the following commands:

KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \
curl -sL https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-windows.yml | \
sed "s|.*kubeAPIServerOverride: \"\"|    kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \
kubectl apply -f -

When using antrea-windows.yml, you will need to install OVS userspace daemons as services when you prepare your Windows worker Nodes, in the next section.

Join Windows worker Nodes

1. (Optional) Install OVS (provided by Antrea or your own)

Depending on which method you are using to install Antrea on Windows, and depending on whether you are using your own signed OVS kernel driver or you want to use the test-signed driver provided by Antrea, you will need to invoke the Install-OVS.ps1 script differently (or not at all).

Containerized OVS daemons? Test-signed OVS driver? Run this command
Yes Yes Not required
Yes No Not required
No Yes .\Install-OVS.ps1 -InstallUserspace $true
No No .\Install-OVS.ps1 -InstallUserspace $true -LocalFile <PathToOVSPackage>

If you used antrea-windows-with-ovs.yml to create the antrea-agent Windows DaemonSet, then you are using "Containerized OVS daemons". For all other methods, you are not using "Containerized OVS daemons".

Antrea provides a pre-built OVS package which contains a test-signed OVS kernel driver. If you don't have a self-signed OVS package and just want to try Antrea on Windows, this package can be used for testing.

[Test-only] If you are using test-signed driver (such as the one provided with Antrea), please make sure to enable test-signed:

Bcdedit.exe -set TESTSIGNING ON
Restart-Computer

If you want to run OVS as Windows native services, and you are bringing your own OVS package with a signed OVS kernel driver, you would run:

curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Install-OVS.ps1
.\Install-OVS.ps1 -InstallUserspace $true -LocalFile <PathToOVSPackage>

# verify that the OVS services are installed
get-service ovsdb-server
get-service ovs-vswitchd
2. Disable Windows Firewall
Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False
3. Install kubelet, kubeadm and configure kubelet startup params

Firstly, install kubelet and kubeadm using the provided PrepareNode.ps1 script. Specify the Node IP, Kubernetes Version and container runtime while running the script. The following command downloads and executes Prepare-Node.ps1:

# Example:
curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Prepare-Node.ps1"
.\Prepare-Node.ps1 -KubernetesVersion v1.30.0 -NodeIP 192.168.1.10
4. Prepare Node environment needed by antrea-agent

Run the following commands to prepare the Node environment needed by antrea-agent:

mkdir c:\k\antrea
cd c:\k\antrea
$TAG="v2.0.0"
curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Clean-AntreaNetwork.ps1
curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Prepare-AntreaAgent.ps1
# use -RunOVSServices $false for containerized OVS!
.\Prepare-AntreaAgent.ps1 [-RunOVSServices $false]

The script Prepare-AntreaAgent.ps1 performs the following tasks:

  • Remove stale network resources created by antrea-agent.

    After the Windows Node reboots, there will be stale network resources which need to be cleaned before starting antrea-agent.

  • Ensure OVS services are running.

    This script starts OVS services on the Node if they are not running. This step needs to be skipped in case of OVS containerization. In that case, you need to specify the parameter RunOVSServices as false.

    .\Prepare-AntreaAgent.ps1 -RunOVSServices $false

The script must be executed every time you restart the Node to prepare the environment for antrea-agent.

You can ensure that the script is executed automatically after each Windows startup by using different methods. Here are two examples for your reference:

  • Example 1: Update kubelet service.

Insert following line in kubelet service script c:\k\StartKubelet.ps1 to invoke Prepare-AntreaAgent.ps1 when starting kubelet service:

& C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false
  • Example 2: Create a ScheduledJob that runs at startup.
$trigger = New-JobTrigger -AtStartup -RandomDelay 00:00:30 
$options = New-ScheduledJobOption -RunElevated
Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger  -ScriptBlock { Invoke-Expression C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false } -ScheduledJobOption $options
5. Run kubeadm to join the Node

On Windows Nodes, run the kubeadm join command to join the cluster. The token is provided by the control-plane Node. If you lost the token, or the token has expired, you can run kubeadm token create --print-join-command (on the control-plane Node) to generate a new token and join command. An example kubeadm join command is like below:

kubeadm join 192.168.101.5:6443 --token tdp0jt.rshv3uobkuoobb4v  --discovery-token-ca-cert-hash sha256:84a163e57bf470f18565e44eaa2a657bed4da9748b441e9643ac856a274a30b9
Verify your installation

There will be temporary network interruption on Windows worker Node on the first startup of antrea-agent. It's because antrea-agent will set the OVS to take over the host network. After that you should be able to view the Windows Nodes and Pods in your cluster by running:

# Show Nodes
kubectl get nodes -o wide -n kube-system
NAME                           STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION       CONTAINER-RUNTIME
control-plane                  Ready    control-plane   1h    v1.29.0   10.176.27.168   <none>        Ubuntu 22.04.3 LTS               6.2.0-1017-generic   containerd://1.6.26
win-5akrf2tpq91                Ready    <none>          1h    v1.29.0   10.176.27.150   <none>        Windows Server 2019 Datacenter   10.0.17763.5206      containerd://1.6.6
win-5akrf2tpq92                Ready    <none>          1h    v1.29.0   10.176.27.197   <none>        Windows Server 2019 Datacenter   10.0.17763.5206      containerd://1.6.6

# Show antrea-agent Pods
kubectl get pods -o wide -n kube-system | grep windows
antrea-agent-windows-6hvkw                             1/1     Running     0          100s

Installation as a Service

Install Antrea as usual. The following command deploys Antrea with the version specified by <TAG>:

kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea.yml

When running the Antrea Agent as a Windows service, no DaemonSet is created for Windows worker Nodes. You will need to ensure that nssm is installed on all your Windows Nodes. nssm is a handy tool to manage services on Windows.

To prepare your Windows worker Nodes, follow the steps in Join Windows worker Nodes. With this installation method, OVS daemons are always run as services (not containerized), and you will need to run Install-OVS.ps1 to install them.

When your Nodes are ready, run the following scripts to install the antrea-agent service. NOTE: <KubernetesVersion>, <KubeconfigPath> and <KubeletKubeconfigPath> should be set by you. For example:

$KubernetesVersion="v1.29.0"
$KubeConfig="C:/Users/Administrator/.kube/config" # admin kubeconfig
$KubeletKubeconfigPath="C:/etc/kubernetes/kubelet.conf"
$TAG="v2.0.0"
$KubernetesVersion="<KubernetesVersion>"
$KubeConfig="<KubeconfigPath>"
$KubeletKubeconfigPath="<KubeletKubeconfigPath>"
$KubernetesHome="c:/k"
$AntreaHome="c:/k/antrea"

curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Helper.psm1"
Import-Module ./Helper.psm1

Install-AntreaAgent -KubernetesVersion "$KubernetesVersion" -KubernetesHome "$KubernetesHome" -KubeConfig "$KubeConfig" -AntreaVersion "$TAG" -AntreaHome "$AntreaHome"

New-DirectoryIfNotExist "${AntreaHome}/logs"
nssm install antrea-agent "${AntreaHome}/bin/antrea-agent.exe" "--config=${AntreaHome}/etc/antrea-agent.conf --logtostderr=false --log_dir=${AntreaHome}/logs --alsologtostderr --log_file_max_size=100 --log_file_max_num=4"

nssm set antrea-agent DependOnService ovs-vswitchd
nssm set antrea-agent Start SERVICE_DELAYED_AUTO_START

Start-Service antrea-agent

Manually run antrea-agent on Windows worker Nodes

Antrea also provides powershell scripts which help install and run the Antrea Agent manually, please complete the steps in Installation section, and skip the Add Windows antrea-agent DaemonSet step. Then run the following commands in powershell:

mkdir c:\k\antrea
cd c:\k\antrea
curl.exe -LO https://github.com/antrea-io/antrea/releases/download/<TAG>/Start-AntreaAgent.ps1
# Run antrea-agent
# $KubeConfigPath is the path of kubeconfig file
./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath

Note: Some features such as supportbundle collection are not supported in this way. It's recommended to run antrea-agent as a Pod.

Known issues

  1. HNS Network is not persistent on Windows. So after the Windows Node reboots, the HNS Network created by antrea-agent is removed, and the Open vSwitch Extension is disabled by default. In this case, the stale OVS bridge and ports should be removed. A help script Clean-AntreaNetwork.ps1 can be used to clean the OVS bridge.

    # If OVS userspace processes were running as a Service on Windows host
./Clean-AntreaNetwork.ps1 -OVSRunMode "service"
# If OVS userspace processes were running inside container in antrea-agent Pod
./Clean-AntreaNetwork.ps1 -OVSRunMode "container"

  2. Hyper-V feature cannot be installed on Windows Node due to the processor not having the required virtualization capabilities.

    If the processor of the Windows Node does not have the required virtualization capabilities. The installation of Hyper-V feature will fail with the following error:

    PS C:\Users\Administrator> Install-WindowsFeature Hyper-V

Success Restart Needed Exit Code      Feature Result
------- -------------- ---------      --------------
False   Maybe          Failed         {}
Install-WindowsFeature : A prerequisite check for the Hyper-V feature failed.
1. Hyper-V cannot be installed: The processor does not have required virtualization capabilities.
At line:1 char:1
+ Install-WindowsFeature hyper-v
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Hyper-V:ServerComponentWrapper) [Install-WindowsFeature], Exception
    + FullyQualifiedErrorId : Alteration_PrerequisiteCheck_Failed,Microsoft.Windows.ServerManager.Commands.AddWindowsF
   eatureCommand

    The capabilities are required by the Hyper-V hypervisor components to support Hyper-V isolation. If you only need Process Isolation on the Nodes. You could apply the following workaround to skip CPU check for Hyper-V feature installation.

    # 1. Install containers feature
Install-WindowsFeature containers

# 2. Install Hyper-V management powershell module
Install-WindowsFeature Hyper-V-Powershell

# 3. Install Hyper-V feature without CPU check and disable the "hypervisor"
dism /online /enable-feature /featurename:Microsoft-Hyper-V /all /NoRestart
dism /online /disable-feature /featurename:Microsoft-Hyper-V-Online /NoRestart

# 4. Restart-Computer to take effect
Restart-Computer