From 2dd9cfb35bfb3c18c91d6c21e60dabb73b339677 Mon Sep 17 00:00:00 2001 From: Lan Luo Date: Fri, 25 Mar 2022 21:07:43 +0800 Subject: [PATCH] Add upgrade guide for Multi-cluster Signed-off-by: Lan Luo --- docs/multicluster/api.md | 25 +++++++++ docs/multicluster/upgrade.md | 93 +++++++++++++++++++++++++++++++++ docs/multicluster/user-guide.md | 2 +- hack/.notableofcontents | 2 + 4 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 docs/multicluster/api.md create mode 100644 docs/multicluster/upgrade.md diff --git a/docs/multicluster/api.md b/docs/multicluster/api.md new file mode 100644 index 00000000000..f61a584d8b0 --- /dev/null +++ b/docs/multicluster/api.md @@ -0,0 +1,25 @@ +# Antrea Multi-cluster API + +This document lists all the API resource versions currently supported by Antrea Mulit-cluster. + +Antrea Multi-cluster is supported since v1.5.0. Most Custom Resource Definitions (CRDs) +used by Antrea Multi-cluster are in the API group `multicluster.crd.antrea.io`, and +two CRDs from [mcs-api](https://github.com/kubernetes-sigs/mcs-api) are in group `multicluster.x-k8s.io` +which is defined by Kubernetes upstream [KEP-1645](https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api). + +## CRDs in `multicluster.crd.antrea.io` + +| CRD | CRD version | Introduced in | Deprecated in / Planned Deprecation | Planned Removal | +| ------------------------ | ----------- | ------------- | ----------------------------------- | --------------- | +| `ClusterClaims` | v1alpha1 | v1.5.0 | N/A | N/A | +| `ClusterSets` | v1alpha1 | v1.5.0 | N/A | N/A | +| `MemberClusterAnnounces` | v1alpha1 | v1.5.0 | N/A | N/A | +| `ResourceExports` | v1alpha1 | v1.5.0 | N/A | N/A | +| `ResourceImports` | v1alpha1 | v1.5.0 | N/A | N/A | + +## CRDs in `multicluster.x-k8s.io` + +| CRD | CRD version | Introduced in | Deprecated in / Planned Deprecation | Planned Removal | +| ---------------- | ----------- | ------------- | ----------------------------------- | --------------- | +| `ServiceExports` | v1alpha1 | v1.5.0 | N/A | N/A | +| `ServiceImports` | v1alpha1 | v1.5.0 | N/A | N/A | diff --git a/docs/multicluster/upgrade.md b/docs/multicluster/upgrade.md new file mode 100644 index 00000000000..d2f813ed786 --- /dev/null +++ b/docs/multicluster/upgrade.md @@ -0,0 +1,93 @@ +# Antrea Multi-cluster Upgrade Guide + +The Antrea Multi-cluster feature is introduced from v1.5.0. There is no data-plane +related changes from release v1.5.0, so Antrea deployment and Antrea Multi-cluster +deployment are indenpendent. However, we suggest to keep Antrea and Antrea Multi-cluster +in the same version considering there will be data-plane change involved in the future. +Please refer to [Antrea upgrade and supported version skew](../versioning.md#antrea-upgrade-and-supported-version-skew) +to learn the requirement of Antrea upgrade. This doc focuses on Multi-cluster deployment only. + +The goal is to support 'graceful' upgrade. Multi-cluster upgrade will not have disruption +to data-plane of member clusters, but there can be downtime of processing new configurations +when individual components restart: + +- During Leader Controller restart, new member cluster, ClusterSet, ClusterClaim or + ResourceExport will not be processed. This is because the Controller also runs the validation + webhooks for MemberClusterAnnounce, ClusterSet, ClusterClaim and ResourceExport. +- During Member Controller restart, new ClusterSet or ClusterClaim will not be processed. + This is because the Controller runs the validation webhooks for ClusterSet and ClusterClaim. + +Our goal is to support version skew for different Antrea Multi-cluster components, but the +Multi-cluster feature is still in Alpha version, and the API is not stable yet. Our recommendation +is always to upgrade Antrea Multi-cluster to the same version for a ClusterSet. + +- **Antrea Leader Controller**: must be upgraded first +- **Antrea Member Controller**: must the same version as the **Antrea Leader Controller**. +- **Antctl**: must not be newer than the **Antrea Leader/Member Controller**. Please + notice Antctl for Multi-cluster is added since v1.6.0. + +## Upgrade in one ClusterSet + +In one ClusterSet, We recommend all member and leader clusters deployed with the same version. +During Leader controller upgrade, resource export/import between member clusters is not +supported. Before all member clusters are upgraded to the same version as Leader controller, +the feature introduced in old version should still work cross clusters, but no guarantee +for the feature in new version. + +It should have no impact during upgrade to those imported resources like Service, Endpoints +or AntreaClusterNetworkPolicy. + +## APIs deprecation policy + +The Antrea Multi-cluster APIs are built using K8s CustomResourceDefinitions and we +follow the same versioning scheme as the K8s APIs and the same [deprecation policy](https://kubernetes.io/docs/reference/using-api/deprecation-policy/). + +Other than the most recent API versions in each track, older API versions must be +supported after their announced deprecation for a duration of no less than: + +- GA: 12 months +- Beta: 9 months +- Alpha: N/A (can be removed immediately) + +K8s has a [moratorium](https://github.com/kubernetes/kubernetes/issues/52185) on the +removal ofAPI object versions that have been persisted to storage. We adopt the following +rules for the CustomResources which are persisted by the K8s apiserver. + +- Alpha API versions may be removed at any time. +- The [`deprecated` field](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#version-deprecation) must be used for CRDs to indicate that a particular version of + the resource has been deprecated. +- Beta and GA API versions must be supported after deprecation for the respective + durations stipulated above before they can be removed. +- For deprecated Beta and GA API versions, a [conversion webhook](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#webhook-conversion) must be provided along with + each Antrea release, until the API version is removed altogether. + +## Supported K8s versions + +Please refer to [Supported K8s versions](../versioning.md#supported-k8s-versions) +to learn the details. + +## Feature list + +Following is the Antrea Multi-cluster feature list. For the details of each feature, +please refer to [Antrea Multi-cluster Architecture](./architecture.md). + +| Feature | Supported in | +| -------------------------------- | ------------ | +| Service Export/Import | v1.5.0 | +| ClusterNetworkPolicy Replication | v1.6.0 | + +## Known Issues + +When you are trying to directly apply a newer Antrea Multi-cluster YAML manifest, as +provided with [an Antrea release](https://github.com/antrea-io/antrea/releases), you will +probably meet an issue like below if you are upgrading Multi-cluster components +from v1.5.0 to a newer one: + +```log +label issue:The Deployment "antrea-mc-controller" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"antrea", "component":"antrea-mc-controller"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable +``` + +The issue is caused by the label change introduced by [PR3266](https://github.com/antrea-io/antrea/pull/3266). +The reason is mutation of label selectors on Deployments is not allowed in `apps/v1beta2` +and forward. You need to delete the Deployment "antrea-mc-controller" first, then run +`kubectl apply -f` with the manifest of the newer version. diff --git a/docs/multicluster/user-guide.md b/docs/multicluster/user-guide.md index f288ea6ed62..e09bc778fc6 100644 --- a/docs/multicluster/user-guide.md +++ b/docs/multicluster/user-guide.md @@ -145,7 +145,7 @@ We recommend creating one ServiceAccount for each member for fine-grained access 2. Copy the access token into the member cluster `test-cluster-east`. E.g. ```bash - kubectl get secret member-east-access-token -n antrea-mcs-ns -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/ namespace: antrea-mcs-ns/namespace: kube-system/g' > member-east-access-token.yml + kubectl get secret member-east-access-token -n antrea-mcs-ns -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-mcs-ns/kube-system/g' > member-east-access-token.yml ``` 3. Replace all `east` to `west` and repeat step 1/2 for the other member cluster `test-cluster-west` diff --git a/hack/.notableofcontents b/hack/.notableofcontents index 05999a08968..f17e9c61686 100644 --- a/hack/.notableofcontents +++ b/hack/.notableofcontents @@ -27,7 +27,9 @@ docs/maintainers/build-kubemark.md docs/maintainers/getting-started-gif.md docs/maintainers/release.md docs/maintainers/updating-ovs-windows.md +docs/multicluster/api.md docs/multicluster/architecture.md +docs/multicluster/upgrade.md docs/multicluster/user-guide.md docs/network-requirements.md docs/noencap-hybrid-modes.md