Update Multi-cluster architecture doc #3638

luolanzone · 2022-04-14T06:07:37Z

Update MC architecture doc and change old diagrams' format from png to svg.

Signed-off-by: Lan Luo [email protected]

codecov-commenter · 2022-04-14T08:09:04Z

Codecov Report

Merging #3638 (5894ae2) into main (ea99ee5) will decrease coverage by 2.71%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3638      +/-   ##
==========================================
- Coverage   63.37%   60.66%   -2.72%     
==========================================
  Files         288      293       +5     
  Lines       41223    43224    +2001     
==========================================
+ Hits        26127    26222      +95     
- Misses      13000    14854    +1854     
- Partials     2096     2148      +52

Flag	Coverage Δ
kind-e2e-tests	`44.88% <ø> (-5.19%)`	⬇️
unit-tests	`44.56% <ø> (+0.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...g/agent/apiserver/handlers/featuregates/handler.go	`4.54% <0.00%> (-77.28%)`	⬇️
pkg/apis/controlplane/v1beta2/helper.go	`40.00% <0.00%> (-60.00%)`	⬇️
...kg/apiserver/registry/system/supportbundle/rest.go	`22.17% <0.00%> (-50.44%)`	⬇️
pkg/support/dump.go	`7.90% <0.00%> (-49.16%)`	⬇️
pkg/support/dump_others.go	`0.00% <0.00%> (-44.74%)`	⬇️
pkg/agent/controller/networkpolicy/fqdn.go	`39.51% <0.00%> (-36.00%)`	⬇️
...g/agent/apiserver/handlers/addressgroup/handler.go	`5.00% <0.00%> (-35.00%)`	⬇️
...agent/apiserver/handlers/appliedtogroup/handler.go	`5.00% <0.00%> (-35.00%)`	⬇️
pkg/agent/controller/networkpolicy/reject.go	`50.58% <0.00%> (-34.71%)`	⬇️
...kg/agent/controller/networkpolicy/audit_logging.go	`56.30% <0.00%> (-27.74%)`	⬇️
... and 75 more

docs/multicluster/architecture.md

docs/multicluster/user-guide.md

jianjuns

I think we generally use only SVG diagram in the repo.

docs/multicluster/architecture.md

luolanzone · 2022-05-31T09:27:01Z

Hi @jianjuns I changed a little bit for MC doc, Could you review again? thanks.

jianjuns

Remove on-prem, public, etc. from the diagrams.

docs/multicluster/user-guide.md

jianjuns · 2022-06-01T01:04:17Z

@luolanzone : the current version of user guide looks too complex to follow. I think let us split the PR to two, one for architecture doc, one for user guide, and let me work together with you on the latter.

luolanzone · 2022-06-09T07:41:59Z

@jianjuns I removed this PR from release v1.7, but could you help to review and check if it's Ok to merge in this release? thanks.

jianjuns · 2022-06-09T21:29:08Z

docs/multicluster/architecture.md

@@ -8,7 +8,7 @@ ClusterSet, and enforced in all member clusters.

 The diagram below depicts a basic multi-cluster topology in Antrea.

-<img src="assets/basic-topology.png" width="500" alt="Antrea Multi-cluster Topology">
+<img src="assets/basic-topology.svg" width="500" alt="Antrea Multi-cluster Topology">


The picture is too small.

Do not use on-prem / public, etc. They are not popular terms with K8s.

jianjuns · 2022-06-09T21:30:21Z

docs/multicluster/architecture.md

@@ -8,7 +8,7 @@ ClusterSet, and enforced in all member clusters.

 The diagram below depicts a basic multi-cluster topology in Antrea.


a basic Antrea Multi-cluster topology with one leader cluster and two member clusters.

jianjuns · 2022-06-09T22:03:13Z

docs/multicluster/architecture.md

@@ -57,7 +57,7 @@ and encapsulates them into ResourceImports.

 ## Service Export and Import

-<img src="assets/resource-export-import-pipeline.png" width="800" alt="Antrea Multi-cluster Resource Export/Import Pipeline">
+<img src="assets/resource-export-import-pipeline.svg" width="800" alt="Antrea Multi-cluster Resource Export/Import Pipeline">


The picture is a little small too (hard for me to read the texts).

In the picture, consider:

Endpoint -> Endpoints

Monitor -> Watch?

jianjuns · 2022-06-09T22:04:55Z

docs/multicluster/architecture.md

+will be responsible to forward all cross-clusters traffic from local cluster to other member clusters.
+Below is the basic architecture for Antrea Multi-cluster Service network connectivity.
+
+<img src="assets/mc-service-connectivity.svg" width="600" alt="Antrea Multi-cluster Service Connectivity">


Again, a little small.

We should remove "node-a1" around the tunnel line?

A minor one - the color of Cluster C box looks too different from that of Cluster A B. Could we adjust the colors?

yea, node-a1 is added by mistake, it's fixed, and color is adjusted.

jianjuns · 2022-06-09T22:08:03Z

docs/multicluster/architecture.md

@@ -106,3 +108,52 @@ Multi-cluster admins can also specify certain ClusterNetworkPolicies to be repli
 ClusterSet. The ACNP to be replicated should be created as a ResourceExport in the leader cluster, and
 the resource export/import pipeline will ensure member clusters receive this ACNP spec to be replicated.
 Each member cluster's Multi-cluster Controller will then create an ACNP in their respective clusters.
+
+## Antrea Multi-cluster Service Connectivity


I feel "Multi-cluster Service Connectivity" is not a good term for people to understand.

How about "Multi-cluster Gateway" or "Cross-cluster Connectivity".

Rename it to Multi-cluster Gateway"

jianjuns · 2022-06-09T22:09:18Z

docs/multicluster/architecture.md

+
+## Antrea Multi-cluster Service Connectivity
+
+Antrea Multi-cluster supports Service connectivity directly since 1.7.0. It introduces a new feature gate


I feel no need to mention feature gate or particular component configurations in the architecture doc.

jianjuns · 2022-06-10T00:53:50Z

docs/multicluster/architecture.md

@@ -84,14 +84,16 @@ been created by the Importer earlier.

 Antrea Multi-cluster Controller only supports Service of type ClusterIP at this moment. In
 order to support multi-cluster Service access between member clusters, Antrea requires member


Probably talk about the new behavior first, like:

"Since Antrea v1.7.0, Multi-cluster Gateways must be configured to support Multi-cluster Service access across member clusters, and Service CIDRs cannot overlap between clusters. Refer to xxx for more information. Before Antrea v1.7.0, Pod IPs must be directly reachable across clusters for Multi-cluster Service access, and Pod CIDRs cannot overlap between clusters.

jianjuns · 2022-06-10T00:57:31Z

docs/multicluster/architecture.md

@@ -84,14 +84,16 @@ been created by the Importer earlier.



I feel this section can be merged to the previous one. Probably just one section called "Antrea Multi-cluster Service", which first introduces what is Multi-cluster Service, and then the implementation, including Service export and import.

Please read your doc and make sure a reader with no knowledge on multi-cluster implementation can understand what you wrote.

The pipeline is introduced first considering both MC Service and MC network policy depend on this, I will check how to refine it. thanks.

Refined the whole structure and some descriptions.

docs/multicluster/architecture.md

jianjuns · 2022-06-10T00:59:32Z

docs/multicluster/architecture.md

@@ -8,7 +8,7 @@ ClusterSet, and enforced in all member clusters.

 The diagram below depicts a basic multi-cluster topology in Antrea.

-<img src="assets/basic-topology.png" width="500" alt="Antrea Multi-cluster Topology">
+<img src="assets/basic-topology.svg" width="500" alt="Antrea Multi-cluster Topology">

 Given a set of Kubernetes clusters, there will be a leader cluster and several member clusters.


An Antrea Multi-cluster ClusterSet includes a leader clusters and multiple member clusters.

jianjuns · 2022-06-10T00:59:55Z

docs/multicluster/architecture.md

@@ -8,7 +8,7 @@ ClusterSet, and enforced in all member clusters.

 The diagram below depicts a basic multi-cluster topology in Antrea.

-<img src="assets/basic-topology.png" width="500" alt="Antrea Multi-cluster Topology">
+<img src="assets/basic-topology.svg" width="500" alt="Antrea Multi-cluster Topology">

 Given a set of Kubernetes clusters, there will be a leader cluster and several member clusters.
 By default, a leader cluster itself is also a member cluster of a ClusterSet. A cluster


Why by default? Probably copy the description in user guide.

docs/multicluster/architecture.md

jianjuns · 2022-06-11T00:48:18Z

docs/multicluster/architecture.md


-<img src="assets/basic-topology.png" width="500" alt="Antrea Multi-cluster Topology">
+<img src="assets/basic-topology.svg" width="650" alt="Antrea Multi-cluster Topology">


Other diagrams use Cluster A/B/C. Probably we should change this one too to be consistent.

jianjuns · 2022-06-11T00:50:10Z

docs/multicluster/architecture.md

-Given a set of Kubernetes clusters, there will be a leader cluster and several member clusters.
-By default, a leader cluster itself is also a member cluster of a ClusterSet. A cluster
-can also be configured as a dedicated leader cluster of multiple ClusterSets.
+An Antrea Multi-cluster ClusterSet includes a leader cluster and multiple member clusters.


I feel better to move this paragraph to be before line 9.

jianjuns · 2022-06-11T01:00:34Z

docs/multicluster/architecture.md

@@ -32,35 +34,50 @@ interface for resource export/import that can be read/written by all member and
 in the ClusterSet. The Common Area is implemented with a Namespace in the leader cluster for a
 given ClusterSet.

+## Antrea Multi-cluster Service
+
+Since Antrea v1.7.0, Multi-cluster Gateways must be configured to support Multi-cluster Service


This section makes no sense to be here. And readers do not even know what is Multi-cluster Service from this doc so far.

Anyway, let us do some small adjustment in this PR, and revise later. Let us move this section to be after "Antrea Multi-cluster Controller". And in this section, let us first have the "Service Export and Import" sub-section; and add another sub-section called "Service Access Across Clusters", and move this paragraph there.

jianjuns · 2022-06-11T01:02:28Z

docs/multicluster/architecture.md

-any extra setting.
+## Antrea Multi-cluster Gateway
+
+Antrea Multi-cluster supports Service connectivity directly since 1.7.0. User can choose one of K8s Nodes


Antrea started to support Multi-cluster Gateway since v1.7.0.

jianjuns · 2022-06-11T01:06:38Z

docs/multicluster/architecture.md

+An Antrea Multi-cluster ClusterSet includes a leader cluster and multiple member clusters.
+Antrea Multi-cluster Controller needs to be deployed in the leader and all member
+clusters. A cluster can serve as the leader, and meanwhile also be a member cluster of the
+ClusterSet.

 ## Terminology



Fix the following:

member of a ClusterSet with unique cluster ID, to claim a ClusterSet with unique ClusterSet ID.

with a unique cluster ID

and to claim a ClusterSet with a unique ClusterSet ID.

It might be more natural to talk about ClusterSet first:

The ClusterClaim CRD is used to claim a ClusterSet with a unique ClusterSet ID, and to claim the cluster itself as a member of a ClusterSet with a unique cluster ID.

jianjuns · 2022-06-13T04:20:31Z

docs/multicluster/architecture.md

 validates the ClusterSet and Clusterclaim. For resource export/import, it watches ResourceExports
 and encapsulates them into ResourceImports.

-## Service Export and Import
+### Service Export and Import


I meant adding a section at the same level of "## Antrea Multi-cluster Controller" called "## Multi-cluster Controller Service", so the sections are like:

## Antrea Multi-cluster Controller ## Multi-cluster Controller Service ### Service Export and Import ### Service Access Across Clusters ## Multi-cluster Gateway ### Multi-cluster Service Traffic Walk ## Antrea Multi-cluster NetworkPolicy

What you think? BTW, I removed "Antrea" from a few section titles. I feel no need to repeat it in all places.

sure, I will refine it. thanks.

I suppose you mean ## Multi-cluster Service in the second item?

docs/multicluster/architecture.md

jianjuns · 2022-06-14T00:30:28Z

docs/multicluster/architecture.md

-In a member cluster, Antrea Multi-cluster creates a Deployment that runs Antrea Multi-cluster
-Controller which is responsible for exporting resource to and importing resource from a leader
-cluster in a ClusterSet.
+Antrea Multi-cluster Controller watches ClusterSet, ClutserClaim, Service, ServiceExport etc.


Remove "watches...". It is not more important than other responsibilities of MC Controller to mention first.

If you want, we can say "Antrea Multi-cluster Controller implements ClusterSet management and resource export/import in the ClusterSet".

jianjuns · 2022-06-14T00:42:41Z

docs/multicluster/architecture.md

-Controller which is responsible for exporting resource to and importing resource from a leader
-cluster in a ClusterSet.
+Antrea Multi-cluster Controller watches ClusterSet, ClutserClaim, Service, ServiceExport etc.
+It has different responsibilities in member and leader clusters.


In either a leader or a member cluster, Antrea Multi-cluster Controller is deployed with a Deployment of a single replica, but it takes different responsibilities in member and leader clusters.

jianjuns · 2022-06-14T00:44:55Z

docs/multicluster/architecture.md

-In a leader cluster, Antrea Multi-cluster creates a Deployment that runs Antrea Multi-cluster
-Controller which is responsible for converting resources from different member clusters into one
-encapsulated resource as long as these resources have the same kind and match Namespace sameness.
+In a member cluster, Antrea Multi-cluster creates a Deployment with one replica that runs


I feel we can remove this paragraph, as resource export/import is talked in a later one too.

jianjuns · 2022-06-14T00:45:21Z

docs/multicluster/architecture.md


-In ClusterSet initialization, Antrea Multi-cluster Controller in a member cluster watches
+In a leader cluster, Antrea Multi-cluster creates a Deployment with one replica that runs


jianjuns · 2022-06-14T00:46:31Z

docs/multicluster/architecture.md

+member clusters into one encapsulated resource as long as these resources have the same kind
+and match Namespace sameness.
+
+When ClusterSet is initialized in member cluster, Antrea Multi-cluster Controller watches


Maybe add a sub-section of "ClusterSet Initialization"

jianjuns · 2022-06-14T00:49:27Z

docs/multicluster/architecture.md

-and writes the ResourceExports to leader cluster. For resource importing, it watches ResourceImports
-from leader cluster, and creates multi-cluster Services and Endpoints with a prefix `antrea-mc-`
-plus exported Service name, and also ServiceImports which have the same name as the Service name.
+For resources exporting in the member cluster, Antrea Multi-cluster Controller watches ServiceExport,


This section is not well organized. I feel harder to explain all my suggestion than just writing down what is in my mind. Probably let me push a version for you to review.

jianjuns

@luolanzone : I updated a version. Please review.

jianjuns · 2022-06-14T05:33:17Z

docs/multicluster/architecture.md

+other member clusters through tunnels. The diagram below depicts Antrea
+Mulit-cluster connectivity with Multi-cluster Gateways.
+
+<img src="assets/mc-gateway.svg" width="800" alt="Antrea Multi-cluster Gateway">


Should we use "pod-a/b/c" to be consistent with "node-a1", etc.?

sure, I will update it.

@jianjuns the change looks good to me overall, I changed a few minor descriptions and the line length for Antrea Multi-cluster NetworkPolicy section as well, you can move forward if you feel it's ready.

Update the Multi-cluster architecture doc with descriptions about Multi-cluster Gateway, and change the diagrams' format from png to svg. Signed-off-by: Lan Luo <[email protected]> Co-authored-by: Jianjun Shen <[email protected]>

luolanzone · 2022-06-14T07:11:27Z

docs/multicluster/architecture.md

+ClusterSet and Clusterclaim CRs, and initializes the ClusterSet. It also validates
+the MemberClusterAnnounce CR created by a member cluster and updates the member
+cluster status to the ClusterSet.


Here is the main one I updated, we have a webhook for MemberClusterAnnounce in leader cluster, so I used validates here, and admin is the one to add/update member clusters to ClusterSet, Controller will validate and update status only.

jianjuns · 2022-06-14T16:10:22Z

/skip-all

…#3899: Revise Multi-cluster Gateway descriptions in user docs (#3908) * Update Multi-cluster architecture doc Update the Multi-cluster architecture doc with descriptions about Multi-cluster Gateway, and change the diagrams' format from png to svg. Signed-off-by: Lan Luo <[email protected]> Co-authored-by: Jianjun Shen <[email protected]> * Revise Multi-cluster Gateway descriptions in user docs Revise the descriptions about Multi-cluster Gateway configuration and multi-cluster Services. Add descriptions about the `gateway-ip` annotation. Remove unnecessary descriptions about implementations from the user documents. Signed-off-by: Jianjun Shen <[email protected]>

luolanzone requested review from jianjuns and Dyanngg April 14, 2022 06:07

luolanzone added the area/multi-cluster Issues or PRs related to multi cluster. label Apr 14, 2022

Dyanngg reviewed Apr 14, 2022

View reviewed changes

jianjuns reviewed Apr 15, 2022

View reviewed changes

docs/multicluster/architecture.md Outdated Show resolved Hide resolved

docs/multicluster/architecture.md Outdated Show resolved Hide resolved

docs/multicluster/architecture.md Outdated Show resolved Hide resolved

docs/multicluster/architecture.md Outdated Show resolved Hide resolved

jianjuns reviewed Apr 16, 2022

View reviewed changes

docs/multicluster/architecture.md Outdated Show resolved Hide resolved

luolanzone added this to the Antrea v1.7 release milestone Apr 28, 2022

luolanzone force-pushed the mc-dp-docs branch 2 times, most recently from 8829bf3 to dea9cdb Compare May 10, 2022 02:35

luolanzone changed the title ~~Add docs about Multi-cluster data plane change~~ Add docs about Multi-cluster Service connectivity May 10, 2022

luolanzone requested review from antoninbas and tnqn May 10, 2022 03:59

luolanzone force-pushed the mc-dp-docs branch from dea9cdb to 0559393 Compare May 11, 2022 02:41

luolanzone force-pushed the mc-dp-docs branch from 0559393 to ea4ef89 Compare May 31, 2022 09:25

jianjuns reviewed Jun 1, 2022

View reviewed changes

docs/multicluster/user-guide.md Outdated Show resolved Hide resolved

docs/multicluster/user-guide.md Outdated Show resolved Hide resolved

docs/multicluster/user-guide.md Outdated Show resolved Hide resolved

jianjuns reviewed Jun 1, 2022

View reviewed changes

docs/multicluster/user-guide.md Outdated Show resolved Hide resolved

docs/multicluster/user-guide.md Outdated Show resolved Hide resolved

luolanzone force-pushed the mc-dp-docs branch 2 times, most recently from 9e04232 to a23f5a7 Compare June 9, 2022 07:40

luolanzone removed this from the Antrea v1.7 release milestone Jun 9, 2022

luolanzone force-pushed the mc-dp-docs branch from a23f5a7 to 9012499 Compare June 9, 2022 07:42

luolanzone changed the title ~~Add docs about Multi-cluster Service connectivity~~ Update MC architecture doc Jun 9, 2022

jianjuns reviewed Jun 9, 2022

View reviewed changes

jianjuns reviewed Jun 10, 2022

View reviewed changes

luolanzone force-pushed the mc-dp-docs branch 2 times, most recently from b7c1c0f to 9fc80ff Compare June 10, 2022 03:23

jianjuns changed the title ~~Update MC architecture doc~~ Update Multi-cluster architecture doc Jun 10, 2022

jianjuns reviewed Jun 11, 2022

View reviewed changes

luolanzone force-pushed the mc-dp-docs branch from 9fc80ff to b13e13f Compare June 13, 2022 01:44

jianjuns reviewed Jun 13, 2022

View reviewed changes

luolanzone force-pushed the mc-dp-docs branch from b13e13f to 068d9e5 Compare June 13, 2022 05:11

jianjuns reviewed Jun 14, 2022

View reviewed changes

jianjuns force-pushed the mc-dp-docs branch from 068d9e5 to 3a5afbc Compare June 14, 2022 05:31

jianjuns reviewed Jun 14, 2022

View reviewed changes

Update Multi-cluster architecture doc

9e34e8a

Update the Multi-cluster architecture doc with descriptions about Multi-cluster Gateway, and change the diagrams' format from png to svg. Signed-off-by: Lan Luo <[email protected]> Co-authored-by: Jianjun Shen <[email protected]>

luolanzone force-pushed the mc-dp-docs branch from 3a5afbc to 9e34e8a Compare June 14, 2022 07:04

luolanzone commented Jun 14, 2022

View reviewed changes

Fixes of typos & formatting and a few minor revisions

5894ae2

jianjuns approved these changes Jun 14, 2022

View reviewed changes

jianjuns merged commit 56e9e4a into antrea-io:main Jun 14, 2022

jianjuns mentioned this pull request Jun 16, 2022

Automated cherry pick of #3638: Update Multi-cluster architecture doc #3899: Revise Multi-cluster Gateway descriptions in user docs #3908

Merged

		@@ -8,7 +8,7 @@ ClusterSet, and enforced in all member clusters.

		The diagram below depicts a basic multi-cluster topology in Antrea.


		## Antrea Multi-cluster Service Connectivity

		Antrea Multi-cluster supports Service connectivity directly since 1.7.0. It introduces a new feature gate

		@@ -84,14 +84,16 @@ been created by the Importer earlier.

		Antrea Multi-cluster Controller only supports Service of type ClusterIP at this moment. In
		order to support multi-cluster Service access between member clusters, Antrea requires member


		<img src="assets/basic-topology.png" width="500" alt="Antrea Multi-cluster Topology">
		<img src="assets/basic-topology.svg" width="650" alt="Antrea Multi-cluster Topology">


		In ClusterSet initialization, Antrea Multi-cluster Controller in a member cluster watches
		In a leader cluster, Antrea Multi-cluster creates a Deployment with one replica that runs

Update Multi-cluster architecture doc #3638

Update Multi-cluster architecture doc #3638

Conversation

luolanzone commented Apr 14, 2022 • edited Loading

codecov-commenter commented Apr 14, 2022 • edited Loading

Codecov Report

jianjuns left a comment

Choose a reason for hiding this comment

luolanzone commented May 31, 2022

jianjuns left a comment

Choose a reason for hiding this comment

jianjuns commented Jun 1, 2022

luolanzone commented Jun 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone Jun 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

jianjuns commented Jun 14, 2022

luolanzone commented Apr 14, 2022 •

edited

Loading

codecov-commenter commented Apr 14, 2022 •

edited

Loading

luolanzone Jun 10, 2022 •

edited

Loading

luolanzone Jun 14, 2022 •

edited

Loading