Skip to content

Commit

Permalink
doc: add en statefulset (spidernet-io#2381)
Browse files Browse the repository at this point in the history 
* doc: add en statefulset

* Update statefulset-zh_CN.md
  • Loading branch information
@Jeanine-tw
Jeanine-tw authored Oct 11, 2023
1 parent 207e181 commit 0b370dd
Show file tree
Hide file tree
Showing 2 changed files with 191 additions and 129 deletions.
48 changes: 23 additions & 25 deletions docs/usage/statefulset-zh_CN.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,12 @@

Deployment 和 StatefulSet 控制器，对于 IP 地址固定的需求是不一样的：

1. 对于 StatefulSet ，Pod 副本重启前后，其 Pod 名保持不变，但是 Pod UUID 发生了变化，其是有状态的，应用管理员希望该 Pod 重启前后，仍能分配到相同的 IP 地址。
- 对于 StatefulSet，Pod 副本重启前后，其 Pod 名保持不变，但是 Pod UUID 发生了变化，其是有状态的，应用管理员希望该 Pod 重启前后，仍能分配到相同的 IP 地址。

2. 对于 Deployment ，Pod 副本重启前后，其 Pod 名字和 Pod UUID 都发生了变化，所以是无状态的，因此并不要新老交替的 Pod 使用相同的 IP 地址，我们可能只希望 Deployment 中所有副本所使用的 IP 是固定在某个 IP 范围内即可。
- 对于 Deployment，Pod 副本重启前后，其 Pod 名字和 Pod UUID 都发生了变化，所以是无状态的，因此并不要新老交替的 Pod 使用相同的 IP 地址，我们可能只希望 Deployment 中所有副本所使用的 IP 是固定在某个 IP 范围内即可。

开源社区的众多 CNI 方案并不能很好的支持 StatefulSet 的固定 IP 的需求。而 Spiderpool 提供的 StatefulSet 方案，能够保证 StatefulSet Pod 在重启、重建场景下，持续获取到相同的 IP 地址。

> NOTE：
>
> - 该功能默认开启。若开启，无任何限制，StatefulSet 可通过有限 IP 地址集合的 IP 池来固化 IP 的范围，但是，无论 StatefulSet 是否使用固定的 IP 池，它的 Pod 都可以持续分配到相同 IP。若关闭，StatefulSet 应用将被当做无状态对待，使用 Helm 安装 Spiderpool 时，可以通过 `--set ipam.enableStatefulSet=false` 关闭。
>
> - 在 StatefulSet 副本经由`缩容`到`扩容`的变化过程中，Spiderpool 并不保证新扩容 Pod 能够获取到之前缩容 Pod 的 IP 地址。
Expand All @@ -34,30 +32,30 @@ Deployment 和 StatefulSet 控制器，对于 IP 地址固定的需求是不一

### 安装 Spiderpool

- 通过 helm 安装 Spiderpool。
1. 通过 Helm 安装 Spiderpool。

```bash
helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set multus.multusCNI.defaultCniCRName="macvlan-ens192"
```
```bash
helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set multus.multusCNI.defaultCniCRName="macvlan-ens192"
```

> 如果您所在地区是中国大陆，可以指定参数 `--set global.imageRegistryOverride=ghcr.m.daocloud.io` ，以帮助您更快的拉取镜像。
>
> 通过 `multus.multusCNI.defaultCniCRName` 指定集群的 Multus clusterNetwork，clusterNetwork 是 Multus 插件的一个特定字段，用于指定 Pod 的默认网络接口。
> 如果您所在地区是中国大陆，可以指定参数 `--set global.imageRegistryOverride=ghcr.m.daocloud.io` ，以帮助您更快的拉取镜像。
>
> 通过 `multus.multusCNI.defaultCniCRName` 指定集群的 Multus clusterNetwork，clusterNetwork 是 Multus 插件的一个特定字段，用于指定 Pod 的默认网络接口。

- 检查安装完成
2. 检查安装完成

```bash
~# kubectl get po -n kube-system | grep spiderpool
NAME READY STATUS RESTARTS AGE
spiderpool-agent-7hhkz 1/1 Running 0 13m
spiderpool-agent-kxf27 1/1 Running 0 13m
spiderpool-controller-76798dbb68-xnktr 1/1 Running 0 13m
spiderpool-init 0/1 Completed 0 13m
spiderpool-multus-7vkm2 1/1 Running 0 13m
spiderpool-multus-rwzjn 1/1 Running 0 13m
```
```bash
~# kubectl get po -n kube-system | grep spiderpool
NAME READY STATUS RESTARTS AGE
spiderpool-agent-7hhkz 1/1 Running 0 13m
spiderpool-agent-kxf27 1/1 Running 0 13m
spiderpool-controller-76798dbb68-xnktr 1/1 Running 0 13m
spiderpool-init 0/1 Completed 0 13m
spiderpool-multus-7vkm2 1/1 Running 0 13m
spiderpool-multus-rwzjn 1/1 Running 0 13m
```

### 安装 CNI 配置

Expand Down Expand Up @@ -113,7 +111,7 @@ EOF

### 创建 StatefulSet 应用

以下的示例 Yaml 中， 会创建 2 个副本的 StatefulSet 应用 ，其中：
以下的示例 Yaml 中，会创建 2 个副本的 StatefulSet 应用，其中：

- `ipam.spidernet.io/ippool`：用于指定 Spiderpool 的 IP 池，Spiderpool 会自动在该池中选择一些 IP 与应用形成绑定，实现 StatefulSet 应用的 IP 固定效果。

Expand Down
272 changes: 168 additions & 104 deletions docs/usage/statefulset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,144 +2,208 @@

**English** ｜ [**简体中文**](./statefulset-zh_CN.md)

## Description
## Introduction

The spiderpool supports IP assignment for StatefulSet.
*Spiderpool ensures that StatefulSet Pods consistently retain the same IP address, even in scenarios such as restarts or rebuilds.*

When the number of statefulset replicas is not scaled up or down, all Pods could hold same IP address even when the Pod is restarting or rebuilding.
## StatefulSet features

* Pod restarts
The requirements for fixed IP address differ between Deployment and StatefulSet:

Once a Pod restarts (its pause container restarts), Spiderpool will keep use the previous IP,
and change the IPPool CR property ContainerID with the new pause container ID.
- For StatefulSet, the Pod's name remains the same throughout Pod restarts despite its changed UUID. As the Pod is stateful, application administrators hope that each Pod continues to be assigned the same IP address after restarts.

In the meanwhile, spiderendpoint will still keep the previous IP but refresh the ContainerID property.
- For Deployment, both the Pod name and its UUID change after restarts. Deployment Pods are stateless, so there is no need to maintain the same IP address between Pod restarts. Instead, administrators often prefer IP addresses to be allocated within a specified range for all replicas in Deployment.

* Pod is deleted and re-created
Many open-source CNI solutions provide limited support for fixing IP addresses for StatefulSet. However, Spiderpool's StatefulSet solution guarantees consistent allocation of the same IP address to the Pods during restarts and rebuilds.

After deleting a StatefulSet Pod, kubernetes will re-create a Pod with the same name.
> - This feature is enabled by default. When it is enabled, StatefulSet Pods can be assigned fixed IP addresses from a specified IP pool range. Whether or not using a fixed IP pool, StatefulSet Pods will consistently receive the same IP address. StatefulSet applications will be treated as stateless if the feature is disabled. You can disable it during the installation of Spiderpool using Helm via the flag `--set ipam.enableStatefulSet=false`.
>
> - During the transition from scaling down to scaling up StatefulSet replicas, Spiderpool does not guarantee that new Pods will inherit the IP addresses previously used by the scaled-down Pods.
>
> - Currently, when a StatefulSet Pod is running, modifying the StatefulSet annotation to specify a different IP pool and restarting the Pod will not cause the Pod IP addresses to be allocated from the new IP pool range. Instead, the Pod will continue using their existing fixed IP addresses.
In this case, Spiderpool will also keep the previous IP and update the ContainerID.
## Prerequisites

### Notice
1. A ready Kubernetes cluster.

* Currently, it's not allowed to change StatefulSet annotation for using another pool when a StatefulSet is ready and its Pods are running.
2. [Helm](https://helm.sh/docs/intro/install/) has already been installed.

* When the statefulset is scaled down and then scaled up, the scaled-up Pod is not guaranteed to get the previous IP.
## Steps

* The [IP-GC](./gc.md) feature (reclaim IP for the Pod of graceful-period timeout) does work for StatefulSet Pod.
### Install Spiderpool

## Get Started
1. Install Spiderpool through Helm:

### Enable StatefulSet support
```bash
helm repo add spiderpool https://spidernet-io.github.io/spiderpool
helm repo update spiderpool
helm install spiderpool spiderpool/spiderpool --namespace kube-system --set multus.multusCNI.defaultCniCRName="macvlan-ens192"
```

Firstly, please ensure you have installed the spiderpool and configure the CNI file. Refer to [install](./install/install.md) for details.
> For users in the Chinese mainland, it is recommended to specify the spec `--set global.imageRegistryOverride=ghcr.m.daocloud.io` to accelerate image pulling.
>
> You can specify the Multus clusterNetwork for your cluster using `multus.multusCNI.defaultCniCRName`. The clusterNetwork is a specific field of the Multus plugin that defines the default network interface for Pods.

Check whether the property `enableStatefulSet` of the configmap `spiderpool-conf` is already set to `true` or not.
2. Check if the installation is successful.

```shell
kubectl -n kube-system get configmap spiderpool-conf -o yaml
```bash
~# kubectl get po -n kube-system | grep spiderpool
NAME READY STATUS RESTARTS AGE
spiderpool-agent-7hhkz 1/1 Running 0 13m
spiderpool-agent-kxf27 1/1 Running 0 13m
spiderpool-controller-76798dbb68-xnktr 1/1 Running 0 13m
spiderpool-init 0/1 Completed 0 13m
spiderpool-multus-7vkm2 1/1 Running 0 13m
spiderpool-multus-rwzjn 1/1 Running 0 13m
```

### Install CNI

To simplify the creation of JSON-formatted Multus CNI configuration, Spiderpool introduces the SpiderMultusConfig CR, which automates the management of Multus NetworkAttachmentDefinition CRs. Here is an example of creating a Macvlan SpiderMultusConfig:

- master: the interface `ens192` is used as the spec for master.

```bash
MACVLAN_MASTER_INTERFACE="ens192"
MACVLAN_MULTUS_NAME="macvlan-$MACVLAN_MASTER_INTERFACE"
cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderMultusConfig
metadata:
name: ${MACVLAN_MULTUS_NAME}
namespace: kube-system
spec:
cniType: macvlan
enableCoordinator: true
macvlan:
master:
- ${MACVLAN_MASTER_INTERFACE}
EOF
```

If you want to set it `true`, run `helm upgrade spiderpool spiderpool/spiderpool --set ipam.enableStatefulSet=true -n kube-system`.
With the provided configuration, we create the following Macvlan SpiderMultusConfig that will automatically generate a Multus NetworkAttachmentDefinition CR.

### Create a StatefulSet
```bash
~# kubectl get spidermultusconfigs.spiderpool.spidernet.io -n kube-system
NAME AGE
macvlan-ens192 26m
This is an example to install a StatefulSet.
~# kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kube-system
NAME AGE
macvlan-ens192 27m
```

```shell
kubectl apply -f https://raw.githubusercontent.com/spidernet-io/spiderpool/main/docs/example/statefulset/statefulset-demo.yaml
### Create an IP pool

```bash
~# cat <<EOF | kubectl apply -f -
apiVersion: spiderpool.spidernet.io/v2beta1
kind: SpiderIPPool
metadata:
name: test-ippool
spec:
subnet: 10.6.0.0/16
ips:
- 10.6.168.101-10.6.168.110
EOF
```

### Validate the Spiderpool related CR data

Here's the created Pod, spiderippool, and spiderendpoint CR information:

```text
$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo-sts-0 1/1 Running 0 8s 172.22.40.181 spider-worker <none> <none>
---------------------------------------------------------------------------------------------------------------------
$ kubectl get sp default-v4-ippool -o yaml
...
"172.22.40.181":{"interface":"eth0","pod":"default/demo-sts-0","podUid":"fa50a7c2-99e7-4e97-a3f1-8d6503067b54"}
...
---------------------------------------------------------------------------------------------------------------------
$ kubectl get se demo-sts-0 -o yaml
...
status:
current:
ips:
- interface: eth0
ipv4: 172.22.40.181/16
ipv4Pool: default-v4-ippool
ipv6: fc00:f853:ccd:e793:f::d/64
ipv6Pool: default-v6-ippool
vlan: 0
node: spider-worker
uid: fa50a7c2-99e7-4e97-a3f1-8d6503067b54
ownerControllerName: demo-sts
ownerControllerType: StatefulSet
...
### Create StatefulSet applications

The following YAML example creates a StatefulSet application with 2 replicas:

- `ipam.spidernet.io/ippool`: specify the IP pool for Spiderpool. Spiderpool automatically selects some IP addresses from this pool and bind them to the application, ensuring that the IP addresses remain fixed for the StatefulSet application.

- `v1.multus-cni.io/default-network`: create a default network interface for the application.

```bash
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test-sts
spec:
replicas: 2
selector:
matchLabels:
app: test-sts
template:
metadata:
annotations:
ipam.spidernet.io/ippool: |-
{
"ipv4": ["test-ippool"]
}
v1.multus-cni.io/default-network: kube-system/macvlan-ens192
labels:
app: test-sts
spec:
containers:
- name: test-sts
image: nginx
imagePullPolicy: IfNotPresent
EOF
```

Try to delete Pod `demo-sts-0` and check whether the rebuilding Pod keeps the previous IP or not.

```text
$ kubectl delete po demo-sts-0
pod "demo-sts-0" deleted
---------------------------------------------------------------------------------------------------------------------
$ kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
demo-sts-0 1/1 Running 0 12s 172.22.40.181 spider-worker <none> <none>
---------------------------------------------------------------------------------------------------------------------
$ kubectl get sp default-v4-ippool -o yaml
...
"172.22.40.181":{"interface":"eth0","pod":"default/demo-sts-0","podUid":"425d6552-63bb-4b4c-aab2-b2db95de0ab1"}
...
---------------------------------------------------------------------------------------------------------------------
$ kubectl get se demo-sts-0 -o yaml
...
status:
current:
ips:
- interface: eth0
ipv4: 172.22.40.181/16
ipv4Pool: default-v4-ippool
ipv6: fc00:f853:ccd:e793:f::d/64
ipv6Pool: default-v6-ippool
vlan: 0
node: spider-worker
uid: 425d6552-63bb-4b4c-aab2-b2db95de0ab1
ownerControllerName: demo-sts
ownerControllerType: StatefulSet
...
When the StatefulSet application is created, Spiderpool will select a random set of IP addresses from the specified IP pool and bind them to the application.

```bash
~# kubectl get spiderippool
NAME VERSION SUBNET ALLOCATED-IP-COUNT TOTAL-IP-COUNT DEFAULT
test-ippool 4 10.6.0.0/16 2 10 false
~# kubectl get po -l app=test-sts -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-sts-0 1/1 Running 0 3m13s 10.6.168.105 node2 <none> <none>
test-sts-1 1/1 Running 0 3m12s 10.6.168.102 node1 <none> <none>
```

And you can see, the re-created Pod still holds the previous IP, spiderippool, and spiderendpoint updated containerID property.
Upon restarting StatefulSet Pods, it is observed that each Pod retains its assigned IP address.

```bash
~# kubectl get pod | grep "test-sts" | awk '{print $1}' | xargs kubectl delete pod
pod "test-sts-0" deleted
pod "test-sts-1" deleted
### clean up
~# kubectl get po -l app=test-sts -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-sts-0 1/1 Running 0 18s 10.6.168.105 node2 <none> <none>
test-sts-1 1/1 Running 0 17s 10.6.168.102 node1 <none> <none>
```

Upon scaling up or down the StatefulSet Pods, the IP addresses of each Pod change as expected.

Delete the StatefulSet object: `demo-sts`.
```bash
~# kubectl scale deploy test-sts --replicas 3
statefulset.apps/test-sts scaled
```text
$ kubectl delete sts demo-sts
statefulset.apps "demo-sts" deleted
~# kubectl get po -l app=test-sts -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-sts-0 1/1 Running 0 4m58s 10.6.168.105 node2 <none> <none>
test-sts-1 1/1 Running 0 4m57s 10.6.168.102 node1 <none> <none>
test-sts-2 1/1 Running 0 4s 10.6.168.109 node2 <none> <none>
---------------------------------------------------------------------------------------------------------------------
~# kubectl get pod | grep "test-sts" | awk '{print $1}' | xargs kubectl delete pod
pod "test-sts-0" deleted
pod "test-sts-1" deleted
pod "test-sts-2" deleted
$ kubectl get sp default-v4-ippool -o yaml | grep demo-sts-0
~# kubectl get po -l app=test-sts -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-sts-0 1/1 Running 0 6s 10.6.168.105 node2 <none> <none>
test-sts-1 1/1 Running 0 4s 10.6.168.102 node1 <none> <none>
test-sts-2 1/1 Running 0 3s 10.6.168.109 node2 <none> <none>
---------------------------------------------------------------------------------------------------------------------
~# kubectl scale sts test-sts --replicas 2
statefulset.apps/test-sts scaled
$ kubectl get se demo-sts-0 -o yaml
Error from server (NotFound): spiderendpoints.spiderpool.spidernet.io "demo-sts-0" not found
~# kubectl get po -l app=test-sts -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test-sts-0 1/1 Running 0 88s 10.6.168.105 node2 <none> <none>
test-sts-1 1/1 Running 0 86s 10.6.168.102 node1 <none> <none>
```

The related data is cleaned up now.
## Conclusion

Spiderpool ensures that StatefulSet Pods maintain a consistent IP address even during scenarios like restarts or rebuilds, satisfying the requirement for fixed IP addresses in StatefulSet.

0 comments on commit 0b370dd

Please sign in to comment.