Add k8s master related code #14870

lixiaoyuner · 2023-04-27T09:20:03Z

Why I did it

Currently, k8s master image is generated from a separate branch which we created by ourselves, not release ones. We need to commit these k8s master related code to master branch for a better way to do k8s master image build out.

Work item tracking

Microsoft ADO (number only):
19998138

How I did it

Install k8s dashboard docker images
Install geneva mds and mdsd and fluentd docker images and tag them as latest, tagging latest will help create container always with the latest version
Install azure-storage-blob and azure-identity, this will help do etcd backup and restore.
Install kubernetes python client packages, this will help read worker and container state, we can send these metric to Geneva.
Remove mdm debian package, will replace it with the mdm docker image
Add k8s master entrance script, this script will be called by rc-local service when system startup. we have some master systemd services in compute-move repo, when VMM service create master VM, VMM will copy all master service files inside VM, the entrance script will setup all services according to the service files.
When the entrance script content changed, the PR build will set include_kubernetes_master=y to help do validation for k8s master related code change. The default value of include_kubernetes_master should be always n for public master branch. We will generate master image from internal master branch

How to verify it

Build with INCLUDE_KUBERNETES_MASTER = y

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

liushilongbuaa

pipeline change looks good.

.azure-pipelines/template-skipvstest.yml

lguohan · 2023-06-30T16:03:15Z

we need much better description for the pr. in the pr description, please document what you did, what you added. for exmaple you added geneva agent, it should be in the description instead asking reviewer to figure it out in the code review.

lguohan · 2023-06-30T16:05:01Z

files/build_templates/sonic_debian_extension.j2

+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION}
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:latest
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION}
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:latest


lots of packages installed, need to how how much disk space consumed. need to measure.

lots of packages installed, need to how how much disk space consumed. need to measure.

k8s master image will run on PilotFish host as a VM, no need to care about the disk space. We will not include this k8s master feature in any released sonic image.

More details:
For this k8s master feature, we have a include_kubernetes_master flag in rules/config to decide whether install all these k8s master packages or not. We plan to build the k8s master image from internal master branch on demand (maybe run official pipeline for internal master branch by passing include_kubernetes_master=y manually, because the master image will not change frequently) after this PR goes into internal maser branch. The default value of include_kubernetes_master will be alway "n". So, any official branch released sonic image will not include k8s master packages. After we get a sonic-vs image with k8s master package from internal master branch, we will create VM by this image on PolitFish host, it means the k8s master image should be only run on PolitFish host, the VM's disk size is under our control, so don't need to care about the package size.

lguohan · 2023-06-30T16:06:47Z

files/build_templates/sonic_debian_extension.j2

@@ -779,11 +779,20 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIV
 sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/kube-proxy:${MASTER_KUBERNETES_CONTAINER_IMAGE_VERSION}
 sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/coredns/coredns:${MASTER_COREDNS_VERSION}
 sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/etcd:${MASTER_ETCD_VERSION}
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/metrics-scraper:${MASTER_UI_METRIC_VERSION}
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/dashboard:${MASTER_UI_DASH_VERSION}
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:${MASTER_MDM_VERSION}


geneva is a separate feature, i thought we are going to add a container? there is risk to add such new things into existing branch

geneva is a separate feature, i thought we are going to add a container? there is risk to add such new things into existing branch

Our k8s master image need this docker image for now, we have used this Geneva tools to send metric and log in our k8s master cluster. We will not include this docker image in any released sonic images, so the risk should be not that serious. I heard @fengpan is going to add Geneva container, I think after sonic support Geneva, I can remove it from k8s master feature and use sonic original Geneva feature.

Add k8s master related code

9a13f78

lixiaoyuner requested review from qiluo-msft, xumia and lguohan as code owners April 27, 2023 09:20

lixiaoyuner added 8 commits April 27, 2023 17:21

Update

a2fed98

Get MDM package from buster source repo

f2ca3c7

Remove mdm and mdsd from this change

1ff1845

Build includes k8s master when k8s file changes

14438a3

Update

537653a

Update

a748143

Use current skip pipeline for test

aa47375

Change the branch back in skip step

4b01f2d

liushilongbuaa previously approved these changes May 19, 2023

View reviewed changes

Add MDM and MDS container image for k8s master

17168db

lixiaoyuner dismissed liushilongbuaa’s stale review via 17168db May 29, 2023 05:00

Change the branch back in skip step

c026527

liushilongbuaa reviewed May 30, 2023

View reviewed changes

.azure-pipelines/template-skipvstest.yml Show resolved Hide resolved

liushilongbuaa approved these changes May 30, 2023

View reviewed changes

losha228 reviewed Jun 9, 2023

View reviewed changes

.azure-pipelines/template-skipvstest.yml Show resolved Hide resolved

losha228 approved these changes Jun 9, 2023

View reviewed changes

lguohan reviewed Jun 30, 2023

View reviewed changes

Merge branch 'master' into add-k8s-master-code

f305a59

lixiaoyuner closed this Jul 5, 2023

This was referenced Jul 5, 2023

Add k8s master code #15713

Closed

Add k8s master code new #15716

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add k8s master related code #14870

Add k8s master related code #14870

lixiaoyuner commented Apr 27, 2023 •

edited

Loading

liushilongbuaa left a comment

lguohan commented Jun 30, 2023

lguohan Jun 30, 2023

lixiaoyuner Jul 3, 2023 •

edited

Loading

lguohan Jun 30, 2023

lixiaoyuner Jul 3, 2023 •

edited

Loading

Add k8s master related code #14870

Add k8s master related code #14870

Conversation

lixiaoyuner commented Apr 27, 2023 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

liushilongbuaa left a comment

Choose a reason for hiding this comment

lguohan commented Jun 30, 2023

lguohan Jun 30, 2023

Choose a reason for hiding this comment

lixiaoyuner Jul 3, 2023 • edited Loading

Choose a reason for hiding this comment

lguohan Jun 30, 2023

Choose a reason for hiding this comment

lixiaoyuner Jul 3, 2023 • edited Loading

Choose a reason for hiding this comment

lixiaoyuner commented Apr 27, 2023 •

edited

Loading

lixiaoyuner Jul 3, 2023 •

edited

Loading

lixiaoyuner Jul 3, 2023 •

edited

Loading