Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add k8s master related code #14870

Closed
wants to merge 12 commits into from
+56 −12
Closed

Add k8s master related code #14870

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9a13f78
Add k8s master related code
lixiaoyuner Apr 27, 2023
a2fed98
Update
lixiaoyuner Apr 27, 2023
f2ca3c7
Get MDM package from buster source repo
lixiaoyuner Apr 28, 2023
1ff1845
Remove mdm and mdsd from this change
lixiaoyuner May 17, 2023
14438a3
Build includes k8s master when k8s file changes
lixiaoyuner May 17, 2023
537653a
Update
lixiaoyuner May 17, 2023
a748143
Update
lixiaoyuner May 17, 2023
aa47375
Use current skip pipeline for test
lixiaoyuner May 18, 2023
4b01f2d
Change the branch back in skip step
lixiaoyuner May 19, 2023
17168db
Add MDM and MDS container image for k8s master
lixiaoyuner May 29, 2023
c026527
Change the branch back in skip step
lixiaoyuner May 29, 2023
f305a59
Merge branch 'master' into add-k8s-master-code
lixiaoyuner Jul 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .azure-pipelines/azure-pipelines-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ jobs:
make $BUILD_OPTIONS ENABLE_ASAN=y target/docker-sonic-vs.gz
mv target/docker-sonic-vs.gz target/docker-sonic-vs-asan.gz
fi
make $BUILD_OPTIONS target/docker-sonic-vs.gz target/sonic-vs.img.gz target/docker-ptf.gz
make $BUILD_OPTIONS $(K8S_OPTIONS) target/docker-sonic-vs.gz target/sonic-vs.img.gz target/docker-ptf.gz
make $BUILD_OPTIONS target/docker-ptf-sai.gz
if [ $(Build.Reason) != 'PullRequest' ];then
gzip -kd target/sonic-vs.img.gz
Expand Down
7 changes: 7 additions & 0 deletions .azure-pipelines/template-skipvstest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@ steps:
- script: |
set -ex
tar_branch=origin/$(System.PullRequest.TargetBranch)
# Check if k8s master entrance script is changed
k8s_master_changed=$(git diff $tar_branch..HEAD --name-only | grep files/image_config/kubernetes/kubernetes_master_entrance.sh)
lixiaoyuner marked this conversation as resolved.
Show resolved Hide resolved
if [ -z "$k8s_master_changed" ]; then
lixiaoyuner marked this conversation as resolved.
Show resolved Hide resolved
echo "##vso[task.setvariable variable=K8S_OPTIONS;]INCLUDE_KUBERNETES_MASTER=n"
else
echo "##vso[task.setvariable variable=K8S_OPTIONS;]INCLUDE_KUBERNETES_MASTER=y"
fi
git diff $tar_branch..HEAD --name-only | grep -v -f .azure-pipelines/vstest-exclude && exit 0
git diff $tar_branch..HEAD --name-only | grep -f .azure-pipelines/vstest-include && exit 0
set +x
Expand Down
9 changes: 0 additions & 9 deletions build_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,17 +295,8 @@ then
echo '[INFO] Install kubernetes master'
install_kubernetes ${MASTER_KUBERNETES_VERSION}

sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -fsSL \
https://packages.microsoft.com/keys/microsoft.asc | \
sudo LANG=C chroot $FILESYSTEM_ROOT apt-key add -
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -fsSL \
https://packages.microsoft.com/keys/msopentech.asc | \
sudo LANG=C chroot $FILESYSTEM_ROOT apt-key add -
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azurecore-debian $IMAGE_DISTRO main" | \
sudo tee $FILESYSTEM_ROOT/etc/apt/sources.list.d/azure.list
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get update
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install hyperv-daemons gnupg xmlstarlet
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install metricsext2
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y remove gnupg
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -o /tmp/cri-dockerd.deb -fsSL \
https://github.com/Mirantis/cri-dockerd/releases/download/v${MASTER_CRI_DOCKERD}/cri-dockerd_${MASTER_CRI_DOCKERD}.3-0.debian-${IMAGE_DISTRO}_amd64.deb
Expand Down
11 changes: 10 additions & 1 deletion files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -791,11 +791,20 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIV
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/kube-proxy:${MASTER_KUBERNETES_CONTAINER_IMAGE_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/coredns/coredns:${MASTER_COREDNS_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/etcd:${MASTER_ETCD_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/metrics-scraper:${MASTER_UI_METRIC_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/dashboard:${MASTER_UI_DASH_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:${MASTER_MDM_VERSION}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

geneva is a separate feature, i thought we are going to add a container? there is risk to add such new things into existing branch

Copy link
Contributor Author

@lixiaoyuner lixiaoyuner Jul 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

geneva is a separate feature, i thought we are going to add a container? there is risk to add such new things into existing branch

Our k8s master image need this docker image for now, we have used this Geneva tools to send metric and log in our k8s master cluster. We will not include this docker image in any released sonic images, so the risk should be not that serious. I heard @fengpan is going to add Geneva container, I think after sonic support Geneva, I can remove it from k8s master feature and use sonic original Geneva feature.

sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:${MASTER_MDM_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:latest
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:latest
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:latest
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lots of packages installed, need to how how much disk space consumed. need to measure.

Copy link
Contributor Author

@lixiaoyuner lixiaoyuner Jul 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lots of packages installed, need to how how much disk space consumed. need to measure.

k8s master image will run on PilotFish host as a VM, no need to care about the disk space. We will not include this k8s master feature in any released sonic image.

More details:
For this k8s master feature, we have a include_kubernetes_master flag in rules/config to decide whether install all these k8s master packages or not. We plan to build the k8s master image from internal master branch on demand (maybe run official pipeline for internal master branch by passing include_kubernetes_master=y manually, because the master image will not change frequently) after this PR goes into internal maser branch. The default value of include_kubernetes_master will be alway "n". So, any official branch released sonic image will not include k8s master packages. After we get a sonic-vs image with k8s master package from internal master branch, we will create VM by this image on PolitFish host, it means the k8s master image should be only run on PolitFish host, the VM's disk size is under our control, so don't need to care about the package size.

echo "kubernetes master docker images pull complete"
# Install python package for mdm service usage
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install psutil
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install statsd
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable mdm.service
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install azure-storage-blob azure-identity
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install kubernetes
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable kubelet.service
# Add kubernetes master entrance
sudo cp files/image_config/kubernetes/kubernetes_master_entrance.sh $FILESYSTEM_ROOT/usr/sbin/
Expand Down
21 changes: 21 additions & 0 deletions files/image_config/kubernetes/kubernetes_master_entrance.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,23 @@
#!/bin/bash

# This script is for kubernetes master image usage
# Will mount kubernetes master disk and execute kubernetes entrance script

set -x

# mount disk from host
mount_point="/from_host"
disk="/dev/sdb1"
mkdir -p $mount_point
mount $disk $mount_point
# check whether it is the first time to boot
first_boot_flag_file="/from_host/first_boot_flag"
if [ -f $first_boot_flag_file ]; then
exit 0
fi
touch $first_boot_flag_file
# execute entrance script
init_file_name="entrance.sh"
init_file=${mount_point}/${init_file_name}
chmod +x $init_file
source $init_file
13 changes: 12 additions & 1 deletion rules/config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,12 +199,23 @@ INCLUDE_KUBERNETES_MASTER ?= n
# MASTER_PAUSE_VERSION - version of pause container image
# MASTER_COREDNS_VERSION - version of coredns container image
# MASTER_ETCD_VERSION = version of etcd container image
# MASTER_CRI_DOCKERD = version of cri-dockerd container image
# MASTER_UI_METRIC_VERSION = version of k8s metrics server container image
# MASTER_UI_DASH_VERSION = version of k8s dashboard container image
# MASTER_MDM_VERSION = version of mdm container image
# MASTER_MDS_VERSION = version of mds container image
# MASTER_FLUENTD_VERSION = version of fluentd container image
MASTER_KUBERNETES_VERSION = 1.22.2-00
MASTER_KUBERNETES_CONTAINER_IMAGE_VERSION = v1.22.2
MASTER_PAUSE_VERSION = 3.5
MASTER_COREDNS_VERSION = v1.8.4
MASTER_ETCD_VERSION = 3.5.0-0
MASTER_CRI_DOCKERD = 0.2.5
MASTER_CRI_DOCKERD = 0.3.1
MASTER_UI_METRIC_VERSION = v1.0.8
MASTER_UI_DASH_VERSION = v2.7.0
MASTER_MDM_VERSION = 2.2023.505.1124-45da18-20230505t1700
MASTER_MDS_VERSION = mariner_20230517.1
MASTER_FLUENTD_VERSION = mariner_20230517.1

# SONIC_ENABLE_IMAGE_SIGNATURE - enable image signature
# To not use the auto-generated self-signed certificate, the required files to sign the image as below:
Expand Down
5 changes: 5 additions & 0 deletions slave.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1448,6 +1448,11 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
MASTER_COREDNS_VERSION=$(MASTER_COREDNS_VERSION) \
MASTER_ETCD_VERSION=$(MASTER_ETCD_VERSION) \
MASTER_CRI_DOCKERD=$(MASTER_CRI_DOCKERD) \
MASTER_UI_METRIC_VERSION=$(MASTER_UI_METRIC_VERSION) \
MASTER_UI_DASH_VERSION=$(MASTER_UI_DASH_VERSION) \
MASTER_MDM_VERSION=$(MASTER_MDM_VERSION) \
MASTER_MDS_VERSION=$(MASTER_MDS_VERSION) \
MASTER_FLUENTD_VERSION=$(MASTER_FLUENTD_VERSION) \
./build_debian.sh $(LOG)

USERNAME="$(USERNAME)" \
Expand Down