Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add k8s master code new #15716

Merged
merged 22 commits into from
Jul 24, 2023
Merged

Add k8s master code new #15716

Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6d8088a
Add k8s master related code
lixiaoyuner Apr 27, 2023
5b37221
Update
lixiaoyuner Apr 27, 2023
50a2ea6
Get MDM package from buster source repo
lixiaoyuner Apr 28, 2023
46960cf
Remove mdm and mdsd from this change
lixiaoyuner May 17, 2023
2cffdc2
Build includes k8s master when k8s file changes
lixiaoyuner May 17, 2023
1a9093f
Update
lixiaoyuner May 17, 2023
36e440f
Update
lixiaoyuner May 17, 2023
9172106
Use current skip pipeline for test
lixiaoyuner May 18, 2023
a63b74a
Change the branch back in skip step
lixiaoyuner May 19, 2023
c2fc4f2
Add MDM and MDS container image for k8s master
lixiaoyuner May 29, 2023
b64c310
Change the branch back in skip step
lixiaoyuner May 29, 2023
f0663da
Build k8s master image for PR validation
lixiaoyuner Jul 5, 2023
be70f19
Fix quotation lack issue
lixiaoyuner Jul 5, 2023
99e1a60
change the target version back
lixiaoyuner Jul 5, 2023
832a0f4
Refine comments and fix grep regex issue
lixiaoyuner Jul 10, 2023
21d2882
Merge branch 'master' into add-k8s-master-code-new
lixiaoyuner Jul 11, 2023
10d6c94
Add one specific service for master entrance
lixiaoyuner Jul 13, 2023
c99cb09
Merge branch 'master' into add-k8s-master-code-new
lixiaoyuner Jul 13, 2023
10754fa
Make the script executable
lixiaoyuner Jul 13, 2023
81ea690
Move the master entrance script to /usr/bin
lixiaoyuner Jul 14, 2023
9fb8945
Mount disk by disk path file
lixiaoyuner Jul 18, 2023
3c305e9
Update comments
lixiaoyuner Jul 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .azure-pipelines/azure-pipelines-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ jobs:
make $BUILD_OPTIONS ENABLE_ASAN=y target/docker-sonic-vs.gz
mv target/docker-sonic-vs.gz target/docker-sonic-vs-asan.gz
fi
if [ $(K8S_OPTIONS) == 'INCLUDE_KUBERNETES_MASTER=y' ]; then
make $BUILD_OPTIONS $(K8S_OPTIONS) target/sonic-vs.img.gz
mv target/sonic-vs.img.gz target/sonic-vs-k8s.img.gz
fi
make $BUILD_OPTIONS target/docker-sonic-vs.gz target/sonic-vs.img.gz target/docker-ptf.gz
make $BUILD_OPTIONS target/docker-ptf-sai.gz
if [ $(Build.Reason) != 'PullRequest' ];then
Expand Down
7 changes: 7 additions & 0 deletions .azure-pipelines/template-skipvstest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@ steps:
- script: |
set -ex
tar_branch=origin/$(System.PullRequest.TargetBranch)
# Check if k8s master entrance script is changed
k8s_master_changed=$(git diff $tar_branch..HEAD --name-only | grep -F files/image_config/kubernetes/kubernetes_master_entrance.sh)
if [ -z "$k8s_master_changed" ]; then
echo "##vso[task.setvariable variable=K8S_OPTIONS;]INCLUDE_KUBERNETES_MASTER=n"
else
echo "##vso[task.setvariable variable=K8S_OPTIONS;]INCLUDE_KUBERNETES_MASTER=y"
fi
git diff $tar_branch..HEAD --name-only | grep -v -f .azure-pipelines/vstest-exclude && exit 0
git diff $tar_branch..HEAD --name-only | grep -f .azure-pipelines/vstest-include && exit 0
set +x
Expand Down
9 changes: 0 additions & 9 deletions build_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,17 +295,8 @@ then
echo '[INFO] Install kubernetes master'
install_kubernetes ${MASTER_KUBERNETES_VERSION}

sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -fsSL \
https://packages.microsoft.com/keys/microsoft.asc | \
sudo LANG=C chroot $FILESYSTEM_ROOT apt-key add -
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -fsSL \
https://packages.microsoft.com/keys/msopentech.asc | \
sudo LANG=C chroot $FILESYSTEM_ROOT apt-key add -
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azurecore-debian $IMAGE_DISTRO main" | \
sudo tee $FILESYSTEM_ROOT/etc/apt/sources.list.d/azure.list
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get update
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install hyperv-daemons gnupg xmlstarlet
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install metricsext2
sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y remove gnupg
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT curl -o /tmp/cri-dockerd.deb -fsSL \
https://github.com/Mirantis/cri-dockerd/releases/download/v${MASTER_CRI_DOCKERD}/cri-dockerd_${MASTER_CRI_DOCKERD}.3-0.debian-${IMAGE_DISTRO}_amd64.deb
Expand Down
20 changes: 16 additions & 4 deletions files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -790,15 +790,27 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIV
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/kube-proxy:${MASTER_KUBERNETES_CONTAINER_IMAGE_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/coredns/coredns:${MASTER_COREDNS_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull k8s.gcr.io/etcd:${MASTER_ETCD_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/metrics-scraper:${MASTER_UI_METRIC_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull kubernetesui/dashboard:${MASTER_UI_DASH_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:${MASTER_MDM_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:${MASTER_MDM_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevamdm:latest
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:${MASTER_MDS_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevamdsd:latest
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS pull linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION}
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT docker $SONIC_NATIVE_DOCKERD_FOR_DOCKERFS tag linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:${MASTER_FLUENTD_VERSION} linuxgeneva-microsoft.azurecr.io/distroless/genevafluentd_td-agent:latest
echo "kubernetes master docker images pull complete"
# Install python package for mdm service usage
# Install python packages for mdm metrics collection service usage
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install psutil
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install statsd
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable mdm.service
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install kubernetes
# Install python packages to upload and download etcd backup files for backup and restore service usage
sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install azure-storage-blob azure-identity
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable kubelet.service
# Add kubernetes master entrance
sudo cp files/image_config/kubernetes/kubernetes_master_entrance.sh $FILESYSTEM_ROOT/usr/sbin/
sudo sed -i '/^exit 0/i\bash /usr/sbin/kubernetes_master_entrance.sh' $FILESYSTEM_ROOT/etc/rc.local
sudo cp files/image_config/kubernetes/kubernetes_master_entrance.service ${FILESYSTEM_ROOT_USR_LIB_SYSTEMD_SYSTEM}/
sudo cp files/image_config/kubernetes/kubernetes_master_entrance.sh $FILESYSTEM_ROOT/usr/bin/
sudo chmod +x $FILESYSTEM_ROOT/usr/bin/kubernetes_master_entrance.sh
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable kubernetes_master_entrance.service
{% endif %}

{% macro get_install_options(set_owner, enabled) -%}
Expand Down
13 changes: 13 additions & 0 deletions files/image_config/kubernetes/kubernetes_master_entrance.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[Unit]
Description=Setup k8s master services
Requires=rc-local.service
After=rc-local.service
Before=sonic.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/kubernetes_master_entrance.sh

[Install]
WantedBy=multi-user.target
45 changes: 45 additions & 0 deletions files/image_config/kubernetes/kubernetes_master_entrance.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,47 @@
#!/bin/bash

# This script is for kubernetes master image usage
# Will mount kubernetes master disk and execute kubernetes entrance script

set -x

# mount disk from host
mount_point="/from_host"
disk_by_path_dir="/dev/disk/by-path"
# We can't ensure the mounted disk name is sda or sdb
# But we can specify the disk logic unit number to 2
# We find the correct disk by the disk logic unit number from by-path directory
target_str="lun-2-part1"
disk_path_file=$(ls $disk_by_path_dir | grep $target_str)

# Check whether the disk path file exists
if [ -z "$disk_path_file" ]; then
echo "Error: Disk path file not found."
exit 1
fi

# Check the number of lines returned
line_count=$(echo "$disk_path_file" | wc -l)

# If there are multiple lines, exit with status 1
if [ "$line_count" -ne 1 ]; then
echo "Error: multiple disk path files found."
exit 1
fi

disk="${disk_by_path_dir}/${disk_path_file}"
mkdir -p $mount_point
mount $disk $mount_point

# check whether it is the first time to boot
first_boot_flag_file="/from_host/first_boot_flag"
Copy link
Collaborator

@qiluo-msft qiluo-msft Jul 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first_boot_flag

Is this a new feature?
Wondering if you can use existing sonic feature:

FIRST_BOOT_FILE="/host/image-${SONIC_VERSION}/platform/firsttime"
``` #WontFix

Copy link
Contributor Author

@lixiaoyuner lixiaoyuner Jul 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to check whether it's the first boot here, if it's the first time, need to call the master script. If there is already one same feature, I can reuse the existing one. But after I checked, I don't think we can leverage the existing flag. Currently, we do "sudo sed -i '/^exit 0/i\bash /usr/sbin/kubernetes_master_entrance.sh' $FILESYSTEM_ROOT/etc/rc.local" when include k8s master feature. I see the "firsttime" flag aslo in rc.local file, but for grub staff. Not good to leverage, except we change rc.local file directly, I think there is no need to change rc.local file directly for k8s master feature.

Copy link
Contributor Author

@lixiaoyuner lixiaoyuner Jul 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean it's not good to use the existing feature. The reason is that we call the k8s master script in the last line of rc.local, but we don't change the rc.local file directly. We insert the call master script line by sed command when include_kubernetes_master=y. If we want to add the calling k8s master script to existing firsttime flag logic, we need to change the rc.local file directly, it's not easy to add lines to rc.local by sed command any more. If we agree to change the rc.local file directly, I can try to use the existing flag.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the point of first_boot_flag, kubernetes_master_entrance.sh and sed -i '/^exit 0/i\bash /usr/sbin/kubernetes_master_entrance.sh' $FILESYSTEM_ROOT/etc/rc.local.

They are introduced for a deployment workflow, but this workflow is fragile. Suggest:

  1. Create and start the Kubernetes master VM
  2. Configure the VM IP by hyperv-daemons, or by console access
  3. SSH into the VM, so you have full control
  4. Copy some scripts into the VM
  5. Enable/start some services.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created #15849, and please consider it in future PR.

if [ -f $first_boot_flag_file ]; then
exit 0
fi
touch $first_boot_flag_file

# execute entrance script
init_file_name="entrance.sh"
init_file=${mount_point}/${init_file_name}
chmod +x $init_file
source $init_file
13 changes: 12 additions & 1 deletion rules/config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,12 +199,23 @@ INCLUDE_KUBERNETES_MASTER ?= n
# MASTER_PAUSE_VERSION - version of pause container image
# MASTER_COREDNS_VERSION - version of coredns container image
# MASTER_ETCD_VERSION = version of etcd container image
# MASTER_CRI_DOCKERD = version of cri-dockerd container image
# MASTER_UI_METRIC_VERSION = version of k8s metrics server container image
# MASTER_UI_DASH_VERSION = version of k8s dashboard container image
# MASTER_MDM_VERSION = version of mdm container image
# MASTER_MDS_VERSION = version of mds container image
# MASTER_FLUENTD_VERSION = version of fluentd container image
MASTER_KUBERNETES_VERSION = 1.22.2-00
MASTER_KUBERNETES_CONTAINER_IMAGE_VERSION = v1.22.2
MASTER_PAUSE_VERSION = 3.5
MASTER_COREDNS_VERSION = v1.8.4
MASTER_ETCD_VERSION = 3.5.0-0
MASTER_CRI_DOCKERD = 0.2.5
MASTER_CRI_DOCKERD = 0.3.1
MASTER_UI_METRIC_VERSION = v1.0.8
MASTER_UI_DASH_VERSION = v2.7.0
MASTER_MDM_VERSION = 2.2023.505.1124-45da18-20230505t1700
MASTER_MDS_VERSION = mariner_20230517.1
MASTER_FLUENTD_VERSION = mariner_20230517.1
xumia marked this conversation as resolved.
Show resolved Hide resolved

# SONIC_ENABLE_IMAGE_SIGNATURE - enable image signature
# To not use the auto-generated self-signed certificate, the required files to sign the image as below:
Expand Down
5 changes: 5 additions & 0 deletions slave.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1448,6 +1448,11 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \
MASTER_COREDNS_VERSION=$(MASTER_COREDNS_VERSION) \
MASTER_ETCD_VERSION=$(MASTER_ETCD_VERSION) \
MASTER_CRI_DOCKERD=$(MASTER_CRI_DOCKERD) \
MASTER_UI_METRIC_VERSION=$(MASTER_UI_METRIC_VERSION) \
MASTER_UI_DASH_VERSION=$(MASTER_UI_DASH_VERSION) \
MASTER_MDM_VERSION=$(MASTER_MDM_VERSION) \
MASTER_MDS_VERSION=$(MASTER_MDS_VERSION) \
MASTER_FLUENTD_VERSION=$(MASTER_FLUENTD_VERSION) \
./build_debian.sh $(LOG)

USERNAME="$(USERNAME)" \
Expand Down
Loading