This repository has been archived by the owner on Jan 11, 2023. It is now read-only.
Protect etcd tls from race conditions #2160
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -165,24 +165,24 @@ coreos: | |
| ExecStart=/opt/azure/containers/provision-setup.sh | ||
| {{else}} | ||
| runcmd: | ||
| - retrycmd_if_failure() { for i in $(seq 1 36); do $@; [ $? -eq 0 ] && break || sleep 5; done ; } | ||
| - echo `date`,`hostname`, startruncmd>>/opt/m | ||
| - apt-mark hold walinuxagent{{GetKubernetesAgentPreprovisionYaml .}} | ||
| - echo `date`,`hostname`, preaptupdate>>/opt/m | ||
| - retrycmd_if_failure apt-get update | ||
| - echo `date`,`hostname`, postaptupdate>>/opt/m | ||
| - retrycmd_if_failure apt-get install -y apt-transport-https ca-certificates nfs-common | ||
| - echo `date`,`hostname`, aptinstall>>/opt/m | ||
| - systemctl enable rpcbind | ||
| - systemctl enable rpc-statd | ||
| - systemctl start rpcbind | ||
| - systemctl start rpc-statd | ||
| - echo `date`,`hostname`, predockerinstall>>/opt/m | ||
| - retrycmd_if_failure curl --retry 5 --retry-delay 10 --retry-max-time 30 --max-time 60 -fsSL https://aptdocker.azureedge.net/gpg | apt-key add - | ||
|
There was a problem hiding this comment. There are two retry loops, do we need both? Also applies to similar code below. There was a problem hiding this comment. Again, taken from #2182 but I assume the two retries are complementary. @jackfrancis can you confirm / correct me? There was a problem hiding this comment. There is only one loop, but we're additionally using curl's retry option. I think this is O.K. for now. |
||
| - echo "deb {{WrapAsVariable "dockerEngineDownloadRepo"}} ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list | ||
| - "echo \"Package: docker-engine\nPin: version {{WrapAsVariable "dockerEngineVersion"}}\nPin-Priority: 550\n\" > /etc/apt/preferences.d/docker.pref" | ||
| - retrycmd_if_failure apt-get update | ||
| - retrycmd_if_failure apt-get install -y ebtables docker-engine | ||
|
There was a problem hiding this comment. What will happen if ebtables is installed but docker-engine failed. Just wonder if we should put them to two separate retries... There was a problem hiding this comment. Good question, this was taken directly from #2182, I'm merging the two since we had overlap. From what I understand (just tested this), apt-get ignores already installed packages so if one gets installed but not the other the retry will only try to install the one missing. |
||
| - echo "ExecStartPost=/sbin/iptables -P FORWARD ACCEPT" >> /etc/systemd/system/docker.service.d/exec_start.conf | ||
| - systemctl daemon-reload | ||
| - echo `date`,`hostname`, postdockerinstall>>/opt/m | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,7 +69,9 @@ fi | |
|
|
||
| if [[ ! -z "${MASTER_NODE}" ]]; then | ||
| echo "executing master node provision operations" | ||
|
|
||
| useradd -U "etcd" > /dev/null 2>&1 | ||
|
|
||
| APISERVER_PRIVATE_KEY_PATH="/etc/kubernetes/certs/apiserver.key" | ||
| touch "${APISERVER_PRIVATE_KEY_PATH}" | ||
| chmod 0600 "${APISERVER_PRIVATE_KEY_PATH}" | ||
|
|
@@ -85,7 +87,7 @@ if [[ ! -z "${MASTER_NODE}" ]]; then | |
| ETCD_SERVER_PRIVATE_KEY_PATH="/etc/kubernetes/certs/etcdserver.key" | ||
| touch "${ETCD_SERVER_PRIVATE_KEY_PATH}" | ||
| chmod 0600 "${ETCD_SERVER_PRIVATE_KEY_PATH}" | ||
| chown etcd:etcd "${ETCD_SERVER_PRIVATE_KEY_PATH}" | ||
| echo "${ETCD_SERVER_PRIVATE_KEY}" | base64 --decode > "${ETCD_SERVER_PRIVATE_KEY_PATH}" | ||
|
|
||
| ETCD_CLIENT_PRIVATE_KEY_PATH="/etc/kubernetes/certs/etcdclient.key" | ||
|
|
@@ -97,7 +99,7 @@ if [[ ! -z "${MASTER_NODE}" ]]; then | |
| ETCD_PEER_PRIVATE_KEY_PATH="/etc/kubernetes/certs/etcdpeer${MASTER_INDEX}.key" | ||
| touch "${ETCD_PEER_PRIVATE_KEY_PATH}" | ||
| chmod 0600 "${ETCD_PEER_PRIVATE_KEY_PATH}" | ||
| chown etcd:etcd "${ETCD_PEER_PRIVATE_KEY_PATH}" | ||
| echo "${ETCD_PEER_KEY}" | base64 --decode > "${ETCD_PEER_PRIVATE_KEY_PATH}" | ||
|
|
||
| ETCD_SERVER_CERTIFICATE_PATH="/etc/kubernetes/certs/etcdserver.crt" | ||
|
|
@@ -119,7 +121,6 @@ if [[ ! -z "${MASTER_NODE}" ]]; then | |
| echo "${ETCD_PEER_CERT}" | base64 --decode > "${ETCD_PEER_CERTIFICATE_PATH}" | ||
|
|
||
| echo `date`,`hostname`, finishedGettingEtcdCerts>>/opt/m | ||
| else | ||
| echo "skipping master node provision operations, this is an agent node" | ||
| fi | ||
|
|
@@ -272,7 +273,7 @@ function configNetworkPolicy() { | |
| function installClearContainersRuntime() { | ||
| # Add Clear Containers repository key | ||
| echo "Adding Clear Containers repository key..." | ||
| curl -sSL --retry 5 --retry-delay 10 --retry-max-time 30 "https://download.opensuse.org/repositories/home:clearcontainers:clear-containers-3/xUbuntu_16.04/Release.key" | apt-key add - | ||
|
There was a problem hiding this comment. not related to this PR, but are we ok to pull data from open internet? I kind of get the impression that we put a copy in Azure and download from there. There was a problem hiding this comment. @jessfraz might be able to answer, but this is out of scope for this PR since I'm only adding retries to all the curl commands. |
||
|
|
||
| # Add Clear Container repository | ||
| echo "Adding Clear Containers repository..." | ||
|
|
@@ -321,13 +322,13 @@ function installGo() { | |
| fi | ||
|
|
||
| # Get the latest Go version | ||
| GO_VERSION=$(curl --retry 5 --retry-delay 10 --retry-max-time 30 -sSL "https://golang.org/VERSION?m=text") | ||
|
|
||
| echo "Installing Go version $GO_VERSION..." | ||
|
|
||
| # subshell | ||
| ( | ||
| curl --retry 5 --retry-delay 10 --retry-max-time 30 -sSL "https://storage.googleapis.com/golang/${GO_VERSION}.linux-amd64.tar.gz" | sudo tar -v -C /usr/local -xz | ||
| ) | ||
|
|
||
| # Set GOPATH and update PATH | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just curious, why we pick 36?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
5 * 36 = 180s = 3 minutes, this is a generous timeout to ensure custom script had time to run. I'll have a follow up PR to parametrize this and clean up the runcmd in general.