Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node Annotations are Only Evaluated on Startup and not Actively Watched During Runtime #676

Open
Feder1co5oave opened this issue Feb 21, 2019 · 13 comments · May be fixed by #1723
Open

Node Annotations are Only Evaluated on Startup and not Actively Watched During Runtime #676

Feder1co5oave opened this issue Feb 21, 2019 · 13 comments · May be fixed by #1723
Assignees
@aauren
Labels
enhancement override-stale Don't allow automatic management of stale issues / PRs
Milestone
v2.2.0

Comments

@Feder1co5oave
Copy link
Contributor

On an already existing kubernetes cluster with kube-router successfully running in route reflector mode (rr.server annotation on 3 master nodes, rr.client annotation on all the workers), my workflow for joining new worker nodes is like this (the cluster was created with kubeadm):

for each new node:

  • run kubeadm join on the node
  • kubectl annotate node kube-router.io/rr.client=23
  • restart kube-router on the new node to put them in RR mode

Turns out the nodes previously joined to the cluster will receive routes for all the new nodes, but the new nodes will only receive routes for the "old" nodes. Restarting kube-router on the route reflector nodes solved this issue.

My troubleshooting suggested that whenever a new node joins, the rr server kube-router daemons will peer with that node right away, even before the rr.client annotation gets added to it. Thus, the new peer is assumed to be in full mesh mode even if in fact it will be in RR mode and will only peer with RR servers. BGP route reflector allows for BGP daemons in the same AS to peer in either full-mesh or RR mode. Export policies are such that RR servers will reflect advertisements:

  • from RR peers to RR peers
  • from full-mesh peers to RR peers
  • from RR peers to full-mesh peers
  • but NOT from full-mesh peers to other full-mesh peers (they're supposed to peer with each other in full-mesh, duh)

So the new nodes, assumed to be forming full-mesh, don't get advertisements about other new nodes.
Restarting RR servers forces them to reload the node list and annotations, and correctly peer with the new nodes in RR mode.

To fix this, kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!
Feder1co5oave added a commit to Feder1co5oave/kube-router that referenced this issue Feb 21, 2019
@Feder1co5oave
document workaround for cloudnativelabs#676
cfd51fd
@Feder1co5oave Feder1co5oave mentioned this issue Feb 21, 2019
Merged
murali-reddy pushed a commit that referenced this issue Mar 10, 2019
@Feder1co5oave @murali-reddy
document workaround for #676 (#677)
7b20ae9
@aauren
Copy link
Collaborator

aauren commented Apr 25, 2020

Closed via #677

@aauren aauren closed this as completed Apr 25, 2020
@ofen
Copy link

ofen commented Apr 29, 2020

Is it possible to make it automatically (without manual POD restart)?

@murali-reddy
Copy link
Member

@ofen reopening as still below make sense to be fixed.

kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!

@murali-reddy murali-reddy reopened this Apr 29, 2020
@zhaixigui
Copy link

Why can't it automatically recognize the newly added node? And to propagate routing。Why have to do such a dangerous operation to kill RR?

@zhaixigui
Copy link

Do we have any choice but to kill the RR?

@Feder1co5oave
Copy link
Contributor Author

@zhaixigui the new nodes are recognized, but they are assumed to be in full mesh mode, and there's no clean way to do the switch to RR mode other than restart kube-router on both RR server and client to reload the nodes' information.

Killing RR servers should not bring any disruption as long as you enabled soft-restart. I've done it several times without any repercussions.

@zhaixigui
Copy link

zhaixigui commented Aug 19, 2020 •
edited
Loading

@Feder1co5oave a bad idea？ when the RR server watched a ”kube-router.io/rr.client=42“ annotation from
most recently joined node. it first calls method DeleteNeighbor to delete this node, and then calls method AddNeighbor to add this node as RR client, and RR will automatically propagate this route ？

@zhaixigui
Copy link

@zhaixigui the new nodes are recognized, but they are assumed to be in full mesh mode, and there's no clean way to do the switch to RR mode other than restart kube-router on both RR server and client to reload the nodes' information.

Killing RR servers should not bring any disruption as long as you enabled soft-restart. I've done it several times without any repercussions.

soft-restart is GracefulRestart ?

@zhaixigui
Copy link

Does calico have the same problem?

@zhaixigui zhaixigui mentioned this issue Aug 22, 2020
Closed
@Feder1co5oave
Copy link
Contributor Author

Feder1co5oave commented Aug 22, 2020 •
edited
Loading

I don't know. Calico has a great reputation as network plugin and also great features. When I considered it for my clusters I found it is more complicated and has more moving parts than kube-router, so I ended up choosing the latter because of its "simplicity".

@Feder1co5oave Feder1co5oave reopened this Aug 22, 2020
@zhaixigui
Copy link

zhaixigui commented Aug 25, 2020 •
edited
Loading

@Feder1co5oave a bad idea？ when the RR server watched a ”kube-router.io/rr.client=42“ annotation from
most recently joined node. it first calls method DeleteNeighbor to delete this node, and then calls method AddNeighbor to add this node as RR client, and RR will automatically propagate this route ？

@Feder1co5oave Feder， Is this a bad idea? is there a better idea or solution?

@Feder1co5oave
Copy link
Contributor Author

@zhaixigui not at all, in fact it is exactly the solution I proposed in my first post. But I'm not a gopher and this would involve a bit of modifications to the control flow, i.e. watching nodes. I believe we'll have to wait for a volunteer to write a PR.

@aauren aauren changed the title Newly joined nodes don't receive all routes when using RR mode Node Annotations are Only Evaluated on Startup and not Actively Watched During Runtime Aug 13, 2021
@aauren aauren mentioned this issue Nov 15, 2022
Closed
@aauren aauren added the override-stale Don't allow automatic management of stale issues / PRs label Sep 4, 2023
@aauren aauren added this to the v2.1.0 milestone Jan 6, 2024
@aauren aauren self-assigned this Jan 6, 2024
@aauren aauren modified the milestones: v2.1.0, v2.2.0 Mar 2, 2024
@aauren aauren mentioned this issue May 15, 2024
Closed
@sebltm
Copy link

sebltm commented Aug 22, 2024

I think it's as easy as doing something like this: #1723
I've tested it in my local environment, seems to work as expected

@sebltm sebltm linked a pull request Aug 22, 2024 that will close this issue
Open
@aauren aauren mentioned this issue Sep 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aauren aauren

Labels
enhancement override-stale Don't allow automatic management of stale issues / PRs
Projects
None yet
Milestone
v2.2.0
Development

Successfully merging a pull request may close this issue.

fix: update peers on node update sebltm/kube-router
6 participants
@ofen @aauren @Feder1co5oave @murali-reddy @sebltm @zhaixigui