No internet connectivity after recent microos update

[andi0b]

A recent (after march 10th 2024) microos update put my ARM nodes into a state that they can't connect to the internet anymore. The nodes seem to work perfectly fine and are reachable via the VNET, but can't get any internet connectivity. Happened on two nodes so far. And a manual rollback fixes the issue.

I didn't have any time to investigate yet, but maybe someone else is facing a similar issue?

[DjVinnii]

I'm also facing this issue with one of my nodes. I noticed a difference between the secondary (internal) interface names. The working node has an interface called eth1 and the non-working node has an interface called enp7s0. Could this be something to do with it? The secondary interface is the preferred route according to ip route:

# ip route
default via 10.0.0.1 dev enp7s0 proto dhcp src 10.255.0.101 metric 20100
default via 172.31.1.1 dev eth0 proto dhcp src x.x.x.x metric 20101
10.0.0.0/8 via 10.0.0.1 dev enp7s0 proto dhcp src 10.255.0.101 metric 100
10.0.0.1 dev enp7s0 proto dhcp scope link src 10.255.0.101 metric 100
172.31.1.1 dev eth0 proto dhcp scope link src x.x.x.x metric 101

[DjVinnii]

I did a little bit more investigation. For some reason the working node also has some references to enp7s0, I suspect that this is provided by the virtualization software used by Hetzner.

Working:

# udevadm info -p  /sys/class/net/eth1
P: /devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio6/net/eth1
M: eth1
R: 1
U: net
I: 3
E: DEVPATH=/devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio6/net/eth1
E: SUBSYSTEM=net
E: INTERFACE=eth1
E: IFINDEX=3
E: USEC_INITIALIZED=35035363
E: ID_NET_NAMING_SCHEME=v253
E: ID_NET_NAME_MAC=enx860000741015
E: ID_NET_NAME_PATH=enp7s0
E: ID_BUS=pci
E: ID_VENDOR_ID=0x1af4
E: ID_MODEL_ID=0x1041
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Red Hat, Inc.
E: ID_MODEL_FROM_DATABASE=Virtio 1.0 network device
E: ID_MM_CANDIDATE=1
E: ID_PATH=pci-0000:07:00.0
E: ID_PATH_TAG=pci-0000_07_00_0
E: ID_NET_DRIVER=virtio_net
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enp7s0
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eth1
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

Non-working:

# udevadm info -p /sys/class/net/enp7s0
P: /devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio6/net/enp7s0
M: enp7s0
R: 0
U: net
I: 3
E: DEVPATH=/devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio6/net/enp7s0
E: SUBSYSTEM=net
E: INTERFACE=enp7s0
E: IFINDEX=3
E: USEC_INITIALIZED=4830913
E: ID_NET_NAMING_SCHEME=v253
E: ID_NET_NAME_MAC=enx860000741013
E: ID_NET_NAME_PATH=enp7s0
E: ID_BUS=pci
E: ID_VENDOR_ID=0x1af4
E: ID_MODEL_ID=0x1041
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Red Hat, Inc.
E: ID_MODEL_FROM_DATABASE=Virtio 1.0 network device
E: ID_MM_CANDIDATE=1
E: ID_PATH=pci-0000:07:00.0
E: ID_PATH_TAG=pci-0000_07_00_0
E: ID_NET_DRIVER=virtio_net
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enp7s0
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp7s0
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

Looking at the eth0 interfaces, these are also referring to enp1s0.

[DjVinnii]

@andi0b Maybe it it worth converting this discussion into an issue? This problem happened to me multiple times right now, which basically makes this project unusable in my opinion

[andi0b]

@DjVinnii I don't think I can convert this to an issue. I also don't have the time right now to investigate this problem, and I don't want to open an issue without providing the necessary details.

I started it as discussion to find out if other people are having similar problems, thanks for sharing your observations, I might look into it again next week.

But feel free to open an issue and reference this discussion.

Edit: there seem to be already at least two related issues to this problem: #1298 #1289

[cobrani]

I'm having the same issue. Unfortunately, I missed setting up backups and snapshots.. too bad :-) I managed to ssh into the control planes via a jump host, but am not sure how to resolve this situation. Ideas are much appreciated

[andi0b]

@mysticaltech I narrowed the issue down, and it looks like in my case the /etc directory is more or less lost during the upgrade. I noticed that the /etc/udev/rules.d/70-persistent-net.rules is gone if I boot into the update.

This seems to be a really weird problem and somehow related to the transactional-update and the etc overlayfs. I can't grasp how this happens. Any ideas?

[andi0b]

@valkenburg-prevue-ch In my case this problem is not related to k3s upgrades at all. Those just worked without me noticing.

As far as I understand it, they are done by kubernetes services, and not via the distributions transactional-upgrade. In my case it seems like the distributions update procedure is failing, and and k3s is doing fine.

[valkenburg-prevue-ch]

Thanks for letting me know. This makes a lot more sense than what I thought.

[mysticaltech]

@andi0b Super good findings, thanks for sharing. Yes please do let them know if you can! 🙏

[mysticaltech]

I have an idea folks, maybe we need to apply the changes from cloudinit, especially network configs on each reboots. We can do that via a systemctl service that runs after each reboots maybe.

@sysrich, Please if you can give your opinion on #1287 (reply in thread) (comment above) it would be super kind. We have been having issues after microos upgrades for some time, and that analysis may explain it.

[andi0b]

TL;DR: I think applying network changes doesn't solve this issue. It is somehow related to transactional-update cleaning up parent snapshots too early

@mysticaltech: In my case applying the network configuration won't be enough at all. My /etc after installing updates is more or less the stock /etc that came with micro os base install, without any configuration changes. So there is not even the k3s systemd service and no automatic k3s startup.

micro os snapshots

From my point of view the issue is: Too early cleanup of snapper snapshots. With automatic updates in microos turned on it usually creates one snapshot daily. Once there are more than 10 snapshots it starts to clean up older snapshots.

Every microos snapshot has a parent, which doesn't have to be the preceding snapshot, if you only reboot after let's say 50 days there might be 50 snapshots between the current one and the parent snapshot.

The parent snapshot seem to be important mostly for /etc, because the parent snapshots /etc is overlay mounted on top of the current snapshot.

snapshot cleanup theory

My current working theory is this:

Let's go back to my example with my snapshot numbers:

• 1 - Initial snapshot - stock /etc
• 3 - After installing k3s and setting it up - /etc with k3s and network config
• 42* - current running snapshot - /etc based on #1, so stock micro os config, but with #3 /etc as overlay
• 65 - updated snapshot that's not working /etc based on #42, so stock micro os config, but with overlay of #42

So in my case all the relevant /etc changes are in the overlay mount of the parent snapshot, but not included in the current snapshot. So I'm running on #42, which includes an /etc overlay based on #42 AND #3. So it includes k3s and networking.

Once a new snapshot is created (#65), transactional updates fstab to mount the overlay from #42 and #65. And tries to rsync the /etc from #3 (the parent). And here starts the issue.

Like I described previously #3 is already cleaned up and can't be mounted (error message Parent snapshot 3 does not exist any more - skipping rsync). The /etc overlay still exists, but the snapshot doesn't any more. But still transactional-update seems to ignore the /etc from #3 and creates #65 /etc from the stock micro os installation (without k3s and network config).

What I don't get is why transactional-update cleaned up the parent snapshot #3, which seems to be a crucial component for creating new snapshots.

Another theory: Something went wrong between#3 and #42. I don't have #3 any more, and I didn't try to go back in the logs that far.

is it micro os or kube-hetzner?

I actually looked into the code in kube-hetzner, if the micro os transactional-update cleanup configuration is somehow customized, but I couldn't find anything. Maybe this is a bug or even a design choice of micro os. I read that it's recommended to reboot micro os as early as possible after running transactional-update, which I didn't do in my case. But I think I'm not the only one that doesn't reboot the nodes daily or at least weekly.

I still doubt this is the intended behavior of micro os, maybe at some point the packer/terraform process somehow interferes with the micro os update procedure in a bad way, that creates this problem.

how to move forward

I'm still quite confused about this issue. I think this could be reproduced by setting up a new node, then running transactional-update 11 times manually and reboot into the newest snapshot that is at least 10 snapshots ahead of the current one. Then running transactional-update once again and reboot. Then it might trigger the issue. Somehow transactional-update need something to do between the reboots, just running it without any updates available is probably a noop.

It might also have been an intermittent issue in a previous version of transactional-update that was current between january and march '24.

If someone from the core team here is interested in looking at my nodes, just send me an email (in my profile) and we can set up a chat/call to look into it (mo-su 10-22h UTC). Maybe there is something you see that I don't.

[v-petukhov]

Hi to all!

I've stuck into similar problem. After upgrade k3s to 1.29.3 agent node became unavailable cause of connection problems. So, there is no ssh connection over external IP and connection timeout errors was thrown. But, I can access node via ssh through other node via local (internal) IP. So, I'm on node there are longer no eth1 interface:

k3s-agent-nbg1-cpx41-wwb:~ # ip route
default via 10.0.0.1 dev enp7s0 proto dhcp src 10.0.1.101 metric 20100
default via 172.31.1.1 dev eth0 proto dhcp src 49.13.160.56 metric 20101
10.0.0.0/16 via 10.0.0.1 dev enp7s0 proto dhcp src 10.0.1.101 metric 100
10.0.0.1 dev enp7s0 proto dhcp scope link src 10.0.1.101 metric 100
172.31.1.1 dev eth0 proto dhcp scope link src 49.13.160.56 metric 101

And I have no idea how to deal with it.

My thoughts are:

• Increase agent node pool
• Drain non-working node (cpx41-wwb)
• Decrease agent node pool

And I don't know, will it or not works?

@andi0b , @DjVinnii did you solve this problem?

[mysticaltech]

@v-petukhov Please see @valkenburg-prevue-ch comment above explaining how to fix this.

[mysticaltech]

@andi0b @v-petukhov FYI here #697 (comment), @maaft confirmed that re-applying the cloudinit commands manually works. For the time being, I will make sure that happens automatically so as to make sure that whatever hopefully temporary issues there is with snapshot syncing, it's not felt.

@jhass FYI the above thread, @andi0b pretty much cornered the underlying issue.

[andi0b]

I'm not sure about that, if it would change a lot.

In my case (I don't know relevant it is to other users) this would not change anything, because my /etc reverts to before k3s is even installed. So there is a lot more missing, and this change would probably also not be inside the snapshot.

One possibility that could be to create another snapshot after installing everything, so the /etc overlays get merged, I don't know though how this would be possible. Maybe by running transactional-update without the --continue flag.

Another thing that would help in the situation I had would be to allow reinstalling existing nodes without creating new nodes/nodepools one by one (to keep the IPs of the nodes, I'm not using a load balancer).

Maybe decreasing the default transactional-update frequency (daily to weekly) and increasing the number of kept snapshots (from 10 to 30 or 50) might just make the issue far less frequent.

PS: It would be great to update the OS much less frequently than daily, I don't know if there is some possibility to identify critical security updates and only then install updates right away, but skip updates for a few days/weeks if the changes are not critical. Security updates should only come once in a while on such a minimal installation.

[mysticaltech]

@andi0b etc is not part of snapshots, it and var are permanent. So maybe your case is unique.

It's definitely possible to change the update frequency and number of snapshots kept. PR welcome for such a functionality, this is important for space management. Normally for the latter we have it stable, but maybe not, maybe we are hitting a snapshot space issue on small disk nodes 🤔

[jhass]

I've definitely seen etc revert too, to the point of loosing the node key.

[mysticaltech]

Damn, ok, thanks for confirming.

[mysticaltech]

@andi0b @v-petukhov If you could try running those commands and let me know, it would be great (in case your nodes happen to still be up).

cloud-init single --frequency always --name write_files
cloud-init single --frequency always --name runcmd

[andi0b]

Yes, my damaged nodes are still up, I'm still a bit scared to re-create them. I can keep them running after draining them and try that.

What is this supposed to do? Update only the network configuration? In my case the damaged nodes don't even have the systemd service for k3s after updating, so they don't even start k3s on boot.

[andi0b]

@mysticaltech I think it's better to continue the discussion here (from #1324 (comment))

In the meantime, please let me know if that issue shows up again. I'm hoping it was just one bad apple update along the way.
And if you find a way to actively prevent it, please let me know. You seem close to deciphering the whole thing lol.

Sadly I think this is a conceptional issue.

I don't know if you took the time to read the long posts I wrote in this discussion before, but it all comes down to the /etc rsync of transactional-update (at least in my case).

It's this catch block in the source code: https://github.com/openSUSE/transactional-update/blob/2d77e2b24a3e958ebede49622c68f46e9adc377f/lib/Overlay.cpp#L106

In the case the base snapshot is not there anymore (and this happens after 10-20 days without a reboot) /etc is not getting merged/rsynced anymore. I don't really get why, because the base snapshot is not really required, the /etc overlays are kept even after the snapshot is removed.

I thought about creating an issue with transactional-update, but I don't really feel confident enough to do that, because I'm still missing a lot of details about it's inner workings.

[mysticaltech]

@andi0b I like your idea, let's try it. PR welcome.

[mysticaltech]

If I'm not mistaken, kured can do stuff pre-reboot and post-reboot too. We could reapply initial setup there too.

[mysticaltech]

@andi0b Not sure it it helps, here's what gpt-4-turbo thinks about this:

Hey Andy,

It sounds like you're trying to tackle the issue of maintaining /etc configurations in OpenSUSE MicroOS through the transactional update process, where changes can be lost if not properly committed before an update and reboot. Let’s break down your proposal and consider its feasibility and effectiveness.

1. Understanding the Overlay/Merge Process: In OpenSUSE MicroOS, the filesystem uses an overlay approach where changes during normal operations are written to a temporary overlay filesystem. Only during a transactional update (which typically involves a reboot) are these changes merged back into the main filesystem. This is designed to maintain system stability and integrity, but it can complicate the management of dynamic configurations like those in /etc.

2. Proposed Workaround - Triggering Merges with Additional Snapshots:

   • Your idea to force additional snapshots and reboots after configuring the node (especially network settings and k3s services) could ensure that the changes in /etc are indeed committed to the disk before an extended runtime could expose them to update conflicts.
   • Adding a dummy file as a trigger for this snapshot is a clever hack; it could manually initiate the transactional-update process, ensuring that the current state of /etc is preserved. This is akin to artificially creating a state change to prompt the system to consolidate the overlay into the base filesystem.

3. Caveat Consideration: The point you raised about the snapshot potentially capturing the state of /etc from the previous boot, rather than the most current operational state, is crucial. This could mean that any changes made between the last snapshot and the current time could be lost unless the snapshot mechanism is precisely timed or managed.

4. Implementation Strategy:

   • Snapshot Timing: To minimize the risk of losing recent changes, you could script the snapshot to occur at times of low configuration activity, or directly after confirmed configuration changes. This might involve integrating with configuration management tools or monitoring file system events.
   • Automation and Testing: Given the complexities and potential risks of this approach (especially in a production environment), it would be prudent to first implement this solution in a controlled testing environment. Monitoring tools can be used to validate that changes to /etc are indeed being captured and retained across reboots and updates.
   • Documentation and Support: Enhancing the documentation on your end or even contributing to the OpenSUSE documentation could help clarify the operational nuances of transactional updates and the handling of /etc for others facing similar issues.

5. Feedback and Iteration: It would be beneficial to gather feedback from other OpenSUSE MicroOS users who might have faced similar issues, potentially through forums or community support channels. This feedback could provide insights into whether additional snapshots are necessary or if there are simpler or more efficient solutions.

Overall, while the solution might sound a bit unconventional, it's rooted in a solid understanding of the system’s mechanics and demonstrates a thoughtful approach to ensuring system stability and configuration integrity. Implementing this would definitely require careful planning and testing to avoid unintended consequences, particularly in how the timing of snapshots interacts with system updates and reboots.

[andi0b]

If I'm not mistaken, kured can do stuff pre-reboot and post-reboot too. We could reapply initial setup there too.

Probably, but this would require to repair and re-configure a broken k3s installation. I'm not sure if this is a good idea, everything's set up correctly, just the part inside /etc. This state is probably unexpected by the k3s package/install scripts. It might cause even more complex issues.

Maybe an even easier solution would be to protect snapshot No.3 with snapper after finishing installation. So it can never be deleted. Marking it as "important" is probably not enough.

[mysticaltech]

@andi0b PR welcome man, you've cornered the issue more than any of us 🙏

[andi0b]

@mysticaltech

I finally drained all the broken nodes and had some time to test. I ran the cloud init commands before rebooting, after rebooting (via a jumphost on the private network), and after another reboot.

cloud-init single --frequency always --name write_files
cloud-init single --frequency always --name runcmd

I didn't notice any changes, and the nodes are still far away from being fine (no internet connectivity, no k3s starting up)

Switching to new nodes worked without issues, so no unexpected surprises there.

[mysticaltech]

Thanks for testing this @andi0b, at least we know now that this does not work. I will recommend using the --force-reboot kured flag for HA setups. It will always attempt proper draining, but will force reboot if that fails.

[laenion]

A new transactional-update release (version 4.6.8) fixing the error is currently staged in https://build.opensuse.org/request/show/1172470. I'm really sorry for the trouble!

[valkenburg-prevue-ch]

Well, I'm very relieved that in the end the culprit has been found! Well done.

[andi0b]

@laenion Thanks a lot for your help, I hope this is now fixed, and no worries, things happen, and now I can confirm my disaster recovery plans work well 😄

I was really questioning my sanity while researching this issue, because I thought this /etc issue couldn't be happening without some configuration mistakes on our side. Could you maybe elaborate how Micro OS or SLE Micro are commonly used, so this was not a more common issue affecting a lot more users?

[mysticaltech]

Thanks @laenion, really appreciate the fix! This was getting pretty stressful but it happens 🙏

[andi0b]

@mysticaltech transactional-update 4.6.8 was installed now on my nodes.

[mysticaltech]

Wonderful @andi0b, thanks for the update. I will make an announcement later today to upgrade the nodes to the latest version.