Secure validator mode

[mrcnski]

Overview

Summary: Running --validator on a platform other than Linux x86-64 requires the --insecure-validator-i-know-what-i-do flag.

Due to #882 becoming high-priority for parathreads, we are now forced to provide a secure validator mode only for Linux x86-64 (to start).

We will still support other platforms with an --insecure-validator-i-know-what-i-do flag. (Naming follows interpreted-i-know-what-i-do.)

Todo

Preparation:

• Announce this change to validators ahead of time in a release notes Secure Mode Announcement w3f/polkadot-wiki#4881
• Document secure-mode. Document secure-validator mode w3f/polkadot-wiki#5346
• Make unavailable security features print a warning that says this will soon be required to run securely, with a link to the announcement. PVF host: Make unavailable security features print a warning #2244

Code:

• Make all candidate-validation security features a requirement to run in secure-mode.
  • Exception: if one FS feature is missing and the other is present, that's fine.
• Have a flag to run in "insecure mode"
• When running in secure-mode, link to https://wiki.polkadot.network/docs/maintain-guides-secure-validator on startup.

Followup:

• Add automatically flag to zombienet

Related

Previous draft PR: paritytech/polkadot#7073
Proposed announcement: w3f/polkadot-wiki#4881

[pepoviola]

Just a note to me. We should add the logic to enable the flag automatically in zombienet so users can still run from the current configs.

[mrcnski]

Once we have this we will officially be treating Linux as a first-class citizen so we can close #881.

[mrcnski]

Currently the "can-unshare-and-change-root" capability is required for secure-mode. But we just had a user for whom it failed: #2304 However, he was on a Linux version (5.15) which should support Landlock. So I think that either the unshare capability or the Landlock capability should be present, since either provides filesystem security. This way the user can still run securely without a CLI flag. :)

[hitchhooker]

I run validators in lx containers where host kernel has CONFIG_SECCOMP=y but client is not able to detect that maybe because kernel files are not mounted inside the lxc at all.

Nov 27 11:01:43 dot01 polkadot[1347648]: 2023-11-27 11:01:43 🚨 Your system cannot securely run a validator.
Nov 27 11:01:43 dot01 polkadot[1347648]: Running validation of malicious PVF code has a higher risk of compromising this machine.
Nov 27 11:01:43 dot01 polkadot[1347648]:   - Optional: Cannot enable landlock, a Linux 5.13+ kernel security feature: landlock ABI 1 not available
Nov 27 11:01:43 dot01 polkadot[1347648]:   - Cannot unshare user namespace and change root, which are Linux-specific kernel security features: not available: mount MS_BIND: No such file or directory (os error 2)
Nov 27 11:01:43 dot01 polkadot[1347648]: 2023-11-27 11:01:43 In the next release this will be a hard error by default.

what options do I have, should I abandon using proxmox completely or move to using less efficient qemu vm:s or just forget any provision at all?

current setup: https://github.com/rotkonetworks/unlabored

[mrcnski]

Hi @hitchhooker! I think you should be fine:

• The landlock error is most likely because the Linux version you are using doesn't support it. It will be optional if the unshare-and-change-root capability is available.
• The unshare-and-change-root error is likely due to this bug which has been fixed: PVF: Fix unshare "no such file or directory" error #2426. You can verify whether it was fixed for you by running the latest master.

If all else fails, you will have the option to disable this error with a CLI flag. If you see the error on the next release, it will tell you exactly what to do.

I hope that helps!

[hitchhooker]

Dec 14 13:33:59 ksm04 polkadot[418871]: 2023-12-14 13:33:59 🚨 Some security issues have been detected.
Dec 14 13:33:59 ksm04 polkadot[418871]: Running validation of malicious PVF code has a higher risk of compromising this machine.
Dec 14 13:33:59 ksm04 polkadot[418871]:   - Optional: Cannot enable landlock, a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced
Dec 14 13:33:59 ksm04 polkadot[418871]: 2023-12-14 13:33:59 👮‍♀️ Running in Secure Validator Mode. It is highly recommended that you operate according to our security guidelines.
Dec 14 13:33:59 ksm04 polkadot[418871]: More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode

7797 🕸 pct config 124                                                                                                             [~]
arch: amd64
cores: 4
description: kusama validator%3A ksm04.rotko.net%0A
features: nesting=1
hostname: ksm04.rotko.net
memory: 8192
mp0: tank04:subvol-124-disk-1,mp=/opt/polkadot,size=700G
net0: name=eth0,bridge=vmbr0,gw=192.168.xx.x,hwaddr=1E:DB:4C:8B:xx:xx,ip=192.168.xx.xxx/24,type=veth
onboot: 1
ostype: debian
rootfs: tank04:subvol-124-disk-0,size=20G
swap: 8192
unprivileged: 1

root@bkk04  𓅨 ~
7794 🕸 cat /boot/config-`uname -r` | grep CONFIG_SECCOMP=                                                                       ⏎ [~]
CONFIG_SECCOMP=y
root@bkk04  𓅨 ~
7796 🕸 pct enter 124                                                                                                            ⏎ [~]
root@ksm04:~# cat /boot/config-`uname -r` | grep CONFIG_SECCOMP=
cat: /boot/config-6.2.16-10-pve: No such file or **directory**

so any instructiuons how do I run secure validator setup inside lx containers?

https://github.com/proxmox/pve-kernel
as far as I understand proxmox kernel is just ubuntu kernel + openzfs support and some further security mitigations.

I tested running client on host instance of the machines(5.15.108-1-pve and 6.2.16-10-pve kernels) and im not getting errors, only when running with exact same kernel inside the LX container so there is some issue how the client detects the SECCOMP support.

[mrcnski]

Thanks @hitchhooker! Looks like the error is about Landlock, which is optional (fortunately in your case) because there is also another feature we use to sandbox the filesystem. The manual seccomp check fails but it looks like a false positive and the actual syscall still works. So unless I missed something, there is no issue. If you want to fix the warning about Landlock, you need at least Linux 5.13+; what does uname say inside the container?

Or, since you say the kernel is the exact same, what do you get when you run this? (from here)

dmesg | grep landlock || journalctl -kg landlock

[hitchhooker]

root@bkk04  𓅨 ~
7800 🕸 dmesg | grep landlock || journalctl -kg landlock                                                                           [~]
-- No entries --
root@bkk04  𓅨 ~
7801 🕸 pct enter 124                                                                                                            ⏎ [~]
root@ksm04:~# dmesg | grep landlock || journalctl -kg landlock
dmesg: read kernel buffer failed: Operation not permitted
-- No entries --
root@ksm04:~#

root@bkk04  𓅨 ~
7806 🕸 uname -r                                                                                                                 ⏎ [~]
6.2.16-10-pve
root@bkk04  𓅨 ~
7807 🕸 cat /boot/config-6.2.16-10-pve | grep CONFIG_SECURITY_LANDLOCK                                                             [~]
CONFIG_SECURITY_LANDLOCK=y

I guess we lack permissions for dmesg/kernelinfos inside the container.