The Keysystem

Motivation

Hyades, as a cluster, needs to be secure. It is a known phenomenon that compute clusters with lots of resources but bad security are sometimes hacked simply so that the resources in them can be sold. Even beyond that, Hyades is designed to hold confidential personal information, like message histories and backups, and needs to be secure so that we don't leak private data to people who shouldn't have it.

Hyades has a threat model: that we have an omnipresent attacker, with full knowledge of how all of our systems work, with the credentials to at least one normal user account, and the ability to sniff, intercept, rewrite, or spoof traffic on our cluster network. (This includes the flannel network, but does not include loopback interfaces within individual containers. It includes loopback interfaces on container host machines, but not within containers or within machines that do not host containers.)

We need to be secure -- to not allow unauthorized access, to not permit data loss, to not allow private data to be read, et cetera -- from this kind of threat. This means that all connections between programs in the cluster must be encrypted, in a way that provides both authentication and confidentiality. All capabilities that should not be available to the general public should require authentication, and no user data should ever travel over an unconfidental link.

General solutions

There are a few approved solutions used by Hyades:

• Connections encrypted with TLS, including both server certificates and client certificates, and verified against certificate authorities that we control.
• Connections encrypted with SSH, with authorization checked against access lists or verified against certificate authorities.
• Downloads over HTTP (without TLS), only for non-confidential data, and only in combination with signature verification against known keys or known certificate authorities.
• Connections encrypted with TLS, where the server is authorized with a certificate as above and the client is authorized by having a unique, only-used-once access token. (Requires additional scrutiny.)
• Insecure connections authorized by another secure mechanism, where confidentiality is provided by being wrapped in a WireGuard network. (Requires additional scrutiny.)

In many of these cases, it is necessary for keys to be provisioned to individual nodes on the network. While Kubernetes can handle some amount of provisioning of secrets to containers, many of these are required even in the event that Kubernetes is not yet initialized or is otherwise nonfunctional, so we need a separate system.

The Keysystem

The Keysystem is the component of Homeworld that handles the distribution of encryption and authentication keys across the cluster, including both public keys and private keys. Each machine will often have a number of different such keys, for different services. The keyserver is generally hosted on one of the cluster's supervisor nodes.

When a new node is admitted, the following steps take place:

• The administrator runs a command to get a new admission token for this particular server.
• The administrator runs the homeworld installer on the new hardware, which includes the public key for the keyserver.
• The administrator types in the new node's IP address and the admission token obtained earlier.
• The new node finishes installing, and then begins the admission process.
• The new node connects to the keyserver (knowing its public key), and transmits its admission token.
• The keyserver verifies the new token and grants a certificate-granting certificate to the new node. (A certificate-granting certificate is a certificate that verifies the node's identity to the keysystem later, so that it can request and renew additional certificates from the keysystem in the future.)
• The new node then uses the certificate-granting certificate to request each of the other certificates that it needs from the keyserver.

The keysystem also has a few extra responsibilties around distributing configuration files to nodes during the admission process.

Unlike many of the components we use, the keysystem is designed expressly for Homeworld and Hyades, and is not an upstream project.

[insert more details about the keysystem here]

Additional Information

You can refer to the (slightly outdated) keysystem design document.