Use new Salt mine's ACL features to share private data between minions

[gdemonet]

Component: salt

Why this is needed:

We need to share some private keys between master nodes (namely, the ServiceAccount token signing key and the etcd encryption key). Currently, we achieve this through a "trick":

• those keys are first generated during bootstrap
• they are mounted (actually, all of /etc/kubernetes/pki) in the Salt master container
• they are shared in a pillar key (metalk8s:private) to minions with the master role

The issue is: this only works if the bootstrap node also has the master role, which is not an obligation (well, because of this behaviour, it currently is).

What should be done:

Since Salt 3000.1, the Salt mine has implemented finer ACL features to control which minions can read which mine functions. We should let the first minion with master role generate these keys, publish them securely to the mine, and let other master minions read from it to write the keys locally.

More details:

• How to share private data between subset of minions: Canonical solution for securely transferring data between minions saltstack/salt#45882
• Salt mine ACL: [Feature]: Salt-mine access restrictions / permission granularity saltstack/salt#6437 (PR: Mine minion acl master saltstack/salt#55760)

Implementation proposal (strongly recommended):

Test plan:

We should make sure this approach works in case the original "owner" of the keys can die, in which case another minion should takeover and start publishing its own copy of the keys to the mine. In case all masters are lost, the first one brought back to highstate should regenerate the keys.