Reconsider packaging changes to support elasticsearch.keystore

[jasontedor]

A previous change tried to create an elasticsearch.keystore on startup, if one does not exist (#26149).

This is fine for the archive distributions, but it is incompatible with the packaging distributions because the elasticsearch group does not have write permissions to /etc/elasticsearch.

To accommodate this, we now create the elasticsearch.keystore on package installation, if it does not exist (#26282).

This creates issues on package removal because now we have an untracked file in the configuration directory (this is why the packaging tests are currently failing).

One way to handle this is write an md5 to the directory and remove the elasticsearch.keystore on package removal if the md5 has not changed. Except, RPM and Debian packaging have different behavior here. RPMs always remove configuration files if they are unchanged, Debian only removes if purging the package but not on plain package removal. By having this configuration file not tracked by the underlying packaging, we have to work to replicate the individual packaging behavior in our post-install scripts. This is annoying.

There are mechanisms for handling files that we want to be tracked by the package that are not installed by the package (e.g., ghost files in RPM, and using ucf in Debian).

We should consider these as options for handling this situation.

Sadly, the dependency that we are using for building the packages (nebula-plugins/gradle-ospackage-plugin) does not support this, so we'll have to roll our sleeves up.

[Mpdreamz]

Cc @elastic/microsoft

[jasontedor]

We have a path forward here. We are going to remove the auto-keystore creation from Elasticsearch and from the package installation. Instead, plugins can register if they require a keystore to exist. If a plugin registers that it does, the plugin installer will create this keystore if it does not exist, and will add the keystore.seed setting if it does not exist.

Separately, we will investigate restricting plugins to only be able to read their own config directory.