Adjust docs for password protected keystore (elastic#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

docs/reference/cluster.asciidoc

@@ -108,3 +108,5 @@ include::cluster/remote-info.asciidoc[]
 include::cluster/tasks.asciidoc[]
 
 include::cluster/voting-exclusions.asciidoc[]
+
+include::cluster/nodes-reload-secure-settings.asciidoc[]

docs/reference/cluster/nodes-reload-secure-settings.asciidoc

@@ -1,13 +1,7 @@
 [[cluster-nodes-reload-secure-settings]]
 == Nodes Reload Secure Settings
 
-The cluster nodes reload secure settings API is used to re-read the
-local node's encrypted keystore. Specifically, it will prompt the keystore
-decryption and reading across the cluster. The keystore's plain content is
-used to reinitialize all compatible plugins. A compatible plugin can be
-reinitialized without restarting the node. The operation is
-complete when all compatible plugins have finished reinitializing. Subsequently,
-the keystore is closed and any changes to it will not be reflected on the node.
+The cluster nodes reload secure settings API is used to re-load the keystore on each node.
 
 [source,console]
 --------------------------------------------------
@@ -21,9 +15,41 @@ The first command reloads the keystore on each node. The seconds allows
 to selectively target `nodeId1` and `nodeId2`. The node selection options are
 detailed <<cluster-nodes,here>>.
 
-Note: It is an error if secure settings are inconsistent across the cluster
-nodes, yet this consistency is not enforced whatsoever. Hence, reloading specific
-nodes is not standard. It is only justifiable when retrying failed reload operations.
+NOTE: {es} requires consistent secure settings across the cluster nodes, but this consistency is not enforced.
+Hence, reloading specific nodes is not standard. It is only justifiable when retrying failed reload operations.
+
+==== Reload Password Protected Secure Settings
+
+When the {es} keystore is password protected and not simply obfuscated, the password for the keystore needs
+to be provided in the request to reload the secure settings.
+Reloading the settings for the whole cluster assumes that all nodes' keystores are protected with the same password
+and is only allowed when {ref}/configuring-tls.html#tls-transport[node to node communications are encrypted]
+
+[source,js]
+--------------------------------------------------
+POST _nodes/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The common password that the {es} keystore is encrypted with in every node of the cluster.
+
+Alternatively the secure settings can be reloaded on a per node basis, locally accessing the API and passing the
+node-specific {es} keystore password.
+
+[source,js]
+--------------------------------------------------
+POST _nodes/_local/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The password that the {es} keystore is encrypted with on the local node.
+
 
 [float]
 [[rest-reload-secure-settings]]

docs/reference/setup/secure-settings.asciidoc

@@ -16,9 +16,6 @@ as part of the keystore.
 NOTE: All the modifications to the keystore take affect only after restarting
 Elasticsearch.
 
-NOTE: The elasticsearch keystore currently only provides obfuscation. In the future,
-password protection will be added.
-
 These settings, just like the regular ones in the `elasticsearch.yml` config file,
 need to be specified on each node in the cluster. Currently, all secure settings
 are node-specific settings that must have the same value on every node.
@@ -31,16 +28,33 @@ To create the `elasticsearch.keystore`, use the `create` command:
 
 [source,sh]
 ----------------------------------------------------------------
-bin/elasticsearch-keystore create
+bin/elasticsearch-keystore create -p
 ----------------------------------------------------------------
 
-The file `elasticsearch.keystore` will be created alongside `elasticsearch.yml`.
+You will be prompted to enter the keystore password and the file `elasticsearch.keystore` will be created alongside `elasticsearch.yml`, protected with the password you specified.
+
+NOTE: If you don't specify the `-p` flag or if you enter an empty password, the {es} keystore will be obfuscated but not password protected.
+
+[float]
+[[changing-keystore-password]]
+=== Changing the password of the keystore
+
+To change the password of the `elasticsearch.keystore`, use the `passwd` command.
+If the {es} keystore is password protected, you will be prompted to enter the current password and then enter the new one
+
+[source,sh]
+----------------------------------------------------------------
+bin/elasticsearch-keystore passwd
+----------------------------------------------------------------
+
+You can use the `passwd` subcommand to set a password to a previously obfuscated-only keystore, and remove the password from an encrypted keystore by setting it to an empty string.
 
 [float]
 [[list-settings]]
 === Listing settings in the keystore
 
-A list of the settings in the keystore is available with the `list` command:
+A list of the settings in the keystore is available with the `list` command.
+If the {es} keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -51,8 +65,8 @@ bin/elasticsearch-keystore list
 [[add-string-to-keystore]]
 === Adding string settings
 
-Sensitive string settings, like authentication credentials for cloud
-plugins, can be added using the `add` command:
+Sensitive string settings, like authentication credentials for cloud plugins, can be added using the `add` command.
+If the {es} keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -67,12 +81,22 @@ through stdin, use the `--stdin` flag:
 cat /file/containing/setting/value | bin/elasticsearch-keystore add --stdin the.setting.name.to.set
 ----------------------------------------------------------------
 
+You can overwrite existing entries in the keystore by setting the `-f` flag
+
+[source,sh]
+----------------------------------------------------------------
+bin/elasticsearch-keystore add -f the.existing.setting.name.to.set
+----------------------------------------------------------------
+
+NOTE: The `-f` flag will also force the creation of an obfuscated-only keystore, if one doesn't already exist.
+
 [float]
 [[add-file-to-keystore]]
 === Adding file settings
 You can add sensitive files, like authentication key files for cloud plugins,
 using the `add-file` command. Be sure to include your file path as an argument
 after the setting name.
+If the {es} keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -83,7 +107,8 @@ bin/elasticsearch-keystore add-file the.setting.name.to.set /path/example-file.j
 [[remove-settings]]
 === Removing settings
 
-To remove a setting from the keystore, use the `remove` command:
+To remove a setting from the keystore, use the `remove` command.
+If the {es} keystore is password protected, you will be prompted to enter the password:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -114,8 +139,8 @@ bin/elasticsearch-keystore upgrade
 Just like the settings values in `elasticsearch.yml`, changes to the
 keystore contents are not automatically applied to the running
 elasticsearch node. Re-reading settings requires a node restart.
-However, certain secure settings are marked as *reloadable*. Such settings
-can be re-read and applied on a running node.
+However, certain secure settings are marked as *reloadable*.
+Such settings can be {ref}/nodes-reload-secure-settings.html[re-read and applied on a running node].
 
 The values of all secure settings, *reloadable* or not, must be identical
 across all cluster nodes. After making the desired secure settings changes,
@@ -124,7 +149,13 @@ using the `bin/elasticsearch-keystore add` command, call:
 [source,console]
 ----
 POST _nodes/reload_secure_settings
+{
+  "reload_secure_settings": "s3cr3t" <1>
+}
 ----
+// NOTCONSOLE
+
+<1> The password that the {es} keystore is encrypted with.
 
 This API will decrypt and re-read the entire keystore, on every cluster node,
 but only the *reloadable* secure settings will be applied. Changes to other
@@ -133,8 +164,8 @@ the reload has been completed, meaning that all internal datastructures dependen
 on these settings have been changed. Everything should look as if the settings
 had the new value from the start.
 
-When changing multiple *reloadable* secure settings, modify all of them, on
-each cluster node, and then issue a `reload_secure_settings` call, instead
+When changing multiple *reloadable* secure settings, modify all of them, on each cluster node, and then issue a
+{ref}/nodes-reload-secure-settings.html[`reload_secure_settings`] call, instead
 of reloading after each modification.
 
 There are reloadable secure settings for: