Docs say elasticsearch-keystore must be run as the elasticsearch user but that fails because of permissions

[jamshid]

Elasticsearch version (bin/elasticsearch --version):

 /usr/share/elasticsearch/bin/elasticsearch --version
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 7.5.2, Build: default/rpm/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z, JVM: 13.0.1

Plugins installed: []

# /usr/share/elasticsearch/bin/elasticsearch-plugin list
prometheus-exporter

JVM version (java -version):

# /usr/share/elasticsearch/jdk/bin/java -version
openjdk version "13.0.1" 2019-10-15

OS version (uname -a if on a Unix-like system):

Linux 5003b6fd18ce 5.4.39-linuxkit #1 SMP Fri May 8 23:03:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

The default install permissions on /etc/elasticsearch is that it's owned by root:elasticsearch but group elasticsearch cannot add files.

drwxr-s--- 1 root elasticsearch  4096 Dec  8 23:23 .
-rw-rw---- 1 root elasticsearch   160 Dec  8 23:23 elasticsearch.keystore

It's mentioned in issues like #44624 the keystore is created if it does not exist:

If the keystore does not exist, we auto create it. This is because we always need the keystore.seed value, and do not want archive users to need additional setup before running Elasticsearch.

But the elasticsearch service cannot auto create it because of directory permissions -- journalctl -u elasticsearch will show

Dec 08 23:23:46 0786d81661fb elasticsearch[757]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: org.elasticsearch.cli.UserException: un
Dec 08 23:23:46 0786d81661fb elasticsearch[757]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore.tmp

Seems the only way this works is to make elasticsearch the owner of /etc/elasticsearch, but that is not documented.

Steps to reproduce:

I don't understand why docs say to only run elasticsearch-keystore as the elasticsearch user
https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-keystore.html

Important: This command should be run as the user that will run Elasticsearch.

when that does not work:

[root@d889d59d4b36 /]# su -s /bin/bash elasticsearch   
bash-4.2$ /usr/share/elasticsearch/bin/elasticsearch-keystore list
keystore.seed
bash-4.2$ /usr/share/elasticsearch/bin/elasticsearch-keystore add test1
Enter value for test1: 
ERROR: unable to create temporary keystore at [/etc/elasticsearch/elasticsearch.keystore.tmp], write permissions required for [/etc/elasticsearch] or run [elasticsearch-keystore upgrade]

(elasticsearch-keystore upgrade did not report an error but it didn't help)

Provide logs (if relevant):

example provided above

This seems related to:
#26309
#64207

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[mark-vieira]

This issue is missing some details such as the Elasticsearch version and packaging type. I assume this is a deb or rpm package since those use /etc/elasticsearch for configuration but we always create a keystore on installation so I don't think this is an issue anymore. I'm going to close this issue for now since I think it's overcome by events.