[DOCS] RHEL Security TMP override #77285

stefnestor · 2021-09-04T02:23:25Z

Adding content for top last 30d Elastic Discuss post which covers a BlackHat RHEL start-up error with this solution.

Not sure if it makes sense to include "if you don't do this, you'll get these errors" or some kind of better framing/context. Putting in raw helpful content for review 🙏🏼

Adding content for [top last 30d Elastic Discuss post](https://discuss.elastic.co/t/elasticsearch-v7-6-2-failed-to-start-killed-by-sigabrt-on-rhel-7-7-urgent/231039) which covers a BlackHat RHEL start-up error with [this solution](https://discuss.elastic.co/t/elasticsearch-v7-6-2-failed-to-start-killed-by-sigabrt-on-rhel-7-7-urgent/231039/11).

elasticmachine · 2021-09-04T02:23:27Z

Pinging @elastic/es-docs (Team:Docs)

elasticmachine · 2021-09-09T17:40:25Z

Pinging @elastic/es-delivery (Team:Delivery)

mark-vieira · 2021-09-09T18:00:27Z

@pugnascotia were you involved in tracking down this issue? If so you can give the 👍 on this?

pugnascotia · 2021-09-10T09:01:36Z

docs/reference/setup/install/rpm.asciidoc

@@ -123,6 +123,17 @@ endif::[]

 include::skip-set-kernel-parameters.asciidoc[]

+[[install-rhel-security]]
+==== Override Red Hat TMP if Security Compliance


Should be something like the following? This feels convoluted.

Suggested change

==== Override Red Hat TMP if Security Compliance

==== Override Red Hat TMP if Security Compliant

Compliant is better than Compliance, but "Security Compliant" is still vague. Can we be more specific here and say something like, "Override Red Hat TMP if FIPS mode is enabled"?

Unfortunately it's nothing to do with FIPS mode - these steps are necessary today if the user is using SELinux with a particular config (which we don't test and haven't even fully characterised). Admittedly there will be significant overlap between users of FIPS mode and users of this kind of SELinux config.

pugnascotia · 2021-09-10T09:02:40Z

docs/reference/setup/install/rpm.asciidoc

+==== Override Red Hat TMP if Security Compliance
+
+If you are applying a security compliance in your RHEL installation 
+you must change the path of the TMP directory that will use elasticsearch as Java.


Suggested change

you must change the path of the TMP directory that will use elasticsearch as Java.

you must change the path of the TMP directory that Java and {es} will use.

pugnascotia · 2021-09-10T09:02:53Z

docs/reference/setup/install/rpm.asciidoc

+
+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`
+2. Add `ES_TMPDIR=/usr/share/elasticsearch/tmp` under `/etc/sysconfig/elasticsearch`
+3. Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are `0755`


Suggested change

3. Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are `0755`

3. Create the `/usr/share/elasticsearch/tmp` directory and make sure that the owner and group are `elasticsearch` and the permissions are `0755`.

If step 4 is deemed unacceptable, I'd add a note to this step indicating that no users other than the elasticsearch user should have access to this directory.

pugnascotia · 2021-09-10T09:03:38Z

@DaveCTurner knows more about this than me, so I've added him as a reviewer.

DaveCTurner

I think we shouldn't copy the instructions from the forum post without validating them and considering some simpler alternatives. I believe we could fix this whole thing ourselves by setting TMPDIR to the value of ES_TMPDIR in our startup scripts. Does anything bad happen if we set TMPDIR for every Linux user?

A more targeted fix will be #77014 but this needs JNA to upgrade their libffi and then we need to upgrade our JNA so it'll be a while before we can do that.

I'm also not sure we actually need to override ES_TMPDIR in these environments either. If you hit the segfault in the linked forum post then we know that JNA's temporary directory supports executables so it should be enough to tell libffi to use the same location.

DaveCTurner · 2021-09-10T11:04:56Z

docs/reference/setup/install/rpm.asciidoc

+If you are applying a security compliance in your RHEL installation 
+you must change the path of the TMP directory that will use elasticsearch as Java.
+
+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`


I think we shouldn't be telling users to edit jvm.options, they should be adding extra options files in jvm.options.d.

But then I think this isn't necessary anyway, we already copy ES_TMPDIR to java.io.tmpdir IIRC.

Yes, we tell people to add custom options files. If this step is necessary, I'd cross reference the info on setting JVM options (https://www.elastic.co/guide/en/elasticsearch/reference/master/advanced-configuration.html#set-jvm-options). Something like:

In a <<set-jvm-options, custom JVM options>> file, override the -Djava.io.tempdir setting:

-Djava.io.tmpdir=${ES_TMPDIR}

I can confirm this is unnecessary: we already have this setting in the default jvm.options file:

elasticsearch/distribution/src/config/jvm.options

Lines 64 to 65 in f8793c6

## JVM temporary directory

-Djava.io.tmpdir=${ES_TMPDIR}

DaveCTurner · 2021-09-10T11:05:39Z

docs/reference/setup/install/rpm.asciidoc

+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`
+2. Add `ES_TMPDIR=/usr/share/elasticsearch/tmp` under `/etc/sysconfig/elasticsearch`
+3. Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are `0755`
+4. Lastly make sure that `/dev/shm` doesn't have the noexec attribute with command: `mount | grep tmpfs | grep '/dev/shm'`


This is probably unacceptable to many users of locked-down systems. Instead we should set TMPDIR (eventually LIBFFI_TMPDIR to point to a location to which only the elasticsearch user has access.

pugnascotia · 2021-09-10T13:14:29Z

@DaveCTurner so we also have #74163, and I don't think it's too much of a stretch to expand that to just using $ES_HOME/tmp for all tmp-related matters. What do you think?

DaveCTurner · 2021-09-10T16:20:15Z

#74163 is related but kinda different, libffi doesn't care about system properties so fixing the temp dir for libffi is fairly orthogonal to overriding java.io.tmpdir. Not that I hold a strong opinion about how we organise these issues, up to you and the core/infra folks really.

debadair

I added some comments and suggestions. It took me a bit to wrap my head around what was going on here, so not 100% sure the details are correct. However, I do thing we need to work in a bit more context about what you're doing and why.

Also, we should add an IMPORTANT note to https://www.elastic.co/guide/en/elasticsearch/reference/master/important-settings.html#es-tmpdir that indicates that you must use a dedicated TMP directory outside of the system TMP directory if you are running on a system with FIPS mode enabled.

debadair · 2021-10-11T22:51:34Z

docs/reference/setup/install/rpm.asciidoc

@@ -123,6 +123,17 @@ endif::[]

 include::skip-set-kernel-parameters.asciidoc[]

+[[install-rhel-security]]
+==== Override Red Hat TMP if Security Compliance


Compliant is better than Compliance, but "Security Compliant" is still vague. Can we be more specific here and say something like, "Override Red Hat TMP if FIPS mode is enabled"?

debadair · 2021-10-11T23:05:10Z

docs/reference/setup/install/rpm.asciidoc

+If you are applying a security compliance in your RHEL installation 
+you must change the path of the TMP directory that will use elasticsearch as Java.
+
+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`


Yes, we tell people to add custom options files. If this step is necessary, I'd cross reference the info on setting JVM options (https://www.elastic.co/guide/en/elasticsearch/reference/master/advanced-configuration.html#set-jvm-options). Something like:

In a <<set-jvm-options, custom JVM options>> file, override the -Djava.io.tempdir setting:

-Djava.io.tmpdir=${ES_TMPDIR}

debadair · 2021-10-11T23:15:01Z

docs/reference/setup/install/rpm.asciidoc

+you must change the path of the TMP directory that will use elasticsearch as Java.
+
+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`
+2. Add `ES_TMPDIR=/usr/share/elasticsearch/tmp` under `/etc/sysconfig/elasticsearch`


I'd make this the first step, even if you do need to explicitly set the JVM TMP directory to ES_TMPDIR.

Suggested change

2. Add `ES_TMPDIR=/usr/share/elasticsearch/tmp` under `/etc/sysconfig/elasticsearch`

2. Configure a dedicated {es} TMP directory outside of the system TMP directory:

+

[source,shell]

----

ES_TMPDIR=/usr/share/elasticsearch/tmp

---

debadair · 2021-10-11T23:17:55Z

docs/reference/setup/install/rpm.asciidoc

+
+1. Uncomment `-Djava.io.tmpdir=${ES_TMPDIR}` under `/etc/elasticsearch/jvm.options`
+2. Add `ES_TMPDIR=/usr/share/elasticsearch/tmp` under `/etc/sysconfig/elasticsearch`
+3. Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are `0755`


If step 4 is deemed unacceptable, I'd add a note to this step indicating that no users other than the elasticsearch user should have access to this directory.

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <[email protected]>

DaveCTurner · 2021-11-15T08:28:16Z

Now that #80651 is merged I think users don't need to do most of this stuff. It should be sufficient to just set ES_TMPDIR to point somewhere other than a subdirectory of /tmp. If we want to write step-by-step instructions for that then IMO it'd be better in docs/reference/setup/sysconfig/executable-jna-tmpdir.asciidoc since they are applicable to all distributions, not just RPM. Admittedly it affects RHEL folks more than others, so a cross-link from these RPM docs would be helpful.

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <[email protected]>

Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <[email protected]> Co-authored-by: Rory Hunter <[email protected]>

* Set LIBFFI_TMPDIR at startup (#80651) Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <[email protected]> * Fix incorrect SSL usage Co-authored-by: Rory Hunter <[email protected]>

elasticsearchmachine · 2022-03-30T17:55:48Z

@stefnestor please enable the option "Allow edits and access to secrets by maintainers" on your PR. For more information, see the documentation.

stefnestor · 2022-03-31T13:47:04Z

@debadair I'm suppose to ... but I don't show the option from the documentation ... . Can you help me convert to whatever Docs needs to move forward? 🙏🏼

@stefnestor please enable the option "Allow edits and access to secrets by maintainers" on your PR. For more information, see the documentation.

screenshot

pugnascotia · 2022-03-31T14:03:57Z

Gah. The GitHub API isn't very helpful when it comes to detecting the "Allow edits and access to secrets by maintainers" setting, sorry for the noise

stefnestor · 2023-04-21T18:34:13Z

Deb's no longer with Elastic ( cc: @elastic/docs ) & per @DaveCTurner ... above, I'll side with this no longer being needed & close. Cheers all!

Now that #80651 is merged I think users don't need to do most of this stuff.

stefnestor added >docs General docs changes Team:Docs Meta label for docs team labels Sep 4, 2021

stefnestor requested a review from jrodewig September 4, 2021 02:23

elasticsearchmachine added v8.0.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Sep 4, 2021

jrodewig changed the title ~~[+DOC] RHEL Security TMP override~~ [DOCS] RHEL Security TMP override Sep 9, 2021

jrodewig added the :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts label Sep 9, 2021

elasticmachine added the Team:Delivery Meta label for Delivery team label Sep 9, 2021

jrodewig requested a review from pugnascotia September 9, 2021 18:05

pugnascotia reviewed Sep 10, 2021

View reviewed changes

pugnascotia requested a review from DaveCTurner September 10, 2021 09:03

DaveCTurner reviewed Sep 10, 2021

View reviewed changes

bitNcode mentioned this pull request Sep 19, 2021

Use ES_TMPDIR instead of java.io.tmpdir #74163

Closed

jrodewig requested review from debadair and removed request for jrodewig October 4, 2021 12:54

debadair reviewed Oct 11, 2021

View reviewed changes

DaveCTurner mentioned this pull request Nov 11, 2021

Set LIBFFI_TMPDIR at startup #80651

Merged

DaveCTurner mentioned this pull request Nov 15, 2021

Set LIBFFI_TMPDIR at startup (#80651) #80698

Merged

DaveCTurner mentioned this pull request Nov 15, 2021

Set LIBFFI_TMPDIR at startup (#80651) #80699

Merged

arteam added v8.1.0 and removed v8.0.0 labels Jan 12, 2022

mark-vieira added v8.2.0 and removed v8.1.0 labels Feb 2, 2022

salvatore-campagna added v8.3.0 and removed v8.2.0 labels Mar 30, 2022

craigtaverner added v8.4.0 and removed v8.3.0 labels May 25, 2022

elasticsearchmachine changed the base branch from master to main July 22, 2022 23:10

mark-vieira added v8.5.0 and removed v8.4.0 labels Jul 27, 2022

csoulios added v8.6.0 and removed v8.5.0 labels Sep 21, 2022

kingherc added v8.7.0 and removed v8.6.0 labels Nov 16, 2022

rjernst added v8.8.0 and removed v8.7.0 labels Feb 8, 2023

stefnestor closed this Apr 21, 2023

stefnestor deleted the stefnestor-patch-2 branch November 9, 2023 23:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] RHEL Security TMP override #77285

[DOCS] RHEL Security TMP override #77285

stefnestor commented Sep 4, 2021

elasticmachine commented Sep 4, 2021

elasticmachine commented Sep 9, 2021

mark-vieira commented Sep 9, 2021

pugnascotia Sep 10, 2021

debadair Oct 11, 2021

DaveCTurner Oct 12, 2021

pugnascotia Sep 10, 2021

pugnascotia Sep 10, 2021 •

edited by debadair

Loading

debadair Oct 11, 2021

pugnascotia commented Sep 10, 2021

DaveCTurner left a comment

DaveCTurner Sep 10, 2021

debadair Oct 11, 2021

DaveCTurner Oct 12, 2021

DaveCTurner Sep 10, 2021

pugnascotia commented Sep 10, 2021

DaveCTurner commented Sep 10, 2021

debadair left a comment

debadair Oct 11, 2021

debadair Oct 11, 2021

debadair Oct 11, 2021

debadair Oct 11, 2021

DaveCTurner commented Nov 15, 2021

elasticsearchmachine commented Mar 30, 2022

stefnestor commented Mar 31, 2022

pugnascotia commented Mar 31, 2022

stefnestor commented Apr 21, 2023

	==== Override Red Hat TMP if Security Compliance
	==== Override Red Hat TMP if Security Compliant

	you must change the path of the TMP directory that will use elasticsearch as Java.
	you must change the path of the TMP directory that Java and {es} will use.

	3. Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are `0755`
	3. Create the `/usr/share/elasticsearch/tmp` directory and make sure that the owner and group are `elasticsearch` and the permissions are `0755`.

[DOCS] RHEL Security TMP override #77285

[DOCS] RHEL Security TMP override #77285

Conversation

stefnestor commented Sep 4, 2021

elasticmachine commented Sep 4, 2021

elasticmachine commented Sep 9, 2021

mark-vieira commented Sep 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pugnascotia Sep 10, 2021 • edited by debadair Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pugnascotia commented Sep 10, 2021

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pugnascotia commented Sep 10, 2021

DaveCTurner commented Sep 10, 2021

debadair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DaveCTurner commented Nov 15, 2021

elasticsearchmachine commented Mar 30, 2022

stefnestor commented Mar 31, 2022

pugnascotia commented Mar 31, 2022

stefnestor commented Apr 21, 2023

pugnascotia Sep 10, 2021 •

edited by debadair

Loading