RPM verification failure on RHEL 8 in FIPS mode

[andrewvillano]

Looks like due to this : https://bugzilla.redhat.com/show_bug.cgi?id=1728031

I am unable to install elasticsearch from the repository as it warns : package elasticsearch-0:7.7.1-1.x86_64 does not verify: no digest.

Running RHEL 8.2

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[rjernst]

The elasticsearch rpm package does not yet have a sha256 header, just sha1 and md5. Unfortunately I don't think we will be able to add this anytime soon. We use the Nebula ospackage gradle plugin to build our rpm, which does not yet have support for the sha256 header. I also looked at Redline, the library ospackage is built on top of, but that also does not have support. It might be possible to insert a custom header in Redline, but the redline builder needed for this is hidden within ospackage and not exposed.

So, short of completely changing the infrastructure we build our rpms on, I don't see a way to add the necessary sha256 that RHEL 8 with FIPS is requiring.

[ejsears]

This is a huge issue for DoD. Manual installation of RPMs across large clusters is going back 10 years in technology. Bypassing file integrity checks does not bode well. This affects every RPM package that Elastic publishes and as more people move to RHEL 8, it will become more and more of a problem.

Same issue in Kibana

[jaymode]

The team discussed this today and will work on getting support added for SHA256 to redline and then see about ospackage moving to that version of redline. Separately we will ensure we have packaging tests that will run in an environment with FIPS/OSPP enabled.

[ejsears]

Any update on these issues?

[jordanenglish]

Just wanted to drop a +1 for this. Hoping to see this implemented.

[fjoenichols]

We're also needing this implemented.

[breskeby]

@jaymode any update on this? Once this is available in redline I might be able to look into providing a patch to ospackage

[breskeby]

A small heads up. I've just raised a Pull Request on the redline library that ES is using to build RPMs that adds sha256 header support. see craigwblake/redline#157 I'll work on a temporally solution to get these changes into the ES build until this PR gets merged and ends up in a released version of redline.

[breskeby]

I've created a PR that brings the changes in redline I've added to our elasticsearch build (see #75569). Now when verifying signatures of the rpm we see

./gradlew :distribution:packages:buildRpm
rpm --checksig -v distribution/packages/rpm/build/distributions/elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm
distribution/packages/rpm/build/distributions/elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    MD5 digest: OK

[breskeby]

It seems just adding the sha256header is not enough to get our RPMs working in FIPS environments. I spun up a centos8 tips enabled ci machine and test out rpm installation there with the latest tweaks:

[rene@breskeby-ci-centos8-fips-test ~]$ sudo rpm -ivh elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
	package elasticsearch-0:8.0.0~SNAPSHOT-1.x86_64 does not verify: no digest

[rene@breskeby-ci-centos8-fips-test ~]$ rpm --checksig -v elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm
elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm:
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: NOTFOUND
    MD5 digest: NOTFOUND

I think its complaining about missing MD5 digests and Payload SHA256 digest. I'll take some time to look into this. The md5 checksums should be there but I assume we see a problem that the digest is stored in outdated RPM tags

[breskeby]

We applied further changes to the redline library we use in elasticsearch to make our RPM packages FIPS compliant: #75569

With this change I was able to install our nightly ES rpm (https://snapshots.elastic.co/8.0.0-36034684/downloads/elasticsearch/elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm) in a centos-8 tips enabled environment:

[rene@breskeby-ci-centos8-fips-test ~]$ rpm --checksig -v elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm
elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm:
    Header V4 RSA/SHA512 Signature, key ID d88e42b4: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID d88e42b4: OK
[rene@breskeby-ci-centos8-fips-test ~]$ sudo rpm -iv elasticsearch-8.0.0-SNAPSHOT-x86_64.rpm
Verifying packages...
Preparing packages...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
elasticsearch-0:8.0.0~SNAPSHOT-1.x86_64
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
[/usr/lib/tmpfiles.d/elasticsearch.conf:1] Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/pesign.conf:1] Line references path below legacy directory /var/run/, updating /var/run/pesign → /run/pesign; please update the tmpfiles.d/ drop-in file accordingly.
[rene@breskeby-ci-centos8-fips-test ~]$

[breskeby]

Do we have the infrastructure to setup a RHEL 8 in FIPS mode to test this against RHEL 8 and not just on CENTOS 8?

[ejsears]

I'd be more than happy to test it on a RHEL system.

[bytebilly]

Thanks @breskeby! We are working to perform tests in a RHEL8 FIPS environment too.

[chuckmilam]

Sorry for the pile-on here, but we hit this today trying to deploy elasticsearch 7.14 RPMs on our FIPS-enabled RHEL 8 systems.

Great to see the progress on the 8.0 builds, when can we expect these fixes on the 7.x RPM packages?

[mark-vieira]

@chuckmilam the changes to the RPM package for FIPS compatibility will be available in the upcoming 7.15 release which will drop shortly.

[chuckmilam]

@chuckmilam the changes to the RPM package for FIPS compatibility will be available in the upcoming 7.15 release which will drop shortly.

@mark-vieira Should I be concerned this issue isn't listed in the 7.15 release notes?

[mark-vieira]

Sorry, not sure why it slipped through the release notes but it should have been mentioned there. This is actually the PR of note: #76440

[chuckmilam]

Excellent! Thank you. Now standing by for the new RPM builds to appear in the Elastic repos. How often does that packaging build pipeline run?

[mark-vieira]

We're having some trouble with the YUM repos at the moment, but you can grab the updated RPMs directly here: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#install-rpm

[chuckmilam]

Thanks! I will test with these, but we need the YUM repos as part of our automation. We're using the elastic/ansible-elasticsearch Ansible roles wherever we can.

[chuckmilam]

Confirming a locally-downloaded elasticsearch-7.15.0-x86_64.rpm installed without complaint on a STIG and FIPS-compliant RHEL 8 system, where the v.7.14.2 RPM would not.

Looks like we have a winner. Thanks, Elastic team.

Looking forward to the Yum repo updates.

[mark-vieira]

Great to hear @chuckmilam. The team is working on the YUM repo as we speak.

[mark-vieira]

@chuckmilam the Elasticsearch YUM repos are now up to date so you should be able to pull from there now:

https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html

[chuckmilam]

Looking good here! Thanks.

[breskeby]

@chuckmilam thanks for the feedback and testing. highly appreciated