Support for FIPS compliance mode #14912

beanuwave · 2024-07-23T15:34:16Z

Description

FIPS gradle build script is removed.
All BC dependencies are replaces by BCFIPS.
Password matcher inside Identity-Shiro that replies on BC to check if hashed passwords matches with OpenBSDBCrypt, is replaced by password4j implementation.
Adds support for BCFKS format (*.bks) for Key & Truststores.
Refactor parsing private keys with formats EC, PKCS8, PKCS1, DSA, w/wo encryption, w/wo parameters.
FIPS approved-only mode can be configured over opensearch.yml file.
java security file is added to the build.
java policy file is altered to grant neccessary security permissions.

This PR provides FIPS 140-2 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running application in approved-only mode restricts BCFIPS provoder to rely solely on FIPS certified cyphers. Due to replacement of BC libraries, BCrypt password matching and private-key loading from file were replaced by alternative implementations.

Reasons for refactoring PemUtils.java that is used by Reindex API, in case of migrating data from a remote cluster that is TLS protected:

PKCS#8 implementation was not supported by BCFIPS library.
java type security.
Password Based Key Derivation Functions such as PKCS#12 and OpenSSL are not supported in BCFIPS approved-only mode, because only PBKDF2 standard is approved for use in FIPS.
generally good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-07-23T15:41:17Z

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-24T08:51:58Z

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-24T08:54:08Z

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dancristiancecoi · 2024-07-24T10:16:03Z

.../identity-shiro/src/main/java/org/opensearch/identity/shiro/realm/BCryptPasswordMatcher.java

What is the use case of the identity-shiro plugin inside OpenSearch? Is it used as an alternative to the security plugin for authentication or does it have a different use?

As far as I understand we will need to add support for PBKDF2 to this plugin as BCrypt itself is not FIPS validated.

Yes this is also mine understanding what identity-shiro plugin/adapter is used for. As we do not really rely on BCrypt algo implementation here, it can be replaced by PBKDF2WithHmacSHA256 or similar.
Sidenote: most of the code is just copy&paste from OS security module, as it has password4j BCrypt hashing and matching for password, related to FIPS compliance.

@dancristiancecoi IMO identity-shiro should be placed sanbox/plugins or sandbox/modules. Its an example implementation of an experimental IdentityPlugin.

Relevant issue: #5834

Thanks for the clarification @cwperks !!

dblock · 2024-07-24T15:11:32Z

Could use some help maybe from @cwperks or @peternied reviewing this, please.

terryquigleysas · 2024-07-24T15:53:07Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java

@@ -1182,6 +1182,7 @@ private void createConfiguration() {
        baseConfig.put("indices.breaker.total.use_real_memory", "false");
        // Don't wait for state, just start up quickly. This will also allow new and old nodes in the BWC case to become the master
        baseConfig.put("discovery.initial_state_timeout", "0s");
+        baseConfig.put("fips.approved", "true");


When we proposed a similar flag previously we were encouraged not to use boolean flags to control behaviour like this opensearch-project/security#3420 (comment)

I am of the opinion that this is fine, in this case, I am not sure what greater configuration would make sense. Unless you can provide different approvers or something, there is no reason for this not to be a boolean.

peternied · 2024-07-25T12:41:58Z

@beanuwave I've marked this pull request as 'draft' in GitHub, feel free to keep iterating and let us know if there is anything you'd like attention on. I'll keep and eye out, but I'll give it a more detailed look when you've marked this change as Ready for review.

terryquigleysas · 2024-07-25T15:07:33Z

@beanuwave @terryquigleysas Thinking about backwards compatibility, would there be an opportunity to pass a flag at build-time to create a distribution that is FIPS-compliant? Would it make sense to have multiple distributions? 1 with bouncycastle (BC) included and 1 without BC?

Looking at the companion PR in the security plugin, I was wondering if it made sense to add logic to check if the BouncyCastleProvider class was on the classpath. If it is on the classpath then can it add the bouncycastle provider? I've written some similar code before in netty: https://github.com/netty/netty/blob/4.1/handler/src/main/java/io/netty/handler/ssl/BouncyCastlePemReader.java#L72-L94

Similar to the provider, is it possible to have different plugin-security.policy files depending on whether bouncycastle is provided or not?

I have replied at opensearch-project/security#4588 (comment)

github-actions · 2024-08-07T17:21:55Z

❌ Gradle check result for 84e7aa9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

beanuwave · 2024-08-07T17:40:14Z

All tests are now successfully running without requiring the BC libraries, by leveraging only the SUN and BCFIPS providers. This demonstrates that including BC libraries in the build process is not essential. The next step is to identify which code cannot operate under FIPS approved-only mode and address these limitations. Notably, the following components have constraints under FIPS compliance (as detailed here):

Azure Classic Discovery Plugin: Utilizes JKS keystore for certificates.
Ingest Attachment Plugin.
KeyStoreWrapper: Handles both v1 and v2 of the internal keystore.
General usage of PKCS#12 and PBE standards.
SQL CLI Plugin.

Plugin support remains an uncertain area. However, as long as plugins do not interfere with the SecurityManager and permissions, there should be minimal concern.

github-actions · 2024-08-08T08:24:30Z

❌ Gradle check result for 219e9dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

dancristiancecoi · 2024-08-08T09:20:39Z

Plugin support remains an uncertain area. However, as long as plugins do not interfere with the SecurityManager and permissions, there should be minimal concern.

There can be instances when other plugins interfere with the SecurityManager and permissions as described in this comment.

github-actions · 2024-08-08T09:47:31Z

❌ Gradle check result for dc753b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-08-08T14:34:19Z

❌ Gradle check result for fd825eb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-08-08T16:27:06Z

❌ Gradle check result for 9bb1591: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-08-09T13:14:12Z

❕ Gradle check result for 5ab8df8: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-08-09T13:16:27Z

Codecov Report

Attention: Patch coverage is 69.23077% with 24 lines in your changes missing coverage. Please review.

Project coverage is 71.89%. Comparing base (97c1bf0) to head (5ab8df8).
Report is 11 commits behind head on main.

Files	Patch %	Lines
.../main/java/org/opensearch/bootstrap/Bootstrap.java	0.00%	7 Missing ⚠️
...ch/identity/shiro/realm/BCryptPasswordMatcher.java	54.54%	2 Missing and 3 partials ⚠️
...g/opensearch/tools/launchers/SystemJvmOptions.java	0.00%	3 Missing ⚠️
...rg/opensearch/gradle/OpenSearchTestBasePlugin.java	0.00%	2 Missing ⚠️
...n/java/org/opensearch/common/ssl/KeyStoreUtil.java	0.00%	1 Missing and 1 partial ⚠️
...va/org/opensearch/common/ssl/SslConfiguration.java	33.33%	1 Missing and 1 partial ⚠️
...g/opensearch/tools/launchers/JvmOptionsParser.java	0.00%	1 Missing ⚠️
...n/java/org/opensearch/common/ssl/PemKeyConfig.java	80.00%	1 Missing ⚠️
...a/org/opensearch/common/settings/FipsSettings.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14912      +/-   ##
============================================
+ Coverage     71.74%   71.89%   +0.14%     
+ Complexity    62904    62900       -4     
============================================
  Files          5178     5179       +1     
  Lines        295167   294957     -210     
  Branches      42679    42636      -43     
============================================
+ Hits         211774   212063     +289     
+ Misses        66011    65472     -539     
- Partials      17382    17422      +40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-08-30T17:12:57Z

❌ Gradle check result for 8e5237f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Iwan Igonin <[email protected]> # Conflicts: # server/build.gradle

Signed-off-by: Iwan Igonin <[email protected]> # Conflicts: # client/rest/build.gradle # distribution/tools/plugin-cli/build.gradle # server/src/main/resources/org/opensearch/bootstrap/test-framework.policy

Signed-off-by: Iwan Igonin <[email protected]>

…ional tests. Signed-off-by: Iwan Igonin <[email protected]>

Signed-off-by: Iwan Igonin <[email protected]>

github-actions · 2024-09-23T09:27:52Z

❌ Gradle check result for 7e202a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

beanuwave changed the title ~~Draft to allow run in FIPS compliace mode~~ Draft to allow run in FIPS compliance mode Jul 24, 2024

beanuwave force-pushed the fips_compliance2 branch from 8e8ed47 to 6016d5d Compare July 24, 2024 08:46

beanuwave marked this pull request as ready for review July 24, 2024 10:02

beanuwave requested review from peternied, anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners July 24, 2024 10:02

dancristiancecoi reviewed Jul 24, 2024

View reviewed changes

terryquigleysas reviewed Jul 24, 2024

View reviewed changes

peternied marked this pull request as draft July 25, 2024 12:31

peternied changed the title ~~Draft to allow run in FIPS compliance mode~~ Support for FIPS compliance mode Jul 25, 2024

beanuwave force-pushed the fips_compliance2 branch from 219e9dc to dc753b0 Compare August 8, 2024 09:43

beanuwave force-pushed the fips_compliance2 branch from fd825eb to 9bb1591 Compare August 8, 2024 15:35

beanuwave force-pushed the fips_compliance2 branch from 9bb1591 to 5ab8df8 Compare August 9, 2024 12:18

iigonin added 7 commits September 20, 2024 11:56

Draft to allow run in FIPS compliace mode

7471cc9

Signed-off-by: Iwan Igonin <[email protected]> # Conflicts: # server/build.gradle

make tests run without BC (not BCFIPS) libraries.

aadacea

Signed-off-by: Iwan Igonin <[email protected]> # Conflicts: # client/rest/build.gradle # distribution/tools/plugin-cli/build.gradle # server/src/main/resources/org/opensearch/bootstrap/test-framework.policy

disable approved-only mode for launch configuration of testcluster

633cc02

Signed-off-by: Iwan Igonin <[email protected]>

update all BC libraries to support JAVA 21

4f8dce4

Signed-off-by: Iwan Igonin <[email protected]>

try to pass bwc staged build by adding expected version

8910870

Signed-off-by: Iwan Igonin <[email protected]>

ensure FIPS-mode can be enabled by fixing existent and creating addit…

af35764

…ional tests. Signed-off-by: Iwan Igonin <[email protected]>

fixing additional tests.

7e202a2

Signed-off-by: Iwan Igonin <[email protected]>

beanuwave force-pushed the fips_compliance2 branch from 8e5237f to 7e202a2 Compare September 23, 2024 09:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for FIPS compliance mode #14912

Support for FIPS compliance mode #14912

beanuwave commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Jul 24, 2024

dancristiancecoi Jul 24, 2024

beanuwave Jul 24, 2024

cwperks Jul 24, 2024 •

edited

Loading

dancristiancecoi Jul 24, 2024

dblock commented Jul 24, 2024

terryquigleysas Jul 24, 2024

stephen-crawford Jul 24, 2024

peternied commented Jul 25, 2024

terryquigleysas commented Jul 25, 2024

github-actions bot commented Aug 7, 2024

beanuwave commented Aug 7, 2024

github-actions bot commented Aug 8, 2024

dancristiancecoi commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 9, 2024

codecov bot commented Aug 9, 2024

github-actions bot commented Aug 30, 2024

github-actions bot commented Sep 23, 2024

Support for FIPS compliance mode #14912

Are you sure you want to change the base?

Support for FIPS compliance mode #14912

Conversation

beanuwave commented Jul 23, 2024

Description

Related Issues

Check List

github-actions bot commented Jul 23, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Jul 24, 2024

dancristiancecoi Jul 24, 2024

Choose a reason for hiding this comment

beanuwave Jul 24, 2024

Choose a reason for hiding this comment

cwperks Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

dancristiancecoi Jul 24, 2024

Choose a reason for hiding this comment

dblock commented Jul 24, 2024

terryquigleysas Jul 24, 2024

Choose a reason for hiding this comment

stephen-crawford Jul 24, 2024

Choose a reason for hiding this comment

peternied commented Jul 25, 2024

terryquigleysas commented Jul 25, 2024

github-actions bot commented Aug 7, 2024

beanuwave commented Aug 7, 2024

github-actions bot commented Aug 8, 2024

dancristiancecoi commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 8, 2024

github-actions bot commented Aug 9, 2024

codecov bot commented Aug 9, 2024

Codecov Report

github-actions bot commented Aug 30, 2024

github-actions bot commented Sep 23, 2024

cwperks Jul 24, 2024 •

edited

Loading