[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent #2122

peterzhuamazon · 2022-09-29T15:56:53Z

Hi @opensearch-project/security we recently see this error in 2.3.0 builds for windows.

How to reproduce:

Download: https://ci.opensearch.org/ci/dbc/bundle-build-zhujiaxi/2.3.0-SNAPSHOT/12/windows/x64/zip/dist/opensearch/opensearch-2.3.0-SNAPSHOT-windows-x64.zip
Use the bin/opensearch.bat file to start on Windows machine, cluster exit with the following errors.
Remove security plugin, cluster start normally.


[2022-09-29T00:04:37,124][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-S5RJAB6] Config directory is C:\Users\Administrator\testknn\test\config/, from there the key- and truststore files are resolved relatively
[2022-09-29T00:04:37,187][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [EC2AMAZ-S5RJAB6] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:420) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:347) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:420) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:347) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        ... 6 more
Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:176) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:262) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:420) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.node.Node.<init>(Node.java:347) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        ... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255)
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:176)
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:262)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
        at org.opensearch.node.Node.<init>(Node.java:420)
        at org.opensearch.node.Node.<init>(Node.java:347)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
For complete error details, refer to the log at C:\Users\Administrator\testknn\test\logs\opensearch.log

Thanks.

The text was updated successfully, but these errors were encountered:

cwperks · 2022-09-29T16:58:13Z

@peterzhuamazon Just want to double check did you perform steps 6 and 7 from the linked Meta issue?

6. Run bash plugins/opensearch-security/tools/install_demo_configuration.sh and follow prompts to create demo certificate.
7. mkdir logs (https://github.com/opensearch-project/OpenSearch/issues/1463)

peterzhuamazon · 2022-09-29T21:48:21Z

@peterzhuamazon Just want to double check did you perform steps 6 and 7 from the linked Meta issue?
6. Run bash plugins/opensearch-security/tools/install_demo_configuration.sh and follow prompts to create demo certificate.
7. mkdir logs (https://github.com/opensearch-project/OpenSearch/issues/1463)

Now that reminds me in ODFE we have a script to run the demo configurations script.
Seems like that step is missed in OpenSearch.
Thanks @cwperks let me add some code and test again.

peternied · 2022-10-04T20:54:26Z

@peterzhuamazon I was able to spin up the OpenSearch process on my local machine any connect to it via https with the admin account. Setup note; I did have to execute the demo cert script, which worked fine with WSL installed.

peterzhuamazon · 2022-10-04T23:05:35Z

Before we verify this change in Jenkins Please do not close the issue.

Thanks.

peternied · 2022-10-06T22:25:27Z

@peterzhuamazon is there a way we can help verify and close out the issue? The repro steps from this issue do not seem to embody the scenario that isn't working.

peterzhuamazon · 2022-10-06T23:01:10Z

Temp solution: opensearch-project/opensearch-build#2704.
Follow up: #2148

Thanks.

stephen-crawford · 2022-10-17T19:38:18Z

[CLOSE] Closing since 2.3.0 is shipped and new changes intended for future release.

haghverdimasoud · 2022-12-28T12:01:21Z

I added OpenSearch to moqui-framwork and i have a same issue. how can i fix it? I don't want to use OpenSearch directly in windows.

NeelamMahato · 2023-11-08T13:29:31Z

I have got the same error while running the batch file(.\opensearch-windows-install.bat) from command prompt in Opensearch and 9200 port is not responding .Please suggest any solution for Windows.

peterzhuamazon added bug Something isn't working windows labels Sep 29, 2022

This was referenced Sep 29, 2022

[WIN M1] WIN (ZIP/MSI) Build/Assemble Process opensearch-project/opensearch-build#2306

Closed

[Meta] OpenSearch / OpenSearch Dashboards - Windows distribution (X64) opensearch-project/opensearch-build#33

Closed

peterzhuamazon changed the title ~~[2.3.0] Security not able to load during OpenSearch Startup in Windows~~ [BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Sep 29, 2022

peterzhuamazon changed the title ~~[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows~~ [BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent Sep 29, 2022

davidlago added the v2.4.0 'Issues and PRs related to version v2.4.0' label Sep 30, 2022

davidlago assigned cwperks Sep 30, 2022

peternied closed this as completed Oct 4, 2022

peterzhuamazon reopened this Oct 4, 2022

This was referenced Oct 6, 2022

Opensearch security plugin windows install certs demo config and add batch install script opensearch-project/opensearch-build#2704

Merged

[BUG] Security Plugin does not have batch script for install demo configurations for windows #2148

Closed

davidlago added triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. sprint backlog labels Oct 10, 2022

stephen-crawford closed this as completed Oct 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent #2122

[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent #2122

peterzhuamazon commented Sep 29, 2022 •

edited

Loading

cwperks commented Sep 29, 2022

peterzhuamazon commented Sep 29, 2022

peternied commented Oct 4, 2022

peterzhuamazon commented Oct 4, 2022

peternied commented Oct 6, 2022

peterzhuamazon commented Oct 6, 2022 •

edited

Loading

stephen-crawford commented Oct 17, 2022

haghverdimasoud commented Dec 28, 2022

NeelamMahato commented Nov 8, 2023 •

edited

Loading

[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent #2122

[BUG] 2.3.0 Security not able to load during OpenSearch Startup in Windows Agent #2122

Comments

peterzhuamazon commented Sep 29, 2022 • edited Loading

cwperks commented Sep 29, 2022

peterzhuamazon commented Sep 29, 2022

peternied commented Oct 4, 2022

peterzhuamazon commented Oct 4, 2022

peternied commented Oct 6, 2022

peterzhuamazon commented Oct 6, 2022 • edited Loading

stephen-crawford commented Oct 17, 2022

haghverdimasoud commented Dec 28, 2022

NeelamMahato commented Nov 8, 2023 • edited Loading

peterzhuamazon commented Sep 29, 2022 •

edited

Loading

peterzhuamazon commented Oct 6, 2022 •

edited

Loading

NeelamMahato commented Nov 8, 2023 •

edited

Loading