SamlAuthenticatorTests fail on zulu8 runtime #49742

jkakavas · 2019-12-02T06:18:10Z

These started failing only on zulu8 after Novemember 30th : https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-60d,mode:quick,to:now))&_a=(columns:!(branch),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:SamlAuthenticatorTests),sort:!(time,desc))

It would make sense if #49512 caused this somehow but this fails in 6.8, 7.4, 7.5 7.x and not on master while the opensaml dependencies were updated in master and 7.x only.

The stack trace is

java.lang.NoClassDefFoundError: Could not initialize class org.jcp.xml.dsig.internal.dom.DOMEnvelopedTransform
	at __randomizedtesting.SeedInfo.seed([13A666290A5F1991:2BB4D77CB77ADE28]:0)
	at org.jcp.xml.dsig.internal.dom.XMLDSigRI$ProviderService.newInstance(XMLDSigRI.java:111)
	at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
	at javax.xml.crypto.dsig.TransformService.getInstance(TransformService.java:215)
	at org.jcp.xml.dsig.internal.dom.DOMXMLSignatureFactory.newTransform(DOMXMLSignatureFactory.java:314)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signElement(SamlAuthenticatorTests.java:2358)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signDoc(SamlAuthenticatorTests.java:2264)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signDoc(SamlAuthenticatorTests.java:2251)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.testIncorrectSubjectConfirmationDataInResponseToIsRejected(SamlAuthenticatorTests.java:1287)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-12-02T06:18:12Z

Pinging @elastic/es-security (:Security/Authentication)

jkakavas · 2019-12-02T08:12:28Z

Underlying exception is

Caused by:
        java.security.AccessControlException: access denied ("java.security.SecurityPermission" "com.sun.org.apache.xml.internal.security.register")
            at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
            at java.security.AccessController.checkPermission(AccessController.java:886)
            at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
            at com.sun.org.apache.xml.internal.security.utils.JavaUtils.checkRegisterPermission(JavaUtils.java:222)
            at com.sun.org.apache.xml.internal.security.utils.XMLUtils.setDsPrefix(XMLUtils.java:103)
            at com.sun.org.apache.xml.internal.security.utils.ElementProxy.setNamespacePrefix(ElementProxy.java:499)
            at com.sun.org.apache.xml.internal.security.utils.ElementProxy.registerDefaultPrefixes(ElementProxy.java:512)
            at com.sun.org.apache.xml.internal.security.Init.dynamicInit(Init.java:118)
            at com.sun.org.apache.xml.internal.security.Init.init(Init.java:94)
            at org.jcp.xml.dsig.internal.dom.ApacheTransform.<clinit>(ApacheTransform.java:56)
            ... 8 more

possibly related to the latest Azul Zulu update in https://github.com/elastic/infra/pull/16252/files

benwtrent · 2019-12-10T14:30:45Z

./gradlew ':x-pack:plugin:security:unitTest' -Dtests.seed=B2DB8D45606A2857 -Dtests.class=org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests -Dtests.method="testAssetionWithoutBearerSubjectConfirmationMethodIsRejected" -Dtests.security.manager=true -Dtests.locale=es-PA -Dtests.timezone=Africa/Maseru -Dcompiler.java=12 -Druntime.java=8

New build failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/378/console

trace:

08:28:13 ERROR   0.02s J4  | SamlAuthenticatorTests.testExpiredSubjectConfirmationDataIsRejected <<< FAILURES!
08:28:13    > Throwable #1: java.lang.NoClassDefFoundError: Could not initialize class org.jcp.xml.dsig.internal.dom.DOMEnvelopedTransform
08:28:13    > 	at __randomizedtesting.SeedInfo.seed([B2DB8D45606A2857:FB9658AEAD7098F9]:0)
08:28:13    > 	at org.jcp.xml.dsig.internal.dom.XMLDSigRI$ProviderService.newInstance(XMLDSigRI.java:111)
08:28:13    > 	at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
08:28:13    > 	at javax.xml.crypto.dsig.TransformService.getInstance(TransformService.java:215)
08:28:13    > 	at org.jcp.xml.dsig.internal.dom.DOMXMLSignatureFactory.newTransform(DOMXMLSignatureFactory.java:314)
08:28:13    > 	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signElement(SamlAuthenticatorTests.java:1888)
08:28:13    > 	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signDoc(SamlAuthenticatorTests.java:1794)
08:28:13    > 	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signDoc(SamlAuthenticatorTests.java:1781)
08:28:13    > 	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.testExpiredSubjectConfirmationDataIsRejected(SamlAuthenticatorTests.java:1012)
08:28:13    > 	at java.lang.Thread.run(Thread.java:748)
08:28:13   1> [2019-12-10T15:28:12,118][INFO ][o.e.x.s.a.s.SamlAuthenticatorTests] [testAssetionWithoutBearerSubjectConfirmationMethodIsRejected] before test
08:28:13   1> [2019-12-10T15:28:12,125][INFO ][o.e.x.s.a.s.SamlAuthenticatorTests] [testAssetionWithoutBearerSubjectConfirmationMethodIsRejected] after test

costin · 2019-12-11T12:14:37Z

More failures here:
https://gradle-enterprise.elastic.co/s/ehawjietnkici
https://gradle-enterprise.elastic.co/s/qgihyqngvatsc
6.8.6: https://gradle-enterprise.elastic.co/s/6u6gdrd3orlly

martijnvg · 2019-12-18T10:37:19Z

Other instances of this failure:

albertzaharovits · 2019-12-18T17:19:32Z

I looked into it a bit, I can repro with zulu8 on MacOS.
What I believe is going on is that in zulu8 a different XML signature provider is picked up compared to the other VMs. Usually it should be the one from xmlsec-2.1.4.jar, but zulu8 picks one from javax.xml.crypto . This is odd, because I believe that packet is added in a module in java 9 but it's somehow available in zulu8 ?
@tvernum Maybe you have an idea about what's actually going on here?

albertzaharovits · 2019-12-18T19:49:41Z

Digging further, looks like zulu8 has a different provider for XML signatures; other VMs use the provider from the xmlsec-2.1.4.jar (somehow without asking for special permissions). But zulu8 requires the java security "java.security.SecurityPermission" "com.sun.org.apache.xml.internal.security.register" to instantiate its provider, disregarding xmlsec-2.1.4.jar - its own provider is based on Apache Santuario anyway.

Thing is, I think this is not only a problem in the test, it must be a problem with the main code running on zulu8 (it shouldn't be able to load the provider).

tvernum · 2019-12-18T21:05:54Z

I'll try and find some cycles to get my head around the details & options but I agree that the likely situation is that right now zulu-latest doesn't work with ES-SAML.

I'm going to ping @elastic/es-core-infra here because if it is the case that the latest Zulu release isn't compatible with some ES features then we need to think about what that means (Zulu is official supported for ES). Fixing it in ES is a good option (assuming we can) but we might consider raising it with Azul as well.

davidkyle · 2019-12-19T11:33:06Z

More occurrences:

https://gradle-enterprise.elastic.co/s/6qyoiuxqvjzes
https://gradle-enterprise.elastic.co/s/zqkvzbffbaacg

ebadyano · 2020-01-02T15:34:25Z

another one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/424/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/157/console

ebadyano · 2020-01-02T23:00:36Z

Seems similar but not on zulu8: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/157/console

Actually it looks slightly different so might not be related.

cbuescher · 2020-01-03T10:03:47Z

This looks similar to me too, on 7.5 branch:

Log	https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/158/console
Build Scans	[6.8.7] https://gradle-enterprise.elastic.co/s/ypgxnqoofbr5o [6.8.7] https://gradle-enterprise.elastic.co/s/6hhr54afwetnshttps://gradle-enterprise.elastic.co/s/dndprlv75g26q

cbuescher · 2020-01-03T15:13:19Z

And there are similar looking failures on 6.8 as well i think. Looks like its using ES_RUNTIME_JAVA=zulu8 as well.

FAILURE elastic/elasticsearch#6.8 fcb62f2 matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose - 20200103114858-C621E61F

java.lang.NoClassDefFoundError: Could not initialize class org.jcp.xml.dsig.internal.dom.DOMEnvelopedTransform
	at __randomizedtesting.SeedInfo.seed([343C023B414D657F:263C06ACA002A7E8]:0)
	at org.jcp.xml.dsig.internal.dom.XMLDSigRI$ProviderService.newInstance(XMLDSigRI.java:111)
	at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
	at javax.xml.crypto.dsig.TransformService.getInstance(TransformService.java:215)
	at org.jcp.xml.dsig.internal.dom.DOMXMLSignatureFactory.newTransform(DOMXMLSignatureFactory.java:314)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signElement(SamlAuthenticatorTests.java:1888)
	at org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.signElement(SamlAuthenticatorTests.java:1848)
[...]

Log	https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/426/console
Build Scans	https://gradle-enterprise.elastic.co/s/c2dijmeatmnx4

davidkyle · 2020-01-06T11:10:32Z

Another failure

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/164/console

pgomulka · 2020-01-07T10:23:21Z

there is another failure on 7.5 and zulu.
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu8,nodes=general-purpose/166/console

https://gradle-enterprise.elastic.co/s/sp2pwionxkl4w/tests/g5u3xltc4r2du-f6oqpzngs7vbw

REPRODUCE WITH: ./gradlew ':x-pack:plugin:security:test' --tests "org.elasticsearch.xpack.security.authc.saml.SamlAuthenticatorTests.testIncorrectDestinationIsRejected" -Dtests.seed=5F02BE6FC9DEAEC2 -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=MET -Dcompiler.java=12 -Druntime.java=8

jkakavas · 2020-01-07T12:03:46Z

This is a test issue and doesn't affect our SAML functionality in Azul Zulu JVM.

This fails in SamlAuthenticatorTests#signElement were we are using javax.xml.crypto.dsig.Transform directly while "manually" creating signatures of SAML messages ( as opposed to using the opensaml API ) as it was deemed more appropriate when we first wrote the tests. In contrast, SamlRedirectTests where we are creating signed Authentication request messages using OpenSAML ( org.opensaml.xmlsec.crypto.XMLSigningUtil ) as our code does, this is not an issue.

As a matter of fact, SamlAuthenticatorTests#signElement is also the reason ( slightly different root cause regarding the lack of XMLSignatureFactory in BouncyCastle ) why we have SamlAuthenticator tests muted in FIPS 140.

We should resolve this long standing TODO and fix both problems at teh same time. I'll assign this to me.

pgomulka · 2020-01-07T15:38:37Z

another failure but it is on 6.8, build with openjdk12 and run with zulu8
https://gradle-enterprise.elastic.co/s/bzgtd6uyt5abm

dnhatn · 2020-01-08T17:33:09Z

Another on 7.x: https://gradle-enterprise.elastic.co/s/zviozoqxyvcao

tvernum · 2020-01-09T00:40:48Z

I think we should mute this test on Zulu until we fix it.
I would do that, but I don't have a Zulu VM installed right now so I can't test it.

A simple

assumeFalse("https://github.com/elastic/elasticsearch/issues/49742",
   System.getProperty("java.vendor", "unknown").contains("azul"));

should do it (but we'd need to check exactly how Zulu identifies itself in the system properties, including case sensitivity, etc).

See elastic#49742

jkakavas · 2020-01-09T05:48:32Z

Started working on the necessary changes yesterday but didn't get to complete them, I'll wrap it up today but raised #50779 to mute the failures in the meantime either way.

See #49742

jkakavas added >test-failure Triaged test failures from CI :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Dec 2, 2019

albertzaharovits self-assigned this Jan 2, 2020

jkakavas assigned jkakavas and unassigned albertzaharovits Jan 7, 2020

jkakavas added a commit to jkakavas/elasticsearch that referenced this issue Jan 9, 2020

Mute SamlAuthenticatorTests in Azulu Zulu

c5c3ebe

See elastic#49742

jkakavas mentioned this issue Jan 9, 2020

Mute SamlAuthenticatorTests in Azul Zulu #50779

Merged

jkakavas added a commit that referenced this issue Jan 9, 2020

Mute SamlAuthenticatorTests in Azulu Zulu (#50779)

d2189b9

See #49742

jkakavas added a commit that referenced this issue Jan 9, 2020

Mute SamlAuthenticatorTests in Azulu Zulu (#50779)

2471dab

See #49742

jkakavas added a commit that referenced this issue Jan 9, 2020

Mute SamlAuthenticatorTests in Azulu Zulu (#50779)

8349bd4

See #49742

jkakavas mentioned this issue Jan 16, 2020

Refactor SamlAuthenticatorTests #51089

Merged

rjernst added the Team:Security Meta label for security team label May 4, 2020

jkakavas closed this as completed in #51089 May 25, 2020

tvernum mentioned this issue Jul 28, 2020

KerberosAuthenticationIT authentication test failures #56507

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SamlAuthenticatorTests fail on zulu8 runtime #49742

SamlAuthenticatorTests fail on zulu8 runtime #49742

jkakavas commented Dec 2, 2019

elasticmachine commented Dec 2, 2019

jkakavas commented Dec 2, 2019

benwtrent commented Dec 10, 2019

costin commented Dec 11, 2019

martijnvg commented Dec 18, 2019 •

edited

Loading

albertzaharovits commented Dec 18, 2019

albertzaharovits commented Dec 18, 2019

tvernum commented Dec 18, 2019

davidkyle commented Dec 19, 2019

ebadyano commented Jan 2, 2020 •

edited

Loading

ebadyano commented Jan 2, 2020 •

edited

Loading

cbuescher commented Jan 3, 2020

cbuescher commented Jan 3, 2020

davidkyle commented Jan 6, 2020

pgomulka commented Jan 7, 2020

jkakavas commented Jan 7, 2020

pgomulka commented Jan 7, 2020

dnhatn commented Jan 8, 2020

tvernum commented Jan 9, 2020

jkakavas commented Jan 9, 2020

SamlAuthenticatorTests fail on zulu8 runtime #49742

SamlAuthenticatorTests fail on zulu8 runtime #49742

Comments

jkakavas commented Dec 2, 2019

elasticmachine commented Dec 2, 2019

jkakavas commented Dec 2, 2019

benwtrent commented Dec 10, 2019

costin commented Dec 11, 2019

martijnvg commented Dec 18, 2019 • edited Loading

albertzaharovits commented Dec 18, 2019

albertzaharovits commented Dec 18, 2019

tvernum commented Dec 18, 2019

davidkyle commented Dec 19, 2019

ebadyano commented Jan 2, 2020 • edited Loading

ebadyano commented Jan 2, 2020 • edited Loading

cbuescher commented Jan 3, 2020

cbuescher commented Jan 3, 2020

davidkyle commented Jan 6, 2020

pgomulka commented Jan 7, 2020

jkakavas commented Jan 7, 2020

pgomulka commented Jan 7, 2020

dnhatn commented Jan 8, 2020

tvernum commented Jan 9, 2020

jkakavas commented Jan 9, 2020

martijnvg commented Dec 18, 2019 •

edited

Loading

ebadyano commented Jan 2, 2020 •

edited

Loading

ebadyano commented Jan 2, 2020 •

edited

Loading