[Infrastructure] Intermittent tests failures blocking pull requests

[peternied]

Blocked pull request: https://github.com/opensearch-project/security/runs/7735891837?check_suite_focus=true
Full logs from associated failed leg: https://github.com/opensearch-project/security/suites/7723457626/artifacts/323573420

Failed tests, TenantInfoActionTest. testTenantInfoAPIUpdate was the blocking test

DlsTest. testDlsWithMinDocCountZeroAggregations
TenantInfoActionTest. testTenantInfoAPIUpdate
TenantInfoActionTest. testTenantInfoAPIUpdate
TenantInfoActionTest. testTenantInfoAPIUpdate
TenantInfoActionTest. testTenantInfoAPIUpdate
TenantInfoActionTest. testTenantInfoAPIUpdate
TenantInfoActionTest. testTenantInfoAPIUpdate
LegacyTenantInfoActionTests. testTenantInfoAPIUpdate

Commonly logged failure message:

[FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:22,997][ERROR][org.opensearch.security.configuration.ConfigurationLoaderSecurity7] Failure [node_utest_n75_fnull_t797968674720_num2][127.0.0.1:8855] Node not connected retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2022-08-08T23:11:32,910][ERROR][org.opensearch.security.rest.TenantInfoAction] OpenSearchException[java.util.concurrent.TimeoutException: Timeout after 10SECONDS while retrieving configuration for [ROLESMAPPING](index=.opendistro_security)]; nested: TimeoutException[Timeout after 10SECONDS while retrieving configuration for [ROLESMAPPING](index=.opendistro_security)];

I think we need more debugging information to determine what is causing these tests to get stuck, any other thoughts @cwperks

[DarshitChanpura]

Additional intermittent test failures found in https://github.com/opensearch-project/security/actions/runs/4019330649/jobs/6943359537 and its re-run attempts

• org.opensearch.security.auditlog.compliance.RestApiComplianceAuditlogTest.testAutoInit
• org.opensearch.security.dlic.rest.api.AuditApiActionTest.testApi
• org.opensearch.security.dlic.dlsfls.DlsTest.testDlsAggregations

Affected PR: #2418

Errors repeating in logs:

[ERROR][org.opensearch.security.test.helper.file.FileHelper] Failed to load auditlog/nodes_dn.yml
[ERROR][org.opensearch.security.test.helper.file.FileHelper] Failed to load auditlog/whitelist.yml
[ERROR][org.opensearch.security.test.helper.file.FileHelper] Failed to load auditlog/allowlist.yml

[ERROR][org.opensearch.security.ssl.transport.SecuritySSLNettyTransport] Exception during establishing a SSL connection: java.io.IOException: Connection reset by peer

[davidlago]

Any updates/new occurrences on this one?

[peternied]

Lets close this out, we can reopen if the failures show back up