[Extensions] Add extensions/extensions.yml setting to enable backward compatible plugin mode for extensions

[cwperks]

A new setting in opensearch.yml extensions/extensions.yml needs to be added to enable backward compatible plugin mode for extensions. The setting can be:

extensions:
  - name: hello-world
    uniqueId: hw
    hostAddress: '127.0.0.1'
    port: '4532'
    version: '1.0'
    opensearchVersion: '3.0.0'
    minimumCompatibleVersion: '3.0.0'
    bwcPluginMode: true

Background

Many plugins currently rely on user information to be present on the ThreadContext to support plugin use-cases. Examples include:

1. resource protection - i.e. sharing detectors in Anomaly Detection via backend roles)
2. asynchronous operation -i.e. running a detector on a schedule and injecting the user's roles from the time of detector creation to evaluate if a user has privileges on an index

See: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java#L198-L210

private void setUserInfoInThreadContext(User user, Set<String> mappedRoles) {
    if (threadContext.getTransient(OPENDISTRO_SECURITY_USER_INFO_THREAD_CONTEXT) == null) {
        StringJoiner joiner = new StringJoiner("|");
        joiner.add(user.getName());
        joiner.add(String.join(",", user.getRoles()));
        joiner.add(String.join(",", Sets.union(user.getSecurityRoles(), mappedRoles)));
        String requestedTenant = user.getRequestedTenant();
        if (!Strings.isNullOrEmpty(requestedTenant)) {
            joiner.add(requestedTenant);
        }
        threadContext.putTransient(OPENDISTRO_SECURITY_USER_INFO_THREAD_CONTEXT, joiner.toString());
    }
}

In order for extensions to initially be a replacement for plugins and support all current use-cases, they will need the data from the above snippet. (Note: specifically the information needed is the username, backend_roles and mapped_roles - mapped roles here refers to the ultimate roles that have been resolved to after all roles mapping - 1. roles explicitly assigned + 2. roles mapped to via backend_roles 3. roles matching the hosts in roles_mapping and 4. And_backend_roles - the user needs all backend roles in the list to be mapped. Full Details: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/securityconf/ConfigModelV7.java#L1206-L1269)

This setting will ultimately determine if the encryption_key from #2615 is used to encrypt the roles and backend_roles claims in the payload of the token sent to an extension.

See design considerations

• [Question] How to support extensions parity for user information #2536

[cwperks]

@peternied Made a good point that this should be configured on the extension level and not globally for all extensions. A good place for this to be is in extensions/extensions.yml. See example in the SDK Dev guide: https://github.com/opensearch-project/opensearch-sdk-java/blob/main/DEVELOPER_GUIDE.md#create-extensionsyml-file

[stephen-crawford]

[Triage] This is part of the Extensions project.

[samuelcostae]

@scrawfor99 commenting so you can assign it to me

[cwperks]

@samuelcostae I assigned the issue to you. FYI there was a change in the initialization process for extensions which removed the usage of extensions.yml and instead uses a REST Request for initialization. Look at the description here for an example of REST Request initialization: cwperks#5

These lines may also be helpful here which shows how the Security plugin can define settings which should be added to the initialization request: https://github.com/cwperks/security/pull/5/files#diff-e7ef66295cba81a49e4349781a2e9678d2b021a570e978bb4feb422b03d5d74aR1133-R1139