[FEATURE] Support OpenSearch Serverless request signing

[pingleig]

Is your feature request related to a problem?

Current builtin AwsSdk2Transport does not work with serverless https://aws.amazon.com/opensearch-service/features/serverless/ because:

• It is only in 2.x version while severless is still using 1.x
• Different signing requirement https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-signing
  • sign uses es while serverless requires aoss
  • signContent-Length header while serverless does not use it
  • uses Aws4Signer while serverless uses Aws4UnsignedPayloadSigner

External library does not work due to Content-Length

• AwsRequestSigningApacheInterceptor.java
  https://github.com/acm19/aws-request-signing-apache-interceptor/blob/031791a8f11c3ecb2d687137ba36cc805442687a/src/main/java/io/github/acm19/aws/interceptor/http/AwsRequestSigningApacheInterceptor.java#L134

The official guide is using AWS SDK v1 https://github.com/awsdocs/amazon-opensearch-service-developer-guide/blob/master/sample_code/AmazonOpenSearchJavaClient-main/src/main/java/AmazonOpenSearchServiceSample.java

What solution would you like?

• Document what customer can use when they want to interact with OpenSearch serverless using Java client. The full interceptor code is actually not long, e.g. my modified version
• Allow configure service name, signer and Content-Length behavior in the AwsSdk2Transport (but I am not sure if we can valid it ...

What alternatives have you considered?

• Update https://github.com/acm19/aws-request-signing-apache-interceptor with an option to always skip content length
• Backport the sign transport to v1 client

Do you have any additional context?

I have a demo repo https://github.com/pingleig/opensearch-unofficial-demo

[dblock]

Thanks for reporting this @pingleig!

• I opened Support OpenSearch Serverless request signing acm19/aws-request-signing-apache-interceptor#88
• A custom interceptor should/can be added to the AWS docs, just like Updated Code samples to AWS SDK 2.x awsdocs/amazon-opensearch-service-developer-guide#63, Add a sample for OpenSearch Serverless request signing awsdocs/amazon-opensearch-service-developer-guide#79
• We need to add whatever support is necessary to opensearch-java for Serverless (this issue)

[dblock]

Beyond request signing, APIs such as info() don't exist/work. We need a plan to support serverless similarly to how we support multiple versions of OpenSearch, except that now we have to deal with multiple target services and multiple semvers.

[wbeckler]

Is it possible that someone could make it unnecessary to specify a service name and that the client would try aoss and es and it would find one that works?

[dblock]

Is it possible that someone could make it unnecessary to specify a service name and that the client would try aoss and es and it would find one that works?

I would say no.

Signatures contain the service name. So unless the service renames aoss to es I expect the client will continue to need to use one or the other.

You can recognize a service by a URL or by connecting to it and looking at headers.

• I've seen code that examines the URL, e.g. ....us-west-2.aoss.amazonaws.com the service is aoss. However that only works when directly referring to the instance - you can very well have an instance behind API gateway, proxy, or, very likely, an internal DNS name like search.mycompany.com.
• The protocol doesn't require a back-and-forth like Negotiate. I've seen code (e.g. ruby client) that makes a request to the server to figure out what it is. I think it's a bad idea because without authentication this is a 401/403, which creates noise in the number of errors counted on the server, and is an unnecessary roundtrip for the client.

[VachaShah]

@pingleig Support for OpenSearch Serverless is now available in opensearch-java 2.2.0.