s2a: Add gRPC S2A

[rmehta19]

Add S2A Java client to gRPC Java.

Context: https://github.com/google/s2a-go/blob/main/README.md

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: rmehta19 / name: Riya Mehta (739ee23, b35f145, 46691df, 2dd1c7e, 72630d8, 217a3e4, 44fe552, f94cc10, 08f8342, f47c560, 1d41d10, 42cd3e8, f96d395, 12586b1, 35084af, 7f26712, d1f413b, 655f0bd, da330cd, 50b7366, 0e059e4, a8cacb0, 3198eec, 3184cdc, 38b0a3a, b021a21, 752627a, 19583a7)

[matthewstevenson88]

Thanks @rmehta19. Can you PTAL at the test failures?

[rmehta19]

@matthewstevenson88, I made 2 changes and it looks like 2/3 linux runs are passing. tests(11) is failing with an error in code that was not affected by this PR, so I don't think that failure is related.

The changes:

• The build originally failed because the generated grpc code in this PR did not match the code generated by the codegen plugin. I couldn't easily compile the codegen myself(due to reasons explained in: protoc-gen-grpc-java not available on apple m1 #7690). So instead I used the latest published binary available on https://mvnrepository.com/artifact/io.grpc/protoc-gen-grpc-java (1.63) to generate S2AServiceGrpc.java (Created and used a separate simple maven project just to run the plugin and get grpc gen code). Then I changed line 8 & 9 of S2AServiceGrpc.java to match other generated code in this repo (ex:

  grpc-java/alts/src/generated/main/grpc/io/grpc/alts/internal/HandshakerServiceGrpc.java

  Lines 8 to 9 in fb9a108

  	value = "by gRPC proto compiler",
  	comments = "Source: grpc/gcp/handshaker.proto")

  ). This resolved the errors thrown by

  grpc-java/.github/workflows/testing.yml

  Lines 65 to 66 in 3c867c9

  	- name: Check for modified codegen
  	run: test -z "$(git status --porcelain)" || (git status && echo Error Working directory is not clean. Forget to commit generated files? && false)

• A failure in the tests was caused when IntegrationTest.java is in package io.grpc.s2a: GetAuthenticationMechanismsTest.java fails because the flag is not being set by JCommander (or perhaps it is getting overwritten). When I moved IntegrationTest.java into the same package as GetAuthenticationMechanismsTest.java: io.grpc.s2a.handshaker, this error does not happen.

[matthewstevenson88]

LGTM, thanks @rmehta19! When you have a chance, please remove "draft" to the PR title and can we say a bit more in the CL description including pointing to the S2A-Go README (similar to what we did in the proto PR)?

After that, can we get sign-off from @erm-g and @ejona86?

[rmehta19]

LGTM, thanks @rmehta19! When you have a chance, please remove "draft" to the PR title and can we say a bit more in the CL description including pointing to the S2A-Go README (similar to what we did in the proto PR)?

After that, can we get sign-off from @erm-g and @ejona86?

Done -- thank you for the review @matthewstevenson88!

[ejona86]

To make sure I understand, s2a is the public API, and s2a/channel and s2a/handshaker are internal APIs. Are those internal APIs to be used anywhere, even within Google?

[rmehta19]

Thanks for the review @larry-safran ! Please let me know if there is anything else to address.

[ejona86]

Some random things I noticed.

[ejona86]

More random stuff. This is mostly from just skimming randomly and noticing unexpected code shapes.

[ejona86]

What is happening here? How would this ever work?

[rmehta19]

This is so that the generated certs have localhost IP addr, so that certificate verification passes in tests

[ejona86]

Isn't localhost ::1?

[rmehta19]

This is true. I ran a local test modifying the IP to be ::1 and regenerating these certs, confirming our tests still pass. I'll modify this in a separate PR along with the other comments on the EC cert data: #11540 (comment) .

I'll also update our go client tests for consistency: https://github.com/google/s2a-go/blob/main/testdata/config.cnf

[rmehta19]

Done in 7a879a2

[rmehta19]

Sorry for the confusion and thanks for pointing out. To be clear, out tests rely on the fact that Common Name is set to localhost (not actually in this config, but when this config is used to create the CSR, it is set. I noted this in the README). So that's why this field was left as ::, and had no effect on tests when set to ::1.

[ejona86]

Then let me repeat my initial question: What is happening here? What is it doing?

[rmehta19]

We use the config to generate certs to be used for client & S2A when testing mTLS to S2A. This is used in an integration test (also in S2AChannelCredentialsTest unit test, but that's just a unit test, no channel actually gets established between the client and S2A). As part of mTLS peer cert verification process, it is required that certificate CommonName matches the hostname (which is localhost).

[ejona86]

I don't understand. This is alt_names aka SAN, which is not CommonName. And :: is not ::1 and that would not validate if the client used "localhost". So that all seems irrelevant.

[ejona86]

(As I said before, as a hint for integration tests, don't put localhost in the cert, which is actually invalid. Instead, use managedChannelBuilder.overrideAuthority(). It is available in all languages; WithAuthority() in Go.)

[ejona86]

This Stub is never closed. This causes IntegrationTest (after Larry's reordering of awaitTermination) to not complete tests quickly, as the RPC is outstanding so awaitTermination() hangs. I think once we shut down the top-level channel that releases the S2AHandshakerServiceChannel.ChannelResource after 1 second, which then calls shutdownNow() on the s2a channel which cancels the RPC.

[rmehta19]

Yes, we never call close on the stub directly. When the
close gets invoked, everything you mentioned happens.

[ejona86]

That's not okay. That means we leak an RPC per connection for the life of all s2a channels, which is probably the lifetime of the process.

[rmehta19]

Would closing the stub in close resolve the issue?

[ejona86]

It resolves "never" but that is still busted. The stub needs to be closed when it is no longer needed. That could be when the future completes (e.g., sslContextFuture.addListener(s2aStub::close, service)). But I don't see a reason to split up the lifetimes between the threads like this. It seems clearer to:

@Override
protected void handlerAdded0(ChannelHandlerContext ctx) {
  // Buffer all reads until the TLS Handler is added.
  BufferReadsHandler bufferReads = new BufferReadsHandler();
  ctx.pipeline().addBefore(ctx.name(), /* name= */ null, bufferReads);
  ListenableFuture<SslContext> sslContextFuture = service.submit(this::createSslContext);
  ...
}

private SslContext createSslContext() {
  Channel ch = channelPool.getChannel();
  try (S2AStub s2aStub = S2AStub.newInstance(S2AServiceGrpc.newStub(ch))) {
    return SslContextFactory.createForClient(s2aStub, hostname, localIdentity);
  } finally {
    channelPool.returnToPool(ch);
  }
}

[rmehta19]

The channel is still needed after the SslContext future is complete (we make RPCs to S2A to verify the peer, and perform private key operations during the handshake). The SslContextBuilder configures the trustManager and PrivateKeyMethod with the stub. In this example, the channel gets returned to the pool before the handshake is complete, which causes an error when S2ATrustManager uses the stub to verify the peer cert.

The stub is no longer needed when the handshake is complete. A probably not great option would be to pass around the channel instead, and create a new stub for every RPC? Or perhaps we plumb the stub down to the ClientTlsProtocolNegotiator and close it when the handshake is done?

[rmehta19]

@ejona86, WDYT?

[ejona86]

I see. A new stub for every RPC is fine. I thought y'all needed to use the same streaming RPC to preserve context. If separate RPCs are fine, that is definitely superior.

We don't want to inject into the handshake completion code itself, but that code is triggering a ProtocolNegotiationEvent. We can use that.

private static final class S2AStubCleanupNegotiationHandler extends ProtocolNegotiationHandler {
  private final s2aStub; // pass in constructor
  @Override
  protected void protocolNegotiationEventTriggered(ChannelHandlerContext ctx) {
    s2aStub.close();
    fireProtocolNegotiationEvent(ctx);
    ctx.pipeline().remove(ctx);
  }
}

And then in S2aProtocolNegotiationHandler, ctx.pipeline().addAfter() that S2AStubCleanupNegotiationHandler before adding the tls handler. (So the TLS handler will be between that cleanup handler and the s2a handler.)

We could do alternatives like adding the tls handler before the s2a handler, but they take a bit more reorganization.

[rmehta19]

Thanks Eric! I've implemented what you mention in #11600

I initially did think that we could do a new stub for each RPC, since we attach the localIdentity to each RPC, but chatted with @matthewstevenson88 and he mentioned there is a requirement from S2A that the lifetime of the bidirectional stream from the client to the S2A must be tied to the lifetime of the handshake.

[rmehta19]

Thanks for the comments @ejona86; I'll address them in separate commits at #11534. Let me know if you prefer separate PRs for them, or any other way of organizing.

[ejona86]

There's plenty of smaller/isolated items. Feel free to group those together as you wish, as they are easy to review. A few of the comments might turn into larger/plumbing changes. Some of those may be best to be separate. But overall, if you do lots of the "easy" stuff together that'd be fine to review/merge while you work on something more difficult.

[rmehta19]

@ejona86 , @larry-safran I have addressed all the comments on this PR in the following 3 PRs:

• s2a: Address comments on PR#11113 #11534
• s2a,netty: S2AHandshakerServiceChannel doesn't use custom event loop. #11539
• s2a: Address minor comments on PR#11113 #11540

PTAL at these 3 and leave any comments.

I will address the comment to move things to internal package and combine the MTLS and non-MTLS apis in the following 2 PRs, once the above 3 are merged.

• s2a: Move s2a implementation to internal package. #11541
• s2a: Combine MtlsToS2ChannelCredentials and S2AChannelCredentials. #11544

Besides these changes, please let me know anything else we can do to enable s2a to be included in a grpc-java release. Thank you!

[ejona86]

It resolves "never" but that is still busted. The stub needs to be closed when it is no longer needed. That could be when the future completes (e.g., sslContextFuture.addListener(s2aStub::close, service)). But I don't see a reason to split up the lifetimes between the threads like this. It seems clearer to:

@Override
protected void handlerAdded0(ChannelHandlerContext ctx) {
  // Buffer all reads until the TLS Handler is added.
  BufferReadsHandler bufferReads = new BufferReadsHandler();
  ctx.pipeline().addBefore(ctx.name(), /* name= */ null, bufferReads);
  ListenableFuture<SslContext> sslContextFuture = service.submit(this::createSslContext);
  ...
}

private SslContext createSslContext() {
  Channel ch = channelPool.getChannel();
  try (S2AStub s2aStub = S2AStub.newInstance(S2AServiceGrpc.newStub(ch))) {
    return SslContextFactory.createForClient(s2aStub, hostname, localIdentity);
  } finally {
    channelPool.returnToPool(ch);
  }
}