HDDS-11041. Add admin request filter for S3 requests and UGI support for GrpcOmTransport

[devabhishekpal]

What changes were proposed in this pull request?

HDDS-11041. Add admin request filter for S3 requests and UGI support for GrpcOmTransport

Please describe your PR in detail:

• Currently the s3secret endpoints do not support checking for the user making the request. This causes issues like HDDS-11040 where any user can modify the secrets
• As a part of this PR we are adding a new filter to check if the user making the request is a part of Ozone Admins, if not we will not trigger the generation/revocation of secret
• We are also adding the passing of UserGroupInformation to the GrpcOmTransport while creating the proxy connection to allow accessing the user ticket similar to the Hadoop3OmTransport

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11041

How was this patch tested?

Patch was tested using unit tests

[adoroszlai]

@ivanzlenko please take a look

[ivanzlenko]

General question: do we have any new tests for this?

[devabhishekpal]

Hi @ivanzlenko, new tests have not yet been added. We will need to modify existing tests to add user as an admin and generate request.
New test will be having a random user try to generate the request and get a forbidden response.

Waiting for go on code change before modifying the tests.

[ivanzlenko]

@devabhishekpal new filter could and should be covered with unit tests at least.

[devabhishekpal]

Yes, these new cases will be added to check for the filter working

[myskov]

Thanks @devabhishekpal for updating the patch.

[ivanzlenko]

Thanks @devabhishekpal for tiding up code!

[adoroszlai]

Thanks @devabhishekpal for working on this.

1. Please enable existing robot tests related to HDDS-11041, e.g.:

ozone/hadoop-ozone/dist/src/main/smoketest/s3/secretgenerate.robot

Line 48 in 8568075

[Tags] robot:skip # TODO: Enable after HDDS-11041 is done.

2. Please add a new test case similar to ... By Username For Other User, but with testuser2 (non-admin) logged in, and expecting request to be rejected.

[devabhishekpal]

Hi @ivanzlenko, @adoroszlai, @myskov, @vtutrinov could you take another look at the changes?
Thanks.

• Added new robot test to verify the filter
• Refactored the common config related utils to OzoneAdmins class
• Re-enabled the skipped robot tests added by @ivanzlenko initially as a part of HDDS-11040
• Added unit tests for the new utils

[adoroszlai]

Thanks @devabhishekpal for updating the patch. Mostly looks good.

I just realized that GrpcOmTransport is not used in any acceptance test environments. Can you please enable it in ozonesecure env. by the following change?

diff --git hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
index 39d26c362f..3fb7525f20 100644
--- hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
+++ hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
@@ -96,7 +96,7 @@ services:
       - 9878:9878
     env_file:
       - ./docker-config
-    command: ["/opt/hadoop/bin/ozone","s3g"]
+    command: ["/opt/hadoop/bin/ozone","s3g","-Dozone.om.transport.class=org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory"]
     environment:
       OZONE_OPTS:
   recon:

[devabhishekpal]

Thanks for the review @adoroszlai , addressed the test failures.

[adoroszlai]

Thanks @devabhishekpal for iterating on the patch.

[ivanzlenko]

@devabhishekpal from my point of view should be enough to have unit tests with mocks to verify that the contract for the filter will remain the same with any changes and extensively cover this functionality with integration tests. Unit tests should be simple and I don't think it will be a good to write something very comprehensive to mimic what could be done with integration tests.
P.S. sorry for waiting for a response from me.

[devabhishekpal]

Thanks for the input @ivanzlenko.
We re-enabled robot tests to verify this as well as new robot tests to verify the admin filter.
New unit tests were also added for the utils. It would be great if you could take a look