BIGTOP-3835: Add Apache Knox package

[cheelio]

Description of PR

Apache Knox provides a gateway for interacting with the REST APIs and UIs of Apache Hadoop deployments.
It's a crucial component in secure environments and therefore I thinks it should be part of the Bigtop stack.
I can understand that it's maybe too late to add this to the into Bigtop 3.2 BOM. If so, maybe we can add this to Bigtop 3.3?

How was this patch tested?

Build with:

#CentOS
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-7 bash -l -c './gradlew knox-clean knox-pkg yum'
#Ubuntu
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-ubuntu-16.04 bash -l -c './gradlew knox-clean knox-pkg repo'

I created basic smoke tests:

cd provisioner/docker
# CentOS:
./docker-hadoop.sh --destroy
 ./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox --smoke-tests knox

# Ubuntu:
./docker-hadoop.sh --destroy
./docker-hadoop.sh --enable-local-repo --create 1 -C config_ubuntu-20.04.yaml --stack knox --smoke-tests knox

Further, the application can be tested by launching a basic LDAP server:

cd provisioner/docker
./docker-hadoop.sh --destroy
./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox
./docker-hadoop.sh --exec 1 /usr/lib/knox/bin/ldap.sh start

Now, you can login by using the username/password combination admin:admin-password.
By default, the application runs on port 8443:

https://localhost:8443/gateway/homepage/home/
https://localhost:8443/gateway/manager/admin-ui/

[cheelio]

I just realized that I have to make a few more modifications on this PR regarding to the paths.

[cheelio]

I just tested Knox basic service integration:

#start container with knox and livy service:
(cd provisioner/docker && ./docker-hadoop.sh --destroy && ./docker-hadoop.sh --enable-local-repo --create 1 -C config_centos-7.yaml --stack knox,livy)

#start integrated (test) LDAP server on container:
(cd provisioner/docker; ./docker-hadoop.sh --exec 1 /usr/lib/knox/bin/ldap.sh start)

#Get IP of container:
export DOCKER_IP=$(cd provisioner/docker; ./docker-hadoop.sh --exec 1 hostname -I)

#Test Knox API Request:
curl -ku admin:admin-password -XGET https://${DOCKER_IP}:8443/gateway/admin/api/v1/topologies/sandbox

#Test Livy with knox:
curl -ku admin:admin-password -XGET https://${DOCKER_IP}:8443/gateway/sandbox/livy/sessions

[guyuqi]

@cheelio
Thank you for working on it. It's awesome.
For we've defined new comming 3.2 release in https://issues.apache.org/jira/browse/BIGTOP-3695,
this PR would be merged in the future 3.3 release after we test it. Thanks again for your contribution.

[JiaLiangC]

@guyuqi is there a plan that we add knox to bigtop 3.3?

[guyuqi]

@guyuqi is there a plan that we add knox to bigtop 3.3?

Hi @JiaLiangC,
Masatake Iwasaki is working on 3.2.1 patch release to upgrade Hadoop to 3.3.5 to fix some critical issues.
For Bigtop-v3.3, I have no objections to adding Knox to into Bigtop from my side.

But we‘d better listen to the voices of other people in the community.
Could you kindly please send the DISCUSSION mail on our Bigtop mailing list?
OR please also add the comments on the relevant Jira.
Let's give others the opportunity to chime in with their opinions.

Thanks.

[guyuqi]

Could you guys kindly rebase the PR to resolve the conficts?

[guyuqi]

@JiaLiangC @cheelio
I just rebased the PR to resolve the conficts and the PR has provided the smoke tests.
Let me test it on some Distros and x86/Arm64 platform.

[JiaLiangC]

@guyuqi
Since cheelio is no longer active, is there anything I can do to help you move forward with this pull request? Do you need assistance with testing or any other tasks?

[kevinw66]

Thanks @cheelio , the work is now moving to #1247