Bump version of Hadoop dependencies to 3.3.6 #6995

tlfeng · 2023-04-05T01:44:18Z

Description

Upgrade the version of Hadoop dependencies from 3.3.4 to 3.3.6

Changed:
in file plugins/repository-hdfs/build.gradle:
upgrade org.apache.hadoop:hadoop-client-api, org.apache.hadoop:hadoop-client-runtime, org.apache.hadoop:hadoop-hdfs to 3.3.6

in file test/fixtures/hdfs-fixture/build.gradle:
upgrade org.apache.hadoop:hadoop-minicluster to 3.3.6

Issues Resolved

Hadoop library introduces many transitive dependencies to OpenSearch, and using outdated dependencies is a red flag for some security scanners.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-04-05T01:51:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13466/
CommitID: 06b5371
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-06T05:42:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13631/
CommitID: 7760cc9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-06T06:03:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13634/
CommitID: 7760cc9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-06T20:56:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13662/
CommitID: 690447c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

tlfeng · 2023-04-06T21:04:15Z

failure occurs in the tests:

    org.opensearch.repositories.hdfs.HdfsBlobStoreContainerTests.testReadOnly
    org.opensearch.repositories.hdfs.HdfsBlobStoreRepositoryTests.testDeleteBlobs
    org.opensearch.repositories.hdfs.HdfsBlobStoreRepositoryTests.testContainerCreationAndDeletion
    org.opensearch.repositories.hdfs.HdfsBlobStoreRepositoryTests.testMultipleSnapshotAndRollback
    org.opensearch.repositories.hdfs.HdfsBlobStoreRepositoryTests.testList
    org.opensearch.repositories.hdfs.HdfsBlobStoreRepositoryTests.testSnapshotAndRestore
    org.opensearch.repositories.hdfs.HdfsRepositoryTests.testCreateSnapshot
    org.opensearch.repositories.hdfs.HdfsRepositoryTests.testCleanup
    org.opensearch.repositories.hdfs.HdfsRepositoryTests.testListChildren
    org.opensearch.repositories.hdfs.HdfsTests.testSimpleWorkflow

All the tests failed with the same reason

REPRODUCE WITH: ./gradlew ':plugins:repository-hdfs:test' --tests "org.opensearch.repositories.hdfs.HdfsBlobStoreContainerTests.testReadOnly" -Dtests.seed=36141267A4F65814 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=th -Dtests.timezone=Africa/Casablanca -Druntime.java=19

org.opensearch.repositories.hdfs.HdfsBlobStoreContainerTests > testReadOnly FAILED
    java.security.AccessControlException: access denied ("org.opensearch.secure_sm.ThreadPermission" "modifyArbitraryThreadGroup")
        at __randomizedtesting.SeedInfo.seed([36141267A4F65814:CFCFBD1E6670234E]:0)
        at java.****@19.0.2/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
        at java.****@19.0.2/java.security.AccessController.checkPermission(AccessController.java:1068)
        at java.****@19.0.2/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
        at app//org.opensearch.secure_sm.SecureSM.checkThreadGroupAccess(SecureSM.java:248)
...
        at app//org.opensearch.repositories.hdfs.HdfsBlobContainer.writeToPath(HdfsBlobContainer.java:178)
        at app//org.opensearch.repositories.hdfs.HdfsBlobContainer.lambda$writeBlobAtomic$5(HdfsBlobContainer.java:155)
        at app//org.opensearch.repositories.hdfs.HdfsBlobStore.lambda$execute$2(HdfsBlobStore.java:127)
        at java.****@19.0.2/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.****@19.0.2/java.security.AccessController.doPrivileged(AccessController.java:891)
        at app//org.opensearch.repositories.hdfs.HdfsSecurityContext.doPrivilegedOrThrow(HdfsSecurityContext.java:154)
        at app//org.opensearch.repositories.hdfs.HdfsBlobStore.execute(HdfsBlobStore.java:125)
        at app//org.opensearch.repositories.hdfs.HdfsBlobContainer.writeBlobAtomic(HdfsBlobContainer.java:154)
        at app//org.opensearch.repositories.blobstore.OpenSearchBlobStoreRepositoryIntegTestCase.writeBlob(OpenSearchBlobStoreRepositoryIntegTestCase.java:228)
        at app//org.opensearch.repositories.hdfs.HdfsBlobStoreContainerTests.testReadOnly(HdfsBlobStoreContainerTests.java:143)

There are some past issues or PRs mentioned the java exception when searching java.security.AccessControlException: access denied

owaiskazi19 · 2023-07-10T20:46:46Z

@reta looks like your were able to upgrade hadoop to 3.3.6? Will this change affect the downstream plugins?

github-actions · 2023-07-10T20:51:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/19735/
CommitID: bf91e6a

reta · 2023-07-10T21:18:21Z

@reta looks like your were able to upgrade hadoop to 3.3.6? Will this change affect the downstream plugins?

@owaiskazi19 Correct but there is a catch - we need to use custom ForkJoinPoolFactory, we probably need to think if that opens up any undesired flaws.

distribution/src/config/jvm.options

github-actions · 2023-07-11T13:34:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19868/
CommitID: 8ce6719
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-11T14:07:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19883/
CommitID: 8ce6719
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

distribution/src/config/jvm.options

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

codecov · 2023-07-25T16:54:36Z

Codecov Report

Merging #6995 (c297c7b) into main (5d78de6) will increase coverage by 0.16%.
The diff coverage is 80.55%.

@@             Coverage Diff              @@
##               main    #6995      +/-   ##
============================================
+ Coverage     70.98%   71.14%   +0.16%     
- Complexity    57166    57254      +88     
============================================
  Files          4757     4758       +1     
  Lines        269823   269841      +18     
  Branches      39474    39479       +5     
============================================
+ Hits         191536   191990     +454     
+ Misses        62158    61693     -465     
- Partials      16129    16158      +29

Files Changed	Coverage Δ
...a/org/opensearch/gradle/test/DistroTestPlugin.java	`0.00% <ø> (ø)`
...org/opensearch/index/seqno/ReplicationTracker.java	`68.36% <74.07%> (-0.91%)`	⬇️
.../secure_sm/SecuredForkJoinWorkerThreadFactory.java	`100.00% <100.00%> (ø)`
.../replication/checkpoint/ReplicationCheckpoint.java	`87.71% <100.00%> (+2.00%)`	⬆️

... and 454 files with indirect coverage changes

github-actions · 2023-07-25T16:58:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20909/
CommitID: 1944a8e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-25T17:36:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/20910/
CommitID: c297c7b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-07-25T18:33:51Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6995-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6786608a87d3e7fa306382a38e5702ae830f60e4
# Push it to GitHub
git push --set-upstream origin backport/backport-6995-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6995-to-2.x.

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 6786608)

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 6786608) Co-authored-by: Tianli Feng <[email protected]>

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

tlfeng added dependencies Pull requests that update a dependency file backport 2.x Backport to 2.x branch v2.7.0 labels Apr 5, 2023

tlfeng marked this pull request as ready for review April 6, 2023 05:36

tlfeng requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, VachaShah and xuezhou25 as code owners April 6, 2023 05:36

tlfeng marked this pull request as draft April 6, 2023 06:57

reta force-pushed the hadoop-3.3.5 branch from 3af1a77 to bf91e6a Compare July 10, 2023 20:14

reta force-pushed the hadoop-3.3.5 branch from bf91e6a to 8ce6719 Compare July 11, 2023 13:05

reta reviewed Jul 11, 2023

View reviewed changes

distribution/src/config/jvm.options Outdated Show resolved Hide resolved

andrross reviewed Jul 25, 2023

View reviewed changes

distribution/src/config/jvm.options Outdated Show resolved Hide resolved

reta force-pushed the hadoop-3.3.5 branch from 8ce6719 to 1944a8e Compare July 25, 2023 16:53

reta marked this pull request as ready for review July 25, 2023 16:53

reta requested review from dbwiddis, sachinpkale and sohami as code owners July 25, 2023 16:53

Bump version of Hadoop dependencies to 3.3.5

c297c7b

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

reta force-pushed the hadoop-3.3.5 branch from 1944a8e to c297c7b Compare July 25, 2023 16:54

andrross approved these changes Jul 25, 2023

View reviewed changes

dreamer-89 approved these changes Jul 25, 2023

View reviewed changes

andrross merged commit 6786608 into opensearch-project:main Jul 25, 2023
9 checks passed

reta mentioned this pull request Jul 25, 2023

[Backport] [2.x] Bump version of Hadoop dependencies to 3.3.6 (#6995) #8881

Merged

baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023

Bump version of Hadoop dependencies to 3.3.5 (opensearch-project#6995)

e9a6bbe

Signed-off-by: Tianli Feng <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump version of Hadoop dependencies to 3.3.6 #6995

Bump version of Hadoop dependencies to 3.3.6 #6995

tlfeng commented Apr 5, 2023 •

edited by reta

Loading

github-actions bot commented Apr 5, 2023

github-actions bot commented Apr 6, 2023

github-actions bot commented Apr 6, 2023

github-actions bot commented Apr 6, 2023

tlfeng commented Apr 6, 2023 •

edited

Loading

owaiskazi19 commented Jul 10, 2023 •

edited

Loading

github-actions bot commented Jul 10, 2023

reta commented Jul 10, 2023

github-actions bot commented Jul 11, 2023

github-actions bot commented Jul 11, 2023

codecov bot commented Jul 25, 2023 •

edited

Loading

github-actions bot commented Jul 25, 2023

github-actions bot commented Jul 25, 2023

opensearch-trigger-bot bot commented Jul 25, 2023

Bump version of Hadoop dependencies to 3.3.6 #6995

Bump version of Hadoop dependencies to 3.3.6 #6995

Conversation

tlfeng commented Apr 5, 2023 • edited by reta Loading

Description

Issues Resolved

Check List

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 6, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 6, 2023

Gradle Check (Jenkins) Run Completed with:

tlfeng commented Apr 6, 2023 • edited Loading

owaiskazi19 commented Jul 10, 2023 • edited Loading

github-actions bot commented Jul 10, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Jul 10, 2023

github-actions bot commented Jul 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 11, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jul 25, 2023 • edited Loading

Codecov Report

github-actions bot commented Jul 25, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 25, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jul 25, 2023

tlfeng commented Apr 5, 2023 •

edited by reta

Loading

tlfeng commented Apr 6, 2023 •

edited

Loading

owaiskazi19 commented Jul 10, 2023 •

edited

Loading

codecov bot commented Jul 25, 2023 •

edited

Loading