Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add explicit dependency to validatePom and generatePom tasks. #12807

Merged
merged 3 commits into from
Mar 22, 2024

Conversation

zelinh
Copy link
Member

@zelinh zelinh commented Mar 20, 2024

Description

This PR is to add explicit task dependency on PomGenerate and PomValidate tasks for different publication types.

For plugin with different publication, all generatePom tasks such as generatePomFileForNebulaPublication or generatePomFileForPluginZipPublication are creating the Pom file at the same cache location: build/distributions/*******.pom; and Pom validation tasks are reading the output of this Pom.

We will get error like this in job-scheduler plugin:

> Task :generatePomFileForPluginZipPublication FAILED

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':generatePomFileForPluginZipPublication' (type 'GenerateMavenPom').
  - Gradle detected a problem with the following location: '/Users/zelinhao/workplace/job-scheduler/build/distributions/opensearch-job-scheduler-2.12.0.0-SNAPSHOT.pom'.
    
    Reason: Task ':validateNebulaPom' uses this output of task ':generatePomFileForPluginZipPublication' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
    
    Possible solutions:
      1. Declare task ':generatePomFileForPluginZipPublication' as an input of ':validateNebulaPom'.
      2. Declare an explicit dependency on ':generatePomFileForPluginZipPublication' from ':validateNebulaPom' using Task#dependsOn.
      3. Declare an explicit dependency on ':generatePomFileForPluginZipPublication' from ':validateNebulaPom' using Task#mustRunAfter.
    
    For more information, please refer to https://docs.gradle.org/8.5/userguide/validation_problems.html#implicit_dependency in the Gradle documentation.

Reason
Gradle treats any task that reads a file as implicitly dependent on each task that writes to that file.
In our case, Gradle is assuming each validation task is depending on all the Pom generation tasks as implicit dependency.

In this PR, we are adding these implicit dependencies for generatePomFile and validatePom into its implementation.

We can see same logic is added in few other plugin repos:
In geospatial: validateNebulaPom.dependsOn(generatePomFileForPluginZipPublication)
In skills: tasks.validateNebulaPom.dependsOn ':generatePomFileForPluginZipPublication'

Check List

- [x] New functionality includes testing.
~~ - [x] All tests pass~~
- [x] New functionality has been documented.
~~ - [x] New functionality has javadoc added~~
- [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)

  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Zelin Hao <[email protected]>
Copy link
Contributor

github-actions bot commented Mar 20, 2024

Compatibility status:

Checks if related components are compatible with change 0729327

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

Copy link
Contributor

✅ Gradle check result for feea594: SUCCESS

Copy link

codecov bot commented Mar 21, 2024

Codecov Report

Attention: Patch coverage is 26.66667% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 71.36%. Comparing base (b15cb0c) to head (0729327).
Report is 73 commits behind head on main.

Files Patch % Lines
...gradle/precommit/PomValidationPrecommitPlugin.java 36.36% 7 Missing ⚠️
.../java/org/opensearch/gradle/pluginzip/Publish.java 0.00% 3 Missing ⚠️
...opensearch/gradle/precommit/PomValidationTask.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12807      +/-   ##
============================================
- Coverage     71.42%   71.36%   -0.06%     
- Complexity    59978    60209     +231     
============================================
  Files          4985     5011      +26     
  Lines        282275   283667    +1392     
  Branches      40946    41117     +171     
============================================
+ Hits         201603   202434     +831     
- Misses        63999    64397     +398     
- Partials      16673    16836     +163     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❕ Gradle check result for 0729327: UNSTABLE

  • TEST FAILURES:
      3 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@reta
Copy link
Collaborator

reta commented Mar 22, 2024

Thanks @zelinh , I think it looks good!

@reta reta merged commit c04dad5 into opensearch-project:main Mar 22, 2024
30 of 31 checks passed
@reta reta added the backport 2.x Backport to 2.x branch label Mar 22, 2024
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12807-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c04dad58bbc941b8d5f18326d5c5cfb7ac312239
# Push it to GitHub
git push --set-upstream origin backport/backport-12807-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12807-to-2.x.

@reta
Copy link
Collaborator

reta commented Mar 22, 2024

@zelinh could you please backport to 2.x manually? thank you

@cwperks
Copy link
Member

cwperks commented Mar 22, 2024

@reta @zelinh I'm not sure yet, but I think this PR is contributing to failures being seen in security plugin's CI: opensearch-project/security#4156

* Where:
Build file '/home/runner/work/security/security/build.gradle' line: 417

* What went wrong:
A problem occurred evaluating root project 'opensearch-security'.
> NamedDomainObjectContainer#configureEach(Action) on Publication container cannot be executed in the current context.

@reta
Copy link
Collaborator

reta commented Mar 22, 2024

Ah, that is unexpected, thanks a lot for @cwperks !

reta added a commit to reta/OpenSearch that referenced this pull request Mar 22, 2024
@@ -48,17 +48,24 @@ public class PomValidationPrecommitPlugin extends PrecommitPlugin {
public TaskProvider<? extends Task> createTask(Project project) {
TaskProvider<Task> validatePom = project.getTasks().register("validatePom");
PublishingExtension publishing = project.getExtensions().getByType(PublishingExtension.class);
publishing.getPublications().all(publication -> {
publishing.getPublications().configureEach(publication -> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line seemed to cause a breaking change in the security plugin:

* Where:
Build file '/home/runner/work/security/security/build.gradle' line: 417

* What went wrong:
A problem occurred evaluating root project 'opensearch-security'.
> NamedDomainObjectContainer#configureEach(Action) on Publication container cannot be executed in the current context.

Is this change from all -> configureEach necessary?

peternied added a commit to peternied/security that referenced this pull request Mar 27, 2024
The underlying issue associated with Pom validation was fixed in OpenSearch [1], this causes the 'hack' that was used to unblock pom generation in security to start causing problems in the distribution builds.

- [1] opensearch-project/OpenSearch#12807

Signed-off-by: Peter Nied <[email protected]>
peternied added a commit to peternied/security that referenced this pull request Mar 27, 2024
The underlying issue associated with Pom validation was fixed in OpenSearch [1], this causes the 'hack' that was used to unblock pom generation in security to start causing problems in the distribution builds.

- [1] opensearch-project/OpenSearch#12807

Signed-off-by: Peter Nied <[email protected]>
cwperks pushed a commit to opensearch-project/security that referenced this pull request Mar 27, 2024
### Description
The underlying issue associated with Pom validation was fixed in
OpenSearch [1], this causes the 'hack' that was used to unblock pom
generation in security to start causing problems in the distribution
builds.

- [1] opensearch-project/OpenSearch#12807

### Issues Resolved
- Fixes #4160

### Testing
Added new CI test that executes the same job used by the distribution
build and checks the expected artifacts exist

### Check List
- [X] New functionality includes testing
- [ ] ~New functionality has been documented~
- [X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Peter Nied <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…arch-project#12807)

* Add explicit dependency to PomValidation

Signed-off-by: Zelin Hao <[email protected]>

* Update CHANGELOG

Signed-off-by: Zelin Hao <[email protected]>

---------

Signed-off-by: Zelin Hao <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
dlin2028 pushed a commit to dlin2028/security that referenced this pull request May 1, 2024
### Description
The underlying issue associated with Pom validation was fixed in
OpenSearch [1], this causes the 'hack' that was used to unblock pom
generation in security to start causing problems in the distribution
builds.

- [1] opensearch-project/OpenSearch#12807

### Issues Resolved
- Fixes opensearch-project#4160

### Testing
Added new CI test that executes the same job used by the distribution
build and checks the expected artifacts exist

### Check List
- [X] New functionality includes testing
- [ ] ~New functionality has been documented~
- [X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Peter Nied <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants