Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Circular dependency on job-scheduler-spi SNAPSHOT #1463

Closed
dblock opened this issue Jan 13, 2022 · 3 comments
Closed

[Bug]: Circular dependency on job-scheduler-spi SNAPSHOT #1463

dblock opened this issue Jan 13, 2022 · 3 comments
Labels
bug Something isn't working

Comments

@dblock
Copy link
Member

dblock commented Jan 13, 2022 •
edited
Loading

Describe the bug

There's a circular dependency on -SNAPSHOT builds being published to maven that require the distribution build to be successful.

To reproduce

  1. In order to build anomaly-detection job-scheduler-spi SNAPSHOT is required, e.g. [Backport to 1.2] fix missing dependency:commons-lang3 (#350) anomaly-detection#360 that is failing because job-scheduler-spi in maven is not up-to-date
  2. In order to publish job-scheduler-spi SNAPSHOT to Maven a successful distribution build is required
  3. The distribution build is failing because anomaly-detection cannot be built, because of (1)

Expected behavior

We shouldn't be able to get into this situation. Most likely this means we need to change our distribution build to publish snapshots after each component is built, and not at the end.

Additional context

To work around this problem we have to break the circular dependency temporarily:

  1. Remove the problematic component from the build, e.g. Remove AD from manifest to get job-scheduler SPI uploaded to maven. #1462
  2. Produce a successful distribution build without anomaly-detection
  3. Fix the issue in anomaly-detection, e.g. make sure [Backport to 1.2] fix missing dependency:commons-lang3 (#350) anomaly-detection#360 passes.
  4. Re-add anomaly-detection to the distribution, e.g. Re-add anomaly-detection. #1464
@dblock dblock added bug Something isn't working untriaged Issues that have not yet been triaged labels Jan 13, 2022
This was referenced Jan 13, 2022
Merged
Merged
Closed
Merged
@gaiksaya gaiksaya removed the untriaged Issues that have not yet been triaged label Jan 25, 2022
@amitgalitz amitgalitz mentioned this issue Apr 14, 2022
Closed
@prudhvigodithi
Copy link
Collaborator

This is closely related to this issue #1366
Right now in the manifest we first build the Core, job-scheduler and common-utils and then update with other components. This is a known dependency model, in terms of automation what we can do is, since we have a --component feature in now for build workflow, we can onboard all the components in the manifest and build in an order.
1st run would be --component OpenSearch job-scheduler common-utils.
@dblock @bbarani @peterzhuamazon

kaituo added a commit to kaituo/anomaly-detection-1 that referenced this issue Nov 4, 2022
@kaituo
remove bwc code related to 1.0 and earlier version
463bae5 
This PR addresses AD compile failure because the core removed 1x Version constants. (check opensearch-project/OpenSearch#5021) OpenSearch does not support N-2 version compatibility. This is inherited from Elasticsearch and Lucene. So version 1x is not compatible with 3.0. Thus removal of deprecated 1x code. This PR follows suite and removes backward compatibility code on OpenSearch 1.0 and older versions. So we won't support direct upgrade from 1.x domains to 3.x.

Testing done:
1. gradle build. Note that CI workflow will fail due to missing job scheduler. We are using a job scheduler from distribution. But due to a circular dependency on -SNAPSHOT builds being published to maven that require the distribution build to be successful (check opensearch-project/opensearch-build#1463), AD compilation failure caused the job scheduler to be taken out.  Not including the latest job scheduler will cause AD build to fail. So we have a chicken and egg problem: this PR fixes AD build failure and the PR itself cannot build due to missing job scheduler. To run gradle build, I changed to use local job scheduler and verified gradle build succeeded. Once I merge this PR. job scheduler should be able to build and be published to maven. Future AD CI should be unblocked after that.
@kaituo kaituo mentioned this issue Nov 4, 2022
Merged
kaituo added a commit to kaituo/anomaly-detection-1 that referenced this issue Nov 4, 2022
@kaituo
remove bwc code related to 1.0 and earlier version
1916c51 
This PR addresses AD compile failure because the core removed 1x Version constants. (check opensearch-project/OpenSearch#5021) OpenSearch does not support N-2 version compatibility. This is inherited from Elasticsearch and Lucene. So version 1x is not compatible with 3.0. Thus removal of deprecated 1x code. This PR follows suite and removes backward compatibility code on OpenSearch 1.0 and older versions. So we won't support direct upgrade from 1.x domains to 3.x.

Testing done:
1. gradle build. Note that CI workflow will fail due to missing job scheduler. We are using a job scheduler from distribution. But due to a circular dependency on -SNAPSHOT builds being published to maven that require the distribution build to be successful (check opensearch-project/opensearch-build#1463), AD compilation failure caused the job scheduler to be taken out.  Not including the latest job scheduler will cause AD build to fail. So we have a chicken and egg problem: this PR fixes AD build failure and the PR itself cannot build due to missing job scheduler. To run gradle build, I changed to use local job scheduler and verified gradle build succeeded. Once I merge this PR. job scheduler should be able to build and be published to maven. Future AD CI should be unblocked after that.

Signed-off-by: Kaituo Li <[email protected]>
kaituo added a commit to opensearch-project/anomaly-detection that referenced this issue Nov 8, 2022
@kaituo
remove bwc code related to 1.0 and earlier version (#714)
7cefb14 
This PR addresses AD compile failure because the core removed 1x Version constants. (check opensearch-project/OpenSearch#5021) OpenSearch does not support N-2 version compatibility. This is inherited from Elasticsearch and Lucene. So version 1x is not compatible with 3.0. Thus removal of deprecated 1x code. This PR follows suite and removes backward compatibility code on OpenSearch 1.0 and older versions. So we won't support direct upgrade from 1.x domains to 3.x.

Testing done:
1. gradle build. Note that CI workflow will fail due to missing job scheduler. We are using a job scheduler from distribution. But due to a circular dependency on -SNAPSHOT builds being published to maven that require the distribution build to be successful (check opensearch-project/opensearch-build#1463), AD compilation failure caused the job scheduler to be taken out.  Not including the latest job scheduler will cause AD build to fail. So we have a chicken and egg problem: this PR fixes AD build failure and the PR itself cannot build due to missing job scheduler. To run gradle build, I changed to use local job scheduler and verified gradle build succeeded. Once I merge this PR. job scheduler should be able to build and be published to maven. Future AD CI should be unblocked after that.

Signed-off-by: Kaituo Li <[email protected]>
@jordarlu
Copy link
Contributor

jordarlu commented Apr 4, 2023

Hi, team, just to quick check if this is still a valid issue/blocker, and if we can close it now ... thanks!!

@jordarlu
Copy link
Contributor

I am closing this issue for now .. pls re-open/create a new one if needed,,, thanks !!

hamersu9t added a commit to hamersu9t/anomaly-detection that referenced this issue Aug 10, 2024
@hamersu9t
remove bwc code related to 1.0 and earlier version (#714)
653477e 
This PR addresses AD compile failure because the core removed 1x Version constants. (check opensearch-project/OpenSearch#5021) OpenSearch does not support N-2 version compatibility. This is inherited from Elasticsearch and Lucene. So version 1x is not compatible with 3.0. Thus removal of deprecated 1x code. This PR follows suite and removes backward compatibility code on OpenSearch 1.0 and older versions. So we won't support direct upgrade from 1.x domains to 3.x.

Testing done:
1. gradle build. Note that CI workflow will fail due to missing job scheduler. We are using a job scheduler from distribution. But due to a circular dependency on -SNAPSHOT builds being published to maven that require the distribution build to be successful (check opensearch-project/opensearch-build#1463), AD compilation failure caused the job scheduler to be taken out.  Not including the latest job scheduler will cause AD build to fail. So we have a chicken and egg problem: this PR fixes AD build failure and the PR itself cannot build due to missing job scheduler. To run gradle build, I changed to use local job scheduler and verified gradle build succeeded. Once I merge this PR. job scheduler should be able to build and be published to maven. Future AD CI should be unblocked after that.

Signed-off-by: Kaituo Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dblock @prudhvigodithi @gaiksaya @jordarlu