Download more dependencies to decrease the chance of network errors #15797
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MiguelWeezardo
changed the title
mvn dependency:go-offlinedependency:go-offline
MiguelWeezardo
changed the title
dependency:go-offline--offline flag to Maven calls after dependency:go-offline
|
Does this also solve fetching Maven plugins? Or is that not an issue? |
A hypothesis I'm currently checking is that Maven tries to download POMs while building the module hierarchy. |
findepi
reviewed
Jan 20, 2023
|
It seems that |
|
I guess we could manually download one dependency, but I see that there may be more missing pieces. |
|
We can make iterative improvements here. |
We'd never catch up - the end of the road is having all dependencies (transitive and direct) from all build plugins downloaded, and unless we manage to do that there's always a window of opportunity for CI breaking. We need something like |
|
I'll make another attempt with common CI profiles enabled during |
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
from
a6cdfc4
to
75937fe
Compare
|
I'm also going to check out https://github.com/qaware/go-offline-maven-plugin |
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
3 times, most recently
from
0f54eb4
to
e699b36
Compare
|
It seems the new plugin worked (at least up until |
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
3 times, most recently
from
e9d020c
to
52e8dd4
Compare
MiguelWeezardo
changed the title
--offline flag to Maven calls after dependency:go-offline
|
Once this gets merged we should purge the GitHub Maven cache. |
MiguelWeezardo
added
the
dx
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
3 times, most recently
from
b166d6f
to
39344d6
Compare
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
from
39344d6
to
37345b4
Compare
findepi
approved these changes
Feb 7, 2023
|
Failed due to #14372 |
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
2 times, most recently
from
d043387
to
269a4f2
Compare
Maven's built-in `dependency:go-offline` goal doesn't download build plugin dependencies, we need an additional tool to download them. The `maven-checks` job is responsible for: - getting as many dependencies as possible, - putting them in GitHub cache for other jobs and pipelines. Experiments have shown the new plugin also has some issues. For example it only downloads a single version of a dependency, which breaks on any module which overrides global versions of some common dependencies - so far `trino-pinot` is the only one. These overridden versions will be missing. This is one of the reasons why we can't use Maven's `--offline` flag. However, a majority of dependencies can still be downloaded at the start of the pipeline, providing quick feedback in case of network problems. Anything missing will have to be downloaded in other jobs. This won't completely eliminate the faulty networking problems, but should decrease the chances of dependency downloads failing in late pipeline stages. Add dynamic dependencies found by CI
Use cache to decrease the number of downloads from Maven Central which can potentially result in failures.
Revapi tries to compare this and previous version of any API. If it can't download the old version, if falls back to comparing with an empty file. This can lead to error reports which are not shown in daily development.
In this approach each job is responsible for downloading all dependencies.
Retries should only be attempted during dependency downloads.
This explains how to handle dynamic dependencies that aren't detected by automated tools.
MiguelWeezardo
force-pushed
the
maven_offline_flag
branch
from
269a4f2
to
8e54b75
Compare
findepi
approved these changes
Feb 8, 2023
|
Unrelated test failure: |
|
thanks! |
MiguelWeezardo
added
the
no-release-notes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
A call to
dependency:go-offlinewas recently added to download all dependencies into the local Maven cache.Unfortunately it turns out the goal has some problems with multi-module projects, and Maven will still attempt to download any missing artifacts during build.
The purpose of this PR is to improve this situation by caching MORE dependencies, as every cached artifact decreases the chance of a Maven Central download failure breaking Maven builds.
Downloading ALL dependencies is NOT a requirement, although it would be ideal.
Other jobs will now depend the
maven-checksjob as they now need to unpack the cache artifact left by it.Additional context and related issues
CI workers sometimes fail to download dependencies in the middle of a build.
A workaround was proposed to download as many dependencies as possible at the start of CI pipeline, cache them for other jobs, to limit the use of Maven Central afterwards.
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: