Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky ci check-commit when compiling individual commits due to network errors #15793

Closed
findepi opened this issue Jan 20, 2023 · 11 comments
Closed

Flaky ci check-commit when compiling individual commits due to network errors #15793

findepi opened this issue Jan 20, 2023 · 11 comments
Labels
bug Something isn't working dx Issues or pull requests related to developer experience test

Comments

@findepi
Copy link
Member

findepi commented Jan 20, 2023

#15710 attempted to address the flakiness, but seems it was insufficient

see eg https://github.com/trinodb/trino/actions/runs/3960144105/jobs/6785253382

2023-01-19T17:52:01.1740939Z + ./mvnw dependency:go-offline
...
2023-01-19T17:52:38.7824614Z [INFO] ------------------------------------------------------------------------
2023-01-19T17:52:38.7824715Z [INFO] BUILD SUCCESS
2023-01-19T17:52:38.7824973Z [INFO] ------------------------------------------------------------------------
2023-01-19T17:52:38.7825074Z [INFO] Total time:  35.249 s
2023-01-19T17:52:38.7825265Z [INFO] Finished at: 2023-01-19T17:52:38Z
2023-01-19T17:52:38.7825524Z [INFO] ------------------------------------------------------------------------
2023-01-19T17:52:38.7937200Z + ./mvnw package -B --strict-checksums --quiet -T C1 -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all=true -Dmaven.javadoc.skip=true --no-snapshot-updates --no-transfer-progress -pl '!:trino-server-rpm' -Dair.check.skip-all=false -Dair.check.skip-basic=true -Dair.check.skip-extended=true -Dair.check.skip-checkstyle=false -P gib -Dgib.referenceBranch=refs/remotes/origin/master
2023-01-19T17:58:11.1779624Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (package) on project trino-thrift-testing-server: Execution package of goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single failed: Plugin org.apache.maven.plugins:maven-assembly-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-assembly-plugin:jar:3.3.0 -> io.airlift:packaging:jar:222: Failed to read artifact descriptor for io.airlift:packaging:jar:222: Could not transfer artifact io.airlift:packaging:pom:222 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/io/airlift/packaging/222/packaging-222.pom: Read timed out -> [Help 1]
2023-01-19T17:58:11.1801290Z [ERROR] 
2023-01-19T17:58:11.1810581Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2023-01-19T17:58:11.1821956Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2023-01-19T17:58:11.1824514Z [ERROR] 
2023-01-19T17:58:11.1831662Z [ERROR] For more information about the errors and possible solutions, please read the following articles:
2023-01-19T17:58:11.1837754Z [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
2023-01-19T17:58:11.1847944Z [ERROR] 
2023-01-19T17:58:11.1854660Z [ERROR] After correcting the problems, you can resume the build with the command
2023-01-19T17:58:11.1862863Z [ERROR]   mvn <args> -rf :trino-thrift-testing-server

cc @nineinchnick @MiguelWeezardo
@findepi findepi added bug Something isn't working test dx Issues or pull requests related to developer experience labels Jan 20, 2023
This was referenced Jan 20, 2023
Merged
Merged
@findepi
Copy link
Member Author

findepi commented Jan 20, 2023

https://github.com/trinodb/trino/actions/runs/3966038737/jobs/6796555367

2023-01-20T09:20:11.9008820Z [INFO] ------------------------------------------------------------------------
2023-01-20T09:20:11.9008915Z [INFO] BUILD SUCCESS
2023-01-20T09:20:11.9009172Z [INFO] ------------------------------------------------------------------------
2023-01-20T09:20:11.9009278Z [INFO] Total time:  37.999 s
2023-01-20T09:20:11.9009454Z [INFO] Finished at: 2023-01-20T09:20:11Z
2023-01-20T09:20:11.9009687Z [INFO] ------------------------------------------------------------------------
2023-01-20T09:20:11.9234903Z + ./mvnw package -B --strict-checksums --quiet -T C1 -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all=true -Dmaven.javadoc.skip=true --no-snapshot-updates --no-transfer-progress -pl '!:trino-server-rpm' -Dair.check.skip-all=false -Dair.check.skip-basic=true -Dair.check.skip-extended=true -Dair.check.skip-checkstyle=false -P gib -Dgib.referenceBranch=refs/remotes/origin/master
2023-01-20T09:24:40.7398209Z Validating /home/runner/work/trino/trino/docs/src/main/sphinx/language/reserved.rst
2023-01-20T09:24:41.3237367Z Validated 81 reserved identifiers
2023-01-20T09:24:41.5004004Z + cd /home/runner/work/trino/trino/docs
2023-01-20T09:24:41.5036552Z + test -t 1
2023-01-20T09:24:41.5046635Z + OPTS=
2023-01-20T09:24:41.5055919Z + SPHINX_IMAGE=ghcr.io/trinodb/build/sphinx:5
2023-01-20T09:24:41.5056379Z ++ id -u
2023-01-20T09:24:41.5145621Z ++ id -g
2023-01-20T09:24:41.5172128Z + docker run --rm -e TRINO_VERSION -u 1001:123 -v /home/runner/work/trino/trino/docs:/docs ghcr.io/trinodb/build/sphinx:5 sphinx-build -q -j auto -b html -W -d target/doctrees src/main/sphinx target/html
2023-01-20T09:24:41.6858231Z Unable to find image 'ghcr.io/trinodb/build/sphinx:5' locally
2023-01-20T09:24:42.3099185Z 5: Pulling from trinodb/build/sphinx
2023-01-20T09:24:42.9068153Z 42c077c10790: Pulling fs layer
...
2023-01-20T09:24:58.3478640Z Digest: sha256:f6281b2cc26d278688f15235a7aa54b66eecf7a23d4ade762bdc1c2412bd8369
2023-01-20T09:24:58.3492748Z Status: Downloaded newer image for ghcr.io/trinodb/build/sphinx:5
2023-01-20T09:26:11.7748209Z Generating sitemap for 511 pages in /docs/target/html/sitemap.xml
2023-01-20T09:26:13.0595532Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (package) on project trino-thrift-testing-server: Execution package of goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single failed: Plugin org.apache.maven.plugins:maven-assembly-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-assembly-plugin:jar:3.3.0 -> io.airlift:packaging:jar:222: Failed to read artifact descriptor for io.airlift:packaging:jar:222: Could not transfer artifact io.airlift:packaging:pom:222 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/io/airlift/packaging/222/packaging-222.pom: Read timed out -> [Help 1]
2023-01-20T09:26:13.0599796Z [ERROR] 
2023-01-20T09:26:13.0626910Z [ERROR] To see the full stack tr

@findepi
Copy link
Member Author

findepi commented Jan 20, 2023

Different problem https://github.com/trinodb/trino/actions/runs/3966038737/jobs/6796555536

2023-01-20T09:25:55.2455213Z a05f6e87ff7d: Pull complete
2023-01-20T09:25:56.2008727Z 0ac1ecec52f8: Pull complete
2023-01-20T09:25:56.2051854Z Digest: sha256:f6281b2cc26d278688f15235a7aa54b66eecf7a23d4ade762bdc1c2412bd8369
2023-01-20T09:25:56.2066238Z Status: Downloaded newer image for ghcr.io/trinodb/build/sphinx:5
2023-01-20T09:26:29.3658505Z ##[error]The operation was canceled.
2023-01-20T09:26:29.4380751Z Post job cleanup.
2023-01-20T09:26:29.4630627Z Post job cleanup.
2023-01-20T09:26:29.8088384Z Post job cleanup.
2023-01-20T09:26:30.8354988Z [command]/usr/bin/git version

@findepi findepi changed the title Flaky ci when compiling individual commits Flaky ci when compiling individual commits (network errors, cancellation) Jan 20, 2023
This was referenced Jan 20, 2023
Closed
Closed
@nineinchnick
Copy link
Member

In the last one the job got cancelled. We can't do much about it, it was either cancelled manually or there's something wrong with GitHub Actions. There was an outage yesterday but it's green today: https://www.githubstatus.com/

@findepi
Copy link
Member Author

findepi commented Jan 20, 2023

thanks @nineinchnick for looking into this.

Only the issue description's link is from yesterday (and it didn't get cancellation), so fortunately all these cases seem unrelated to yesterday GH outage.

@MiguelWeezardo
Copy link
Member

We need to add an --offline option to all Maven calls after mvn dependency:go-offline, otherwise Maven will still try to fetch POMs.

@MiguelWeezardo
Copy link
Member

MiguelWeezardo commented Jan 20, 2023 •
edited
Loading

See #15797 for a hypothesis how to approach the "go-offline succeeded, Maven still failed while trying to download dependencies" scenario.

@findepi findepi mentioned this issue Jan 24, 2023
Merged
@findepi
Copy link
Member Author

findepi commented Jan 24, 2023

Flaky ci when compiling individual commits (network errors, cancellation)

Cancellation part is trivial -- this is simply how we configured the matrix to run.
Covered by #15824

@findepi findepi changed the title Flaky ci when compiling individual commits (network errors, cancellation) Flaky ci when compiling individual commits due to network errors Jan 24, 2023
@findepi
Copy link
Member Author

findepi commented Jan 24, 2023

https://github.com/trinodb/trino/actions/runs/3994838475/jobs/6853654995

2023-01-24T10:49:34.2933152Z 0ac1ecec52f8: Pull complete
2023-01-24T10:49:34.2977170Z Digest: sha256:f6281b2cc26d278688f15235a7aa54b66eecf7a23d4ade762bdc1c2412bd8369
2023-01-24T10:49:34.2998878Z Status: Downloaded newer image for ghcr.io/trinodb/build/sphinx:5
2023-01-24T10:50:48.8094534Z Generating sitemap for 511 pages in /docs/target/html/sitemap.xml
2023-01-24T10:50:50.1024746Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (package) on project trino-thrift-testing-server: Execution package of goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single failed: Plugin org.apache.maven.plugins:maven-assembly-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-assembly-plugin:jar:3.3.0 -> io.airlift:packaging:jar:222: Failed to read artifact descriptor for io.airlift:packaging:jar:222: Could not transfer artifact io.airlift:packaging:pom:222 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/io/airlift/packaging/222/packaging-222.pom: Read timed out -> [Help 1]
2023-01-24T10:50:50.1033127Z [ERROR] 
2023-01-24T10:50:50.1041649Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2023-01-24T10:50:50.1050483Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.
2023-01-24T10:50:50.1059600Z [ERROR] 
2023-01-24T10:50:50.1065059Z [ERROR] For more information about the errors and possible solutions, please read the following articles:
2023-01-24T10:50:50.1073314Z [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
2023-01-24T10:50:50.1077958Z [ERROR] 
2023-01-24T10:50:50.1084821Z [ERROR] After correcting the problems, you can resume the build with the command
2023-01-24T10:50:50.1089039Z [ERROR]   mvn <args> -rf :trino-thrift-testing-server

@findepi findepi mentioned this issue Jan 24, 2023
Merged
@findepi
Copy link
Member Author

findepi commented Jan 26, 2023 

2023-01-26T13:23:50.8921236Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (package) on project trino-thrift-testing-server: Execution package of goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single failed: Plugin org.apache.maven.plugins:maven-assembly-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-assembly-plugin:jar:3.3.0 -> io.airlift:packaging:jar:222: Failed to read artifact descriptor for io.airlift:packaging:jar:222: Could not transfer artifact io.airlift:packaging:pom:222 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/io/airlift/packaging/222/packaging-222.pom: Read timed out -> [Help 1]

3 times of 6 attempts within single build (https://github.com/trinodb/trino/actions/runs/4014078961/jobs/6895456517)

@MiguelWeezardo @nineinchnick @electrum any thoughts how I can solve this?

@findepi
Copy link
Member Author

findepi commented Jan 27, 2023

https://github.com/trinodb/trino/actions/runs/4018647063/jobs/6907005915

2023-01-27T00:58:50.5865889Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single (package) on project trino-thrift-testing-server: Execution package of goal org.apache.maven.plugins:maven-assembly-plugin:3.3.0:single failed: Plugin org.apache.maven.plugins:maven-assembly-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-assembly-plugin:jar:3.3.0 -> io.airlift:packaging:jar:222: Failed to read artifact descriptor for io.airlift:packaging:jar:222: Could not transfer artifact io.airlift:packaging:pom:222 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/io/airlift/packaging/222/packaging-222.pom: Read timed out -> [Help 1]

@findepi findepi changed the title Flaky ci when compiling individual commits due to network errors Flaky ci check-commit when compiling individual commits due to network errors Jan 27, 2023
@nineinchnick
Copy link
Member

@findepi #15797 should help with this, it's ready for review, PTAL.

@findepi findepi mentioned this issue Jan 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working dx Issues or pull requests related to developer experience test
Milestone
No milestone
Development

No branches or pull requests
3 participants
@findepi @nineinchnick @MiguelWeezardo