[SPARK-47186][DOCKER][TESTS] Add some timeouts options and logs to improve the debuggability for docker integration test #45284
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…prove the debuggability for docker integration test
dongjoon-hyun
approved these changes
Feb 27, 2024
|
Thanks @dongjoon-hyun, About the oracle issue, I'm still consulting with upstream authors gvenzl/oci-oracle-free#35 |
|
Thank you. Feel free to merge~ |
|
Thank you @dongjoon-hyun, merged to master |
TakawaAkirayo
pushed a commit
to TakawaAkirayo/spark
that referenced
this pull request
Mar 4, 2024
…prove the debuggability for docker integration test ### What changes were proposed in this pull request? This PR adds test-scoped options: - Timeout for pulling the Docker image before the tests start. - `spark.test.docker.imagePullTimeout` - Timeout for container to spin up. - `spark.test.docker.startContainerTimeout` - Timeout for connecting the inner service in the container - `spark.test.docker.connectionTimeout` This PR also adds loggings(excluding the downloading/extracting details) for the imaging pulling step which is time-consuming: ``` 24/02/27 19:03:17.112 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling from gvenzl/oracle-free 23.3-slim 24/02/27 19:03:17.112 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 5cbb6d705282 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer f1544b3116d0 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 1dff807126c4 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 603266ad0104 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 10f286d1795c 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 7c4de5471fcf 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 603266ad0104 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 10f286d1795c 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 7c4de5471fcf 24/02/27 19:03:59.725 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 5cbb6d705282 24/02/27 19:03:59.725 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 5cbb6d705282 24/02/27 19:04:12.512 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 5cbb6d705282 62.3 MiB 24/02/27 19:04:12.801 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 5cbb6d705282 24/02/27 19:04:25.905 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum f1544b3116d0 24/02/27 19:04:25.906 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete f1544b3116d0 24/02/27 19:04:39.533 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting f1544b3116d0 103.5 MiB 24/02/27 19:04:39.647 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete f1544b3116d0 24/02/27 19:04:46.451 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 10f286d1795c 24/02/27 19:04:46.452 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 10f286d1795c 24/02/27 19:05:39.623 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 7c4de5471fcf 24/02/27 19:05:39.623 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 7c4de5471fcf 24/02/27 19:05:40.889 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 1dff807126c4 24/02/27 19:05:40.890 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 1dff807126c4 24/02/27 19:05:51.976 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 603266ad0104 24/02/27 19:05:51.976 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 603266ad0104 24/02/27 19:05:59.357 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 1dff807126c4 178.3 MiB 24/02/27 19:05:59.429 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 1dff807126c4 24/02/27 19:06:10.751 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 603266ad0104 110.2 MiB 24/02/27 19:06:11.117 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 603266ad0104 ``` ### Why are the changes needed? Some districts might suffer from network issues with the official docker registry ### Does this PR introduce _any_ user-facing change? no, dev-only ### How was this patch tested? docker it ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#45284 from yaooqinn/SPARK-47186. Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]>
ericm-db
pushed a commit
to ericm-db/spark
that referenced
this pull request
Mar 5, 2024
…prove the debuggability for docker integration test ### What changes were proposed in this pull request? This PR adds test-scoped options: - Timeout for pulling the Docker image before the tests start. - `spark.test.docker.imagePullTimeout` - Timeout for container to spin up. - `spark.test.docker.startContainerTimeout` - Timeout for connecting the inner service in the container - `spark.test.docker.connectionTimeout` This PR also adds loggings(excluding the downloading/extracting details) for the imaging pulling step which is time-consuming: ``` 24/02/27 19:03:17.112 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling from gvenzl/oracle-free 23.3-slim 24/02/27 19:03:17.112 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 5cbb6d705282 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer f1544b3116d0 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 1dff807126c4 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 603266ad0104 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 10f286d1795c 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pulling fs layer 7c4de5471fcf 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 603266ad0104 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 10f286d1795c 24/02/27 19:03:17.113 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Waiting 7c4de5471fcf 24/02/27 19:03:59.725 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 5cbb6d705282 24/02/27 19:03:59.725 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 5cbb6d705282 24/02/27 19:04:12.512 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 5cbb6d705282 62.3 MiB 24/02/27 19:04:12.801 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 5cbb6d705282 24/02/27 19:04:25.905 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum f1544b3116d0 24/02/27 19:04:25.906 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete f1544b3116d0 24/02/27 19:04:39.533 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting f1544b3116d0 103.5 MiB 24/02/27 19:04:39.647 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete f1544b3116d0 24/02/27 19:04:46.451 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 10f286d1795c 24/02/27 19:04:46.452 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 10f286d1795c 24/02/27 19:05:39.623 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 7c4de5471fcf 24/02/27 19:05:39.623 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 7c4de5471fcf 24/02/27 19:05:40.889 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 1dff807126c4 24/02/27 19:05:40.890 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 1dff807126c4 24/02/27 19:05:51.976 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Verifying Checksum 603266ad0104 24/02/27 19:05:51.976 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Download complete 603266ad0104 24/02/27 19:05:59.357 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 1dff807126c4 178.3 MiB 24/02/27 19:05:59.429 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 1dff807126c4 24/02/27 19:06:10.751 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Extracting 603266ad0104 110.2 MiB 24/02/27 19:06:11.117 docker-java-stream--1665796424 INFO OracleIntegrationSuite: Pull complete 603266ad0104 ``` ### Why are the changes needed? Some districts might suffer from network issues with the official docker registry ### Does this PR introduce _any_ user-facing change? no, dev-only ### How was this patch tested? docker it ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#45284 from yaooqinn/SPARK-47186. Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds test-scoped options:
spark.test.docker.imagePullTimeoutspark.test.docker.startContainerTimeoutspark.test.docker.connectionTimeoutThis PR also adds loggings(excluding the downloading/extracting details) for the imaging pulling step which is time-consuming:
Why are the changes needed?
Some districts might suffer from network issues with the official docker registry
Does this PR introduce any user-facing change?
no, dev-only
How was this patch tested?
docker it
Was this patch authored or co-authored using generative AI tooling?
no