Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI][FS][Azure] Azurite tests are flaking on main #40121

Open
Tom-Newton opened this issue Feb 18, 2024 · 2 comments
Open

[CI][FS][Azure] Azurite tests are flaking on main #40121

Tom-Newton opened this issue Feb 18, 2024 · 2 comments
Labels
Component: C++ Component: Continuous Integration Type: bug

Comments

@Tom-Newton
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

Flaky failures like

C++ exception with description "Connection closed before getting full response or response is less than expected. Expected response length = 254. Read until now = 231" thrown in the test body.
2024-02-18T12:50:20.039Z ada6933e-9c33-47d2-86f6-29e9aa01f713 info: BlobStorageContextMiddleware: RequestMethod=DELETE RequestURL=http://127.0.0.1/devstoreaccount1/container?restype=container RequestHeaders:{"authorization":"SharedKey devstoreaccount1:hYh+JRj5cBYqdqOyM2wB3EZizQ/s2DiIoDI0CIF2EXM=","host":"127.0.0.1:10000","user-agent":"azsdk-cpp-storage-blobs/12.10.0-beta.1 (Linux 6.2.0-1019-azure x86_64 #19~22.04.1-Ubuntu SMP Wed Jan 10 22:57:03 UTC 2024)","x-ms-client-request-id":"be6819a2-72b8-4630-8eb0-4a88e7cb3061","x-ms-date":"Sun, 18 Feb 2024 12:50:20 GMT","x-ms-version":"2022-11-02"} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1

I've seen the occur in different test cases and in different test suites.

Example failures:
I've seen one flake on main: https://github.com/apache/arrow/actions/runs/7915689559/job/21608061673
Flakes on my recent PRs:
https://github.com/apache/arrow/actions/runs/7951594516/job/21705210845?pr=40080
https://github.com/apache/arrow/actions/runs/7949050250/job/21699789831?pr=40080

Component(s)

C++, Continuous Integration
@Tom-Newton Tom-Newton mentioned this issue Feb 18, 2024
Merged
@kou
Copy link
Member

kou commented Feb 19, 2024

Hmm, it seems that failed tests aren't same...
Can we re-run failed tests?

diff --git a/ci/scripts/cpp_test.sh b/ci/scripts/cpp_test.sh
index 1d685c51a9..a23ea8eb1c 100755
--- a/ci/scripts/cpp_test.sh
+++ b/ci/scripts/cpp_test.sh
@@ -86,6 +86,7 @@ ctest \
     --label-regex unittest \
     --output-on-failure \
     --parallel ${n_jobs} \
+    --repeat until-pass:3 \
     --timeout ${ARROW_CTEST_TIMEOUT:-300} \
     "${ctest_options[@]}" \
     "$@"

@Tom-Newton
Copy link
Contributor Author

I expect retries would be an effective mitigation.

Tom-Newton added a commit to Tom-Newton/arrow that referenced this issue Feb 19, 2024
@Tom-Newton
Add retry mitigation suggested for apacheGH-40121
31e4920
Tom-Newton added a commit to Tom-Newton/arrow that referenced this issue Feb 19, 2024
@Tom-Newton
Add retry mitigation suggested for apacheGH-40121
5e3f846
kou pushed a commit that referenced this issue Feb 22, 2024
@Tom-Newton
GH-40079: [CI][Packaging] Enable Azure in more tests and builds (#40080)
b089c6a 
### Rationale for this change
We want python side tests of `AzureFileSystem` to run in CI. 

### What changes are included in this PR?
- Add missing `export` to enable Azure pyarrow tests
- Enable azure in sdist tests.
- Enable Azure on macos python builds
- Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
- Enable retries on C++ tests to mitigate #40121

Probably all of this should have been included in #39971

### Are these changes tested?
There is no new functionality to test

### Are there any user-facing changes?
No

* Closes: #40079
* GitHub Issue: #40079

Authored-by: Thomas Newton <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
@Tom-Newton @zanmato1984
apacheGH-40079: [CI][Packaging] Enable Azure in more tests and builds (…
eeacd01 
…apache#40080)

### Rationale for this change
We want python side tests of `AzureFileSystem` to run in CI. 

### What changes are included in this PR?
- Add missing `export` to enable Azure pyarrow tests
- Enable azure in sdist tests.
- Enable Azure on macos python builds
- Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
- Enable retries on C++ tests to mitigate apache#40121

Probably all of this should have been included in apache#39971

### Are these changes tested?
There is no new functionality to test

### Are there any user-facing changes?
No

* Closes: apache#40079
* GitHub Issue: apache#40079

Authored-by: Thomas Newton <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
@Tom-Newton @thisisnic
apacheGH-40079: [CI][Packaging] Enable Azure in more tests and builds (…
3fa227b 
…apache#40080)

### Rationale for this change
We want python side tests of `AzureFileSystem` to run in CI. 

### What changes are included in this PR?
- Add missing `export` to enable Azure pyarrow tests
- Enable azure in sdist tests.
- Enable Azure on macos python builds
- Enable azure in conda builds and install dependencies (Azure C++ SDK and azurite)
- Enable retries on C++ tests to mitigate apache#40121

Probably all of this should have been included in apache#39971

### Are these changes tested?
There is no new functionality to test

### Are there any user-facing changes?
No

* Closes: apache#40079
* GitHub Issue: apache#40079

Authored-by: Thomas Newton <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: C++ Component: Continuous Integration Type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kou @Tom-Newton