ios_ui_test runner is flaky #185
Comments
|
I isolated the test target in a |
|
@sergiocampama isn't this going to be the test runner and not the |
|
Flakiness has gone down now that I’ve marked ui tests with tags = ["exclusive"], but I am still seeing flakes on a runner that uses a conditional device type. E.g.: Still working on isolating this. |
|
Am continuing to update this PR: material-components/material-components-ios#4182 You can view the kokoro log outputs on the pr commit statuses. |
|
More failure logs: |
|
@thomasvl likely that it's the runner and not the target — I'm still grokking how bazel's built under the hood as I dive in to this. |
|
I have four ios_ui_test targets running on our CI, but I've been noticing for some time now that only one of the tests flakes on any given run. I duplicated these test targets and am still only seeing one target flaking now that there are eight, which makes me believe that there is something to do with either the first or last simulator run that's causing this flakiness. |
|
I can't find any of the strings output in the build logs in the bazelbuild org. Who is generating the strings like "Generating xctestrun file" and "xcodebuild command can not launch test"? |
|
I think I've hit the edge of what I can explore options-wise from our end. The only solution I found that worked ok'ish was to retry the tests after failure a couple times with the hopes that the flakiness eventually resolves itself. Would it be possible to build simulator flakiness resilience into bazel in some manner? |
|
Additional context:
|
|
The runner is an external dep: https://github.com/bazelbuild/rules_apple#quick-setup, _ xctestrunner_ That's what is defaulted into the |
|
Ah! Thank you :) Will dig into that... |
|
Hi @jverkoey, we've just updated the test runner dependency to 0.2.3, can you try with that one? |
sergiocampama
added
P3
|
We just hit this case with xctestrunner 0.2.7. Our log doesn't add much here: This is with Xcode 10.2.1 on macOS 14.4 testing on iOS 12.2 |
|
I was also running into this issue at a fairly consistent rate and isolated the issue to
Prior to running
|
|
I just hit also this issue. It is running perfectly locally, but failing multiple times on CI (github actions). The very few tries I have done so far, I have worked around the problem thanks to @jerrymarino solution, executing the following line before running bazel: The log is the following: Environment:
EDIT1: I still see the problem, despite @jerrymarino proposed solution. EDIT2: I did not realise that EDIT3: I experienced the problem again, despite the updated |
|
New update. Using I will need some more time to confirm if this is a reliable workaround, will update in a while EDIT: Yep, since using |
|
Bump--this is still a pretty big issue and Seems like everyone is basically creating their own test infrastructure when they adopt bazel for ios AFAICT |
|
I stumbled across this thread when digging into related issues. IMO the issue "simulator is flaky" demonstrates the core of it: this rule fails for hundreds of reasons but there is no way for someone to identify what part of the system is flaky or why. I estimate it's not actually tied to rules_apple - but components inside of
I like @zachgray-slack 's idea overall: replacing |
|
We wrote our own test runner for bluepill called We use it something like this |
Try to unveil problems with xctestrunner: bazelbuild/rules_apple#185 Here is 1 quick idea: we can attach sample when it fails here - obviously it extends to other possibilities. This is just a start - it should have more logging
|
Awesome, thanks for posting up the example - I'll have to take a look at it. We've also had some rules_ios PRs to dice up the methods / enable sharding them with RBE. I imagine that'll work with bluepill: configurable via For |
We're attempting to make use of
ios_ui_teston the MDC iOS repo and encountering fairly flaky builds as a result.The errors we're seeing are:
Example failure logs from this PR:
The text was updated successfully, but these errors were encountered: