Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mac_x64 module_test_ios is 3.00% flaky #144680

Closed
fluttergithubbot opened this issue Mar 6, 2024 · 15 comments · Fixed by #148264
Closed

Mac_x64 module_test_ios is 3.00% flaky #144680

fluttergithubbot opened this issue Mar 6, 2024 · 15 comments · Fixed by #148264
Assignees
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P1 High-priority issues at the top of the work list team-tool Owned by Flutter Tool team

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Mac_x64 module_test_ios had a flaky ratio 3.00% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_x64%20module_test_ios/692
Commit: d3c274b

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_x64%20module_test_ios/692
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_x64%20module_test_ios/673
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_x64%20module_test_ios/668

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_x64%20module_test_ios

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

@fluttergithubbot fluttergithubbot added c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-tool Owned by Flutter Tool team labels Mar 6, 2024
@jmagman
Copy link
Member

jmagman commented Mar 7, 2024

 [2024-03-03 19:56:51.382516] [STDOUT] stdout:     t =    34.02s Get number of matches for: Elements matching predicate '"This is Marquee" IN identifiers'
 [2024-03-03 19:56:52.057342] [STDOUT] stdout: /Volumes/Work/s/w/ir/x/t/flutter_module_test.tfsA5B/hello_host_app/FlutterUITests/FlutterUITests.m:122: error: -[FlutterUITests testDualCold] : Asynchronous wait failed: Exceeded timeout of 30 seconds, with unfulfilled expectations: "Expect predicate `count == 2` for object <XCUIElementQuery: 0x600002140230>".
 [2024-03-03 19:56:52.059265] [STDOUT] stdout:     t =    34.70s Tear Down
 [2024-03-03 19:56:52.303887] [STDOUT] stdout: Test Case '-[FlutterUITests testDualCold]' failed (34.944 seconds).

The simulator rebooted in the middle of the test, which would be an Xcode issue.
Screenshot 2024-03-07 at 10 10 41 AM

This looks the same as #142125. #143967 added retry logic for testFlutterViewWarm but this one is testDualCold.

@jmagman
Copy link
Member

jmagman commented Mar 7, 2024

 [2024-03-01 04:45:29.765690] [STDOUT] stdout:     t =    66.75s     Checking `Expect predicate `exists == 1` for object "Flutter iOS Demos Home" NavigationBar`
 [2024-03-01 04:45:29.766683] [STDOUT] stdout:     t =    66.75s         Checking existence of `"Flutter iOS Demos Home" NavigationBar`
 [2024-03-01 04:45:29.791278] [STDOUT] stdout:     t =    66.77s         Capturing element debug description
 [2024-03-01 04:45:29.968312] [STDOUT] stdout:     t =    66.95s Checking existence of `"Flutter iOS Demos Home" NavigationBar`
 [2024-03-01 04:45:29.991019] [STDOUT] stdout:     t =    66.97s Capturing debug information
 [2024-03-01 04:45:29.992448] [STDOUT] stdout:     t =    66.97s     Requesting snapshot of accessibility hierarchy for app with pid 45473
 [2024-03-01 04:45:30.174742] [STDOUT] stdout: /Volumes/Work/s/w/ir/x/t/flutter_module_test.Yt9NmR/hello_host_app/FlutterUITests/FlutterUITests.m:126: error: -[FlutterUITests testDualCold] : (([app.navigationBars[@"Flutter iOS Demos Home"] waitForExistenceWithTimeout:kStandardTimeOut]) is true) failed
 [2024-03-01 04:45:30.176602] [STDOUT] stdout:     t =    67.16s Tear Down
 [2024-03-01 04:45:30.445736] [STDOUT] stdout: Test Case '-[FlutterUITests testDualCold]' failed (67.426 seconds).

It's rebooting randomly in the middle of these tests on random "waits" so just doing the retry on "Increment via Flutter" isn't enough, it needs to be all waits. 🤮

auto-submit bot pushed a commit that referenced this issue Mar 7, 2024
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
  "name": "Mac_x64 module_test_ios"
}
-->
Issue link: #144680
@christopherfujino christopherfujino added P1 High-priority issues at the top of the work list and removed P0 Critical issues such as a build break or regression labels Mar 26, 2024
@fluttergithubbot
Copy link
Contributor Author

[staging pool] flaky ratio for the past (up to) 100 commits between 2024-03-19 and 2024-03-26 is 2.00%. Flaky number: 2; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/360
Commit: 3373961
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/360
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/311

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_x64%20module_test_ios

@fluttergithubbot
Copy link
Contributor Author

@fluttergithubbot
Copy link
Contributor Author

[staging pool] flaky ratio for the past (up to) 100 commits between 2024-03-26 and 2024-04-02 is 2.00%. Flaky number: 2; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/449
Commit: bca1d6a
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/449
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/437

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_x64%20module_test_ios

@fluttergithubbot
Copy link
Contributor Author

@fluttergithubbot
Copy link
Contributor Author

[staging pool] flaky ratio for the past (up to) 100 commits between 2024-04-17 and 2024-04-23 is 1.00%. Flaky number: 1; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/814
Commit: d737b7b
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/814

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_x64%20module_test_ios

@fluttergithubbot
Copy link
Contributor Author

@fluttergithubbot
Copy link
Contributor Author

[staging pool] flaky ratio for the past (up to) 100 commits between 2024-05-01 and 2024-05-07 is 2.00%. Flaky number: 2; total number: 100.
One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/1025
Commit: 2e8c9fb
Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/1025
https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_x64%20module_test_ios/1023

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_x64%20module_test_ios

@jmagman
Copy link
Member

jmagman commented May 13, 2024

Deleting with #148264, this test isn't adding anything.

Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
c: flake Tests that sometimes, but not always, incorrectly pass P1 High-priority issues at the top of the work list team-tool Owned by Flutter Tool team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants