Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry on transient Skia failure. #139182

Merged
merged 1 commit into from
Dec 1, 2023
Merged

Retry on transient Skia failure. #139182

merged 1 commit into from
Dec 1, 2023

Conversation

Hixie
Copy link
Contributor

@Hixie Hixie commented Nov 29, 2023

This works around https://g-issues.skia.org/issues/40044713 using exponential backoff.

This is completely untested because I have no idea how to test it.

Also I only changed one of the code paths here. I figure if we get success here then we can start propagating the change to other places in this file that generate errors, maybe factoring out the retry and error reporting logic so it's not duplicated multiple times.

@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie or stuartmorgan on the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@github-actions github-actions bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Nov 29, 2023
@Hixie Hixie force-pushed the gold502 branch 2 times, most recently from 8d36f90 to d123f2d Compare November 29, 2023 01:25
@Hixie
Copy link
Contributor Author

Hixie commented Nov 29, 2023

I added a test. It uses zones. I don't know how else to test this.

This works around https://g-issues.skia.org/issues/40044713 using exponential backoff.

This is completely untested because I have no idea how to test it.

Also I only changed one of the code paths here. I figure if we get success here then we can start propagating the change to other places in this file that generate errors, maybe factoring out the retry and error reporting logic so it's not duplicated multiple times.
Copy link
Contributor

@Piinks Piinks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of nits, but LGTM regardless!

@@ -0,0 +1,85 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not recall why, but all the tests for the client are with the rest of the flutter_goldens tests. There is already a FakeProcessManager etc. there if you want to move them there to share code.

Then again, it does make more sense to test the client.. in the client package. 🙃

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird. Why are they two packages then? Should we merge them? There's no README for either so I'm not sure I really understand the intended use of these packages...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm going to land this as-is for now and we can talk about what we want to do with these packages separately. Either way we're going to have to follow-up here (either this helps the flakiness or not, and if it does, we'll want to more aggressively do this approach in other places in this file).

final List<String> log = <String>[];
await runZoned(
zoneSpecification: ZoneSpecification(
print: (Zone self, ZoneDelegate parent, Zone zone, String line) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nifty. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's horrific is what it is but there we are. :-)

sdk: flutter
test: 1.24.9

_fe_analyzer_shared: 65.0.0 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holy dependencies batman!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah.

// Ideally we'd use something like package:test's printOnError, but best reliability
// in getting logs on CI for now we're just using print.
// See also: https://github.com/flutter/flutter/issues/91285
print('Transient failure from Skia Gold, retrying in ${delay.inSeconds} seconds.'); // ignore: avoid_print
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe at this point we should add avoid_print for the whole file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to increase the pain I feel each time I add one so that eventually I snap and fix the tech debt...

@Hixie Hixie added the autosubmit Merge PR when tree becomes green via auto submit App label Dec 1, 2023
@auto-submit auto-submit bot merged commit b2d90eb into flutter:master Dec 1, 2023
57 checks passed
@Hixie Hixie added the revert Autorevert PR (with "Reason for revert:" comment) label Dec 2, 2023
@Hixie
Copy link
Contributor Author

Hixie commented Dec 2, 2023

00:09 +6 -1: SkiaGoldClient throws for error state from tryjobAdd [E]                                                                                                                                  
  Expected: throws <Instance of 'SkiaException'> with `message`: contains 'result-state.json'
    Actual: <Instance of 'Future<void>'>
     Which: threw SkiaException:<SkiaException: Golden test for "golden_file_test" failed with exit code 1 for a reason unrelated to pixel comparison.
                  
                  stdout from gold:
                    Fallback failure
                  
                  stderr from gold:
                    Fallback failure
                  >
            stack package:flutter_goldens_client/skia_client.dart 394:7  SkiaGoldClient.tryjobAdd
                  ===== asynchronous gap ===========================
                  package:matcher                                        expect
                  package:flutter_test/src/widget_tester.dart 458:18     expect
                  test/flutter_goldens_test.dart 569:7                   main.<fn>.<fn>
                  
            which has `message` with value 'Golden test for "golden_file_test" failed with exit code 1 for a reason unrelated to pixel comparison.\n'
                    '\n'
                    'stdout from gold:\n'
                    '  Fallback failure\n'
                    '\n'
                    'stderr from gold:\n'
                    '  Fallback failure\n'
                    '' which does not contain 'result-state.json'
  
  package:matcher                                     expect
  package:flutter_test/src/widget_tester.dart 458:18  expect
  test/flutter_goldens_test.dart 569:7                main.<fn>.<fn>
  

To run this test again: /b/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test /b/s/w/ir/x/w/flutter/packages/flutter_goldens/test/flutter_goldens_test.dart -p vm --plain-name 'SkiaGoldClient throws for error state from tryjobAdd'

@auto-submit auto-submit bot removed the revert Autorevert PR (with "Reason for revert:" comment) label Dec 2, 2023
auto-submit bot pushed a commit that referenced this pull request Dec 2, 2023
auto-submit bot added a commit that referenced this pull request Dec 2, 2023
Reverts #139182
Initiated by: Hixie
This change reverts the following previous change:
Original Description:
This works around https://g-issues.skia.org/issues/40044713 using exponential backoff.

This is completely untested because I have no idea how to test it.

Also I only changed one of the code paths here. I figure if we get success here then we can start propagating the change to other places in this file that generate errors, maybe factoring out the retry and error reporting logic so it's not duplicated multiple times.
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 2, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 2, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Dec 2, 2023
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Dec 2, 2023
flutter/flutter@918e336...d861ce4

2023-12-02 [email protected] Roll Flutter Engine from f0122c32c5cc to cfabe42bc0c6 (1 revision) (flutter/flutter#139423)
2023-12-02 [email protected] Roll Flutter Engine from f23c33f3831c to f0122c32c5cc (1 revision) (flutter/flutter#139422)
2023-12-02 [email protected] Roll Flutter Engine from d441f087052c to f23c33f3831c (2 revisions) (flutter/flutter#139421)
2023-12-02 [email protected] Roll Flutter Engine from 27d37db84b8e to d441f087052c (1 revision) (flutter/flutter#139419)
2023-12-02 [email protected] Roll Flutter Engine from 5a9f33e3a41e to 27d37db84b8e (1 revision) (flutter/flutter#139418)
2023-12-02 [email protected] Roll Flutter Engine from 9f8502c4e255 to 5a9f33e3a41e (1 revision) (flutter/flutter#139416)
2023-12-02 [email protected] Roll Flutter Engine from 43a1598713bb to 9f8502c4e255 (1 revision) (flutter/flutter#139414)
2023-12-02 [email protected] Roll Flutter Engine from 039439c1ffe8 to 43a1598713bb (1 revision) (flutter/flutter#139412)
2023-12-02 [email protected] Roll Flutter Engine from 4d19fedb7617 to 039439c1ffe8 (1 revision) (flutter/flutter#139410)
2023-12-02 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Retry on transient Skia failure." (flutter/flutter#139407)
2023-12-01 [email protected] Roll Flutter Engine from 162ad29a576f to 4d19fedb7617 (1 revision) (flutter/flutter#139397)
2023-12-01 [email protected] [l10n] Update Material shareButtonLabel (flutter/flutter#138899)
2023-12-01 [email protected] Retry on transient Skia failure. (flutter/flutter#139182)
2023-12-01 [email protected] Roll Flutter Engine from 820cb686d17d to 162ad29a576f (1 revision) (flutter/flutter#139394)
2023-12-01 [email protected] Roll Flutter Engine from 00316e4b7680 to 820cb686d17d (2 revisions) (flutter/flutter#139390)
2023-12-01 [email protected] Roll Flutter Engine from 95995c48d591 to 00316e4b7680 (1 revision) (flutter/flutter#139389)
2023-12-01 [email protected] Roll Flutter Engine from 69f0e5550702 to 95995c48d591 (6 revisions) (flutter/flutter#139388)
2023-12-01 [email protected] Added vscode-insiders path installed via snap (flutter/flutter#137117)
2023-12-01 [email protected] Typo fix in dartdoc in tool test (flutter/flutter#139386)
2023-12-01 [email protected] Roll Flutter Engine from 51ef7642750f to 69f0e5550702 (1 revision) (flutter/flutter#139348)
2023-12-01 [email protected] Roll Flutter Engine from 894360cca1ec to 51ef7642750f (1 revision) (flutter/flutter#139346)
2023-12-01 [email protected] Roll Flutter Engine from c26e6ced11df to 894360cca1ec (1 revision) (flutter/flutter#139345)
2023-12-01 [email protected] Roll Flutter Engine from 74d2df52514a to c26e6ced11df (26 revisions) (flutter/flutter#139342)
2023-12-01 [email protected] Roll Flutter Engine from 35939ca8534f to 74d2df52514a (1 revision) (flutter/flutter#139264)
2023-11-30 [email protected] Add `undoStackModifier` to `UndoHistory` (flutter/flutter#138674)
2023-11-30 [email protected] Migrate docs_test to shard. (flutter/flutter#139282)
2023-11-30 [email protected] Write Tests for API Examples of `cupertino_text_field.0`, `data_table.0`, `icon_button.2` & `ink_well.0` (flutter/flutter#139258)
2023-11-30 [email protected] Refactor prepare_package.dart (flutter/flutter#139277)
2023-11-30 [email protected] Roll Packages from e4aaba8 to bc72d15 (4 revisions) (flutter/flutter#139307)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
HugoOlthof pushed a commit to moneybird/packages that referenced this pull request Dec 13, 2023
flutter/flutter@918e336...d861ce4

2023-12-02 [email protected] Roll Flutter Engine from f0122c32c5cc to cfabe42bc0c6 (1 revision) (flutter/flutter#139423)
2023-12-02 [email protected] Roll Flutter Engine from f23c33f3831c to f0122c32c5cc (1 revision) (flutter/flutter#139422)
2023-12-02 [email protected] Roll Flutter Engine from d441f087052c to f23c33f3831c (2 revisions) (flutter/flutter#139421)
2023-12-02 [email protected] Roll Flutter Engine from 27d37db84b8e to d441f087052c (1 revision) (flutter/flutter#139419)
2023-12-02 [email protected] Roll Flutter Engine from 5a9f33e3a41e to 27d37db84b8e (1 revision) (flutter/flutter#139418)
2023-12-02 [email protected] Roll Flutter Engine from 9f8502c4e255 to 5a9f33e3a41e (1 revision) (flutter/flutter#139416)
2023-12-02 [email protected] Roll Flutter Engine from 43a1598713bb to 9f8502c4e255 (1 revision) (flutter/flutter#139414)
2023-12-02 [email protected] Roll Flutter Engine from 039439c1ffe8 to 43a1598713bb (1 revision) (flutter/flutter#139412)
2023-12-02 [email protected] Roll Flutter Engine from 4d19fedb7617 to 039439c1ffe8 (1 revision) (flutter/flutter#139410)
2023-12-02 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Retry on transient Skia failure." (flutter/flutter#139407)
2023-12-01 [email protected] Roll Flutter Engine from 162ad29a576f to 4d19fedb7617 (1 revision) (flutter/flutter#139397)
2023-12-01 [email protected] [l10n] Update Material shareButtonLabel (flutter/flutter#138899)
2023-12-01 [email protected] Retry on transient Skia failure. (flutter/flutter#139182)
2023-12-01 [email protected] Roll Flutter Engine from 820cb686d17d to 162ad29a576f (1 revision) (flutter/flutter#139394)
2023-12-01 [email protected] Roll Flutter Engine from 00316e4b7680 to 820cb686d17d (2 revisions) (flutter/flutter#139390)
2023-12-01 [email protected] Roll Flutter Engine from 95995c48d591 to 00316e4b7680 (1 revision) (flutter/flutter#139389)
2023-12-01 [email protected] Roll Flutter Engine from 69f0e5550702 to 95995c48d591 (6 revisions) (flutter/flutter#139388)
2023-12-01 [email protected] Added vscode-insiders path installed via snap (flutter/flutter#137117)
2023-12-01 [email protected] Typo fix in dartdoc in tool test (flutter/flutter#139386)
2023-12-01 [email protected] Roll Flutter Engine from 51ef7642750f to 69f0e5550702 (1 revision) (flutter/flutter#139348)
2023-12-01 [email protected] Roll Flutter Engine from 894360cca1ec to 51ef7642750f (1 revision) (flutter/flutter#139346)
2023-12-01 [email protected] Roll Flutter Engine from c26e6ced11df to 894360cca1ec (1 revision) (flutter/flutter#139345)
2023-12-01 [email protected] Roll Flutter Engine from 74d2df52514a to c26e6ced11df (26 revisions) (flutter/flutter#139342)
2023-12-01 [email protected] Roll Flutter Engine from 35939ca8534f to 74d2df52514a (1 revision) (flutter/flutter#139264)
2023-11-30 [email protected] Add `undoStackModifier` to `UndoHistory` (flutter/flutter#138674)
2023-11-30 [email protected] Migrate docs_test to shard. (flutter/flutter#139282)
2023-11-30 [email protected] Write Tests for API Examples of `cupertino_text_field.0`, `data_table.0`, `icon_button.2` & `ink_well.0` (flutter/flutter#139258)
2023-11-30 [email protected] Refactor prepare_package.dart (flutter/flutter#139277)
2023-11-30 [email protected] Roll Packages from e4aaba8 to bc72d15 (4 revisions) (flutter/flutter#139307)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
caseycrogers pushed a commit to caseycrogers/flutter that referenced this pull request Dec 29, 2023
This works around https://g-issues.skia.org/issues/40044713 using exponential backoff.

This is completely untested because I have no idea how to test it.

Also I only changed one of the code paths here. I figure if we get success here then we can start propagating the change to other places in this file that generate errors, maybe factoring out the retry and error reporting logic so it's not duplicated multiple times.
caseycrogers pushed a commit to caseycrogers/flutter that referenced this pull request Dec 29, 2023
Reverts flutter#139182
Initiated by: Hixie
This change reverts the following previous change:
Original Description:
This works around https://g-issues.skia.org/issues/40044713 using exponential backoff.

This is completely untested because I have no idea how to test it.

Also I only changed one of the code paths here. I figure if we get success here then we can start propagating the change to other places in this file that generate errors, maybe factoring out the retry and error reporting logic so it's not duplicated multiple times.
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 16, 2024
arc-yong pushed a commit to Arctuition/packages-arc that referenced this pull request Jun 14, 2024
flutter/flutter@918e336...d861ce4

2023-12-02 [email protected] Roll Flutter Engine from f0122c32c5cc to cfabe42bc0c6 (1 revision) (flutter/flutter#139423)
2023-12-02 [email protected] Roll Flutter Engine from f23c33f3831c to f0122c32c5cc (1 revision) (flutter/flutter#139422)
2023-12-02 [email protected] Roll Flutter Engine from d441f087052c to f23c33f3831c (2 revisions) (flutter/flutter#139421)
2023-12-02 [email protected] Roll Flutter Engine from 27d37db84b8e to d441f087052c (1 revision) (flutter/flutter#139419)
2023-12-02 [email protected] Roll Flutter Engine from 5a9f33e3a41e to 27d37db84b8e (1 revision) (flutter/flutter#139418)
2023-12-02 [email protected] Roll Flutter Engine from 9f8502c4e255 to 5a9f33e3a41e (1 revision) (flutter/flutter#139416)
2023-12-02 [email protected] Roll Flutter Engine from 43a1598713bb to 9f8502c4e255 (1 revision) (flutter/flutter#139414)
2023-12-02 [email protected] Roll Flutter Engine from 039439c1ffe8 to 43a1598713bb (1 revision) (flutter/flutter#139412)
2023-12-02 [email protected] Roll Flutter Engine from 4d19fedb7617 to 039439c1ffe8 (1 revision) (flutter/flutter#139410)
2023-12-02 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Retry on transient Skia failure." (flutter/flutter#139407)
2023-12-01 [email protected] Roll Flutter Engine from 162ad29a576f to 4d19fedb7617 (1 revision) (flutter/flutter#139397)
2023-12-01 [email protected] [l10n] Update Material shareButtonLabel (flutter/flutter#138899)
2023-12-01 [email protected] Retry on transient Skia failure. (flutter/flutter#139182)
2023-12-01 [email protected] Roll Flutter Engine from 820cb686d17d to 162ad29a576f (1 revision) (flutter/flutter#139394)
2023-12-01 [email protected] Roll Flutter Engine from 00316e4b7680 to 820cb686d17d (2 revisions) (flutter/flutter#139390)
2023-12-01 [email protected] Roll Flutter Engine from 95995c48d591 to 00316e4b7680 (1 revision) (flutter/flutter#139389)
2023-12-01 [email protected] Roll Flutter Engine from 69f0e5550702 to 95995c48d591 (6 revisions) (flutter/flutter#139388)
2023-12-01 [email protected] Added vscode-insiders path installed via snap (flutter/flutter#137117)
2023-12-01 [email protected] Typo fix in dartdoc in tool test (flutter/flutter#139386)
2023-12-01 [email protected] Roll Flutter Engine from 51ef7642750f to 69f0e5550702 (1 revision) (flutter/flutter#139348)
2023-12-01 [email protected] Roll Flutter Engine from 894360cca1ec to 51ef7642750f (1 revision) (flutter/flutter#139346)
2023-12-01 [email protected] Roll Flutter Engine from c26e6ced11df to 894360cca1ec (1 revision) (flutter/flutter#139345)
2023-12-01 [email protected] Roll Flutter Engine from 74d2df52514a to c26e6ced11df (26 revisions) (flutter/flutter#139342)
2023-12-01 [email protected] Roll Flutter Engine from 35939ca8534f to 74d2df52514a (1 revision) (flutter/flutter#139264)
2023-11-30 [email protected] Add `undoStackModifier` to `UndoHistory` (flutter/flutter#138674)
2023-11-30 [email protected] Migrate docs_test to shard. (flutter/flutter#139282)
2023-11-30 [email protected] Write Tests for API Examples of `cupertino_text_field.0`, `data_table.0`, `icon_button.2` & `ink_well.0` (flutter/flutter#139258)
2023-11-30 [email protected] Refactor prepare_package.dart (flutter/flutter#139277)
2023-11-30 [email protected] Roll Packages from e4aaba8 to bc72d15 (4 revisions) (flutter/flutter#139307)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: tests "flutter test", flutter_test, or one of our tests autosubmit Merge PR when tree becomes green via auto submit App framework flutter/packages/flutter repository. See also f: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants