Skip to content

Commit

Permalink
Add more explicit logging (just to stderr) if a try-job detects an …
Browse files Browse the repository at this point in the history
…untriaged image (#51454)

Work towards flutter/flutter#145219.

Previously all logging would be silent in the case that `test_foo` uploads a digest that is considered "untriaged", and we'd be entirely reliant on the `flutter-gold` check to pick this up asynchronously. 

As part of debugging flutter/flutter#145219 (but probably to keep this code indefinitely, it's not harmful), we now unconditionally log the swallowed failures to `stderr` so they will show up in our LUCI logs.

/cc @gaaclarke @jonahwilliams
  • Loading branch information
matanlurey authored Mar 15, 2024
1 parent 80a93a0 commit ba1115c
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 16 deletions.
47 changes: 31 additions & 16 deletions testing/skia_gold_client/lib/skia_gold_client.dart
Original file line number Diff line number Diff line change
Expand Up @@ -398,23 +398,38 @@ interface class SkiaGoldClient {
];

final io.ProcessResult result = await _runCommand(tryjobCommand);

final String resultStdout = result.stdout.toString();
if (result.exitCode != 0 &&
!(resultStdout.contains('Untriaged') || resultStdout.contains('negative image'))) {
final StringBuffer buf = StringBuffer()
..writeln('Unexpected Gold tryjobAdd failure.')
..writeln('Tryjob execution for golden file test $testName failed for')
..writeln('a reason unrelated to pixel comparison.');
throw SkiaGoldProcessError(
command: tryjobCommand,
stdout: resultStdout,
stderr: result.stderr.toString(),
message: buf.toString(),
);
} else if (verbose) {
_stderr.writeln('stdout:\n${result.stdout}');
_stderr.writeln('stderr:\n${result.stderr}');
if (result.exitCode == 0) {
// In "verbose" (debugging) mode, print the output of the tryjob anyway.
if (verbose) {
_stderr.writeln('stdout:\n${result.stdout}');
_stderr.writeln('stderr:\n${result.stderr}');
}
} else {
// Neither of these conditions are considered failures during tryjobs.
final bool isUntriaged = resultStdout.contains('Untriaged');
final bool isNegative = resultStdout.contains('negative image');
if (!isUntriaged && !isNegative) {
final StringBuffer buf = StringBuffer()
..writeln('Unexpected Gold tryjobAdd failure.')
..writeln('Tryjob execution for golden file test $testName failed for')
..writeln('a reason unrelated to pixel comparison.');
throw SkiaGoldProcessError(
command: tryjobCommand,
stdout: resultStdout,
stderr: result.stderr.toString(),
message: buf.toString(),
);
}
// ... but we want to know about them anyway.
// See https://github.com/flutter/flutter/issues/145219.
// TODO(matanlurey): Update the documentation to reflect the new behavior.
if (isUntriaged) {
_stderr
..writeln('NOTE: Untriaged image detected in tryjob.')
..writeln('Triage should be required by the "Flutter Gold" check')
..writeln('stdout:\n$resultStdout');
}
}
}

Expand Down
4 changes: 4 additions & 0 deletions testing/skia_gold_client/test/skia_gold_client_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,10 @@ void main() {
io.File(p.join(fixture.workDirectory.path, 'temp', 'golden.png')),
screenshotSize: 1000,
);

// Expect a stderr log message.
final String log = fixture.outputSink.toString();
expect(log, contains('Untriaged image detected'));
} finally {
fixture.dispose();
}
Expand Down

0 comments on commit ba1115c

Please sign in to comment.