fastlane notices error code when generating l10n screenshots but ignores it #6549
Comments
JohanLorenzo
added
the
Bug 🐞
isabelrios
added
the
eng:automation
|
I'm not sure I understand @JohanLorenzo, the test suite finishes successfully |
|
Okay! I agree, the logs give mixed signs, so maybe the crash is not related to the lack of screenshots. I've got an idea when I chatted with @gueroJeff last Thursday. He told me the l10n team usually spot check screenshots once they're uploaded on Google Drive, because sometimes some are missing. I realized this check can easily be done by Taskcluster instead of a human. I can write a script that looks into the zip archives and ensure there are exactly X screenshots (X being 54 at the moment). If some pictures are missing, the script would bail out. This way, this new script would tell us if fastlane silently failed. Moreover, the new script can fail with a specific error code and Taskcluster could automatically rerun the failed job. We probably need more data before doing so, thus I'll first make the task explicitly fail. I'm tracking the work in bug 1636842 |
Good news! I made a mistake last week. The number of failed locales is much lower: 11 out of 92. Moreover, fastlane does somehow report an error (which is ignored). Let me explain what I did. I started to write the script I mentioned in my previous comment. I needed to re-download all archives (like I did in bug 1622800 comment 15). Then, I started to test out my script. I stumbled upon different results: en-GB turned out to be complete! I first thought there was a bug in my script, so I used I don't know what I did wrong last time. Maybe extracting all 93 archives with MacOS' Finder wasn't a good idea and maybe it encountered a bug. So, now I know only 11 locales failed and each of them is just missing a handful of them: I looked into the ar logs. They do show a failure. Fastlane even calls out exit code 65 and shows a nice ❌ emoji. That said, fastlane keeps exiting with a 0 code. Thus, writing an additional script is still a good idea, in my opinion. Debugging fastlane should be easier now that we have meaningful logs 🙂. I'm changing the title of this bug. |
JohanLorenzo
changed the title
|
To add a little bit of info about the missing screenshots...
Both failures are due to different options depending on the locale that are not being considered by the test. I'm working on a fix for those issues. |
|
Excellent news: Thanks to @isabelrios great effort, the last run was entirely error-free! We're now in a position where no locale fails. This doesn't address the original bug where |
See bug 1622800 comment 15 for the full context.
Steps to reproduce
Expected behavior
fastlane should report an error whenever l10n screenshots aren't fully generated, because an emulator crash for instance.
Actual behavior
The fastlane command outputs it succeeded and returns 0 as an exit code. See logs below, as an example.
EDIT: The following is wrong. See #6549 (comment) for accurate data.
Repro rate: It happened on 31 locales out of 92.@isabelrios Would you have an idea of what might cause this discrepancy?EDIT: These logs don't show the actual problem.
The text was updated successfully, but these errors were encountered: