start checking some third party builds for regressions #537
Conversation
|
Note that this is fixing the issue even though I've only implemented the basic form of what was discussed. This is on purpose; I'd rather not leave an umbrella issue open with vague ideas, because then it could stay open for a long time. I've also grown less convinced that running the tests will be useful. Some tests will simply break with obfuscation by design, if they rely on position information or reflection. Other tests may be slow or flaky. Overall, I'm not convinced that running them as part of CI will catch significantly more bugs; the majority of bugs and regressions in garble are because We could always add a mode to the script, like |
Our tests should already be pretty extensive, and any bug fixes should result in more regression test cases, but testing against a few diverse and popular third party modules will help prevent unintended regressions while developing garble. The list is short for now. More can be added later. This adds protobuf and wireguard from the original issue, but not cobra and logrus, as they aren't particularly complex nor add significant variety on top of protobuf and wireguard. While here, we remove the job that only runs crlf-test.sh, as we don't really need a separate job for a tiny script. Fixes burrowers#240.
|
the ubuntu-latest 1.18 job goes from ~12m to ~13m, so not too bad. |
Test asynchronously... how? I see two main options:
In fact, either one of these can be done via separate jobs/workflows in the same repository, so a separate repo doesn't gain us anything. I'd personally only go for a separate repo if we absolutely had to, if e.g. these third party tests required checking in tons of files into git and we wanted to keep the main repo small. But that isn't the case. |
(see commit message)
Fixes #240.