[CI] Add tests to github workflow #983
Conversation
Add unit tests to github workflow and also creating a "bad apples" environment variable. Some unit tests just fail on the CI for hardware issues. They should be improved but step one will be calling out the bad apples. Also due to the flakiness we can cache the previous run results and only run the tests that failed. It's too random to catch with the bad apples mechanism. But still added the continue on error for unit tests because it takes so long to re-run on the CI. So instead if it does fail we automatically comment there was a failure and ask them to re-run. Next step will be improving. Also needed to limit the amount of workers because otherwise the hardware can't handle well so then it will accidentally create conflicts. This means we get an accurate test run but it is slower on the CI. Included integration tests which worked out of the box. Included e2e tests as well but it the chrome driver for the application was different from github's chrome so to run it I just upgraded it for the test run. Not ideal, ideally we should probably set up a docker env and install the specific versions since we are now depending on github's virtual env and the dependencies they installed there. But at least this is a first pace. Signed-off-by: Kawika Avilla <[email protected]>
|
Variant of this PR: #906 Except this one differs by allowing re-runs if failures and will only run the specific steps that failed. Please note, if someone pushing again it will create a new job and re-run all the steps. The biggest con about this is that it will basically not logged the steps that were successful in previous runs. But I prefer this way because it's really painful to wait 2 hours because for tests that are intermittent on github (but dont really fall into the bad apples category because it doesn't always fail on the github CI), then re-run and have another flaky failure. If you check out this run: https://github.com/kavilla/OpenSearch-Dashboards-1/actions/runs/1509682521 you can see the first run the unit tests were flaky, then the second run was a re-run and it didn't re-run the previous successful steps just the build and unit tests. But it kind of failed because I timestamp didn't exist so I resorted to using the hash here's this example: https://github.com/kavilla/OpenSearch-Dashboards-1/actions/runs/1517404343 |
Description
Add unit tests to github workflow and also creating a "bad apples"
environment variable. Some unit tests just fail on the CI for
hardware issues. They should be improved but step one will be
calling out the bad apples.
Also due to the flakiness we can cache the previous run results
and only run the tests that failed. It's too random to catch
with the bad apples mechanism. But still added the continue on
error for unit tests because it takes so long to re-run on the
CI. So instead if it does fail we automatically comment there
was a failure and ask them to re-run.
Next step will be improving.
Also needed to limit the amount of workers because otherwise the
hardware can't handle well so then it will accidentally create conflicts.
This means we get an accurate test run but it is slower on the CI.
Included integration tests which worked out of the box.
Included e2e tests as well but it the chrome driver for the application
was different from github's chrome so to run it I just upgraded it for
the test run. Not ideal, ideally we should probably set up a
docker env and install the specific versions since we are now
depending on github's virtual env and the dependencies they installed
there. But at least this is a first pace.
Signed-off-by: Kawika Avilla [email protected]
Issues Resolved
n/a
Check List
yarn test:jestyarn test:jest_integrationyarn test:ftrNew functionality has been documented.