Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove or improve the "error threshold" #83

Open
adgad opened this issue Aug 14, 2018 · 1 comment
Open

Remove or improve the "error threshold" #83

adgad opened this issue Aug 14, 2018 · 1 comment

Comments

@adgad
Copy link
Collaborator

adgad commented Aug 14, 2018

The threshold was introduced to accomodate for "random errors" (e.g. browserstack queue full, or random puppeteer timeouts), so as not to get noisy tests.

However it's not a great solution. In cases like https://circleci.com/workflow-run/fa1e2bc6-450b-4c3c-b648-40f51f0580b1, there's only one smoke tests, and it "errored" for a genuine problem - however the tests passed because it was within the threshold.

A better solution would catch and handle specific errors (like the browserstack queue one) and handle them specifically. For example, we could retry tests if it's a queue timeout. Or have specific types of errors that we do allow to fail. And fail for all other errors.

@taktran
Copy link

taktran commented Nov 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants