Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[notifications] flaky test analyser #753

Closed
v1v opened this issue Oct 6, 2020 · 2 comments
Closed

[notifications] flaky test analyser #753

v1v opened this issue Oct 6, 2020 · 2 comments

Comments

@v1v
Copy link
Member

v1v commented Oct 6, 2020

We would like to provide a flaky test analyser reporting as a GH comment for all the PRs for a given project.

For such, the automation in place should raise an issue for each flaky test match, unless there is an existing issue for that particular flaky test.

The notification should always be reported even when there are test failures without a flaky test, this will always help to audit and track every PR.

What are the components that are required?

  • Component to look up for any open issues that match the given flaky test.
    • This component will return a dictionary with all the flaky tests and their GH issues if any or empty if nothing.
  • Component to create issues for the given flaky test if it does not exist. There will be a template issue.
  • Component to decorate the GH comment with the given list of flaky tests and their issues (the ones that already exist or the new ones that were created.

How does it work?

notifyBuildResult step will be the wrapper with the arguments analyzeFlakey

@v1v
Copy link
Member Author

v1v commented Oct 6, 2020

The below are the examples notifications we will use:

image

🐛 Flaky test report

❕ There are not known flaky tests.

And issues creations with the below format:

image

@v1v v1v mentioned this issue Oct 6, 2020
3 tasks
@v1v v1v closed this as completed Oct 30, 2020
@v1v
Copy link
Member Author

v1v commented Oct 30, 2020

See elastic/beats#21853

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant