Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix for flaky filestream parser tests #31773

Merged
merged 2 commits into from
May 31, 2022
Merged

Fix for flaky filestream parser tests #31773

merged 2 commits into from
May 31, 2022

Conversation

cmacknz
Copy link
Member

@cmacknz cmacknz commented May 28, 2022 •
edited
Loading

What does this PR do?

Attempts to fix flaky filestream parser tests.

Poll on the registry having the expected state instead of just looking once after the expected number of events were propagated through the pipeline. Fixes the flakiness that could be reproduced locally.

Most of the tests had a stack trace that looked like this, expecting the registry to be in a particular state at the end of the test:

=== RUN   TestParsersMultilineMaxBytes
    parsers_integration_test.go:439: 
        	Error Trace:	environment_test.go:211
        	            				parsers_integration_test.go:439
        	Error:      	Not equal: 
        	            	expected: 1275
        	            	actual  : 1158
        	Test:       	TestParsersMultilineMaxBytes
--- FAIL: TestParsersMultilineMaxBytes (3.00s)

Why is it important?

Flaky tests suck.

Checklist

  • [ x ] My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run the filestream integration tests continuously, long enough to convince yourself they aren't going to fail later. I found the TestParsersCStyleLog regularly failed on my machine and it was quick to run, so I mostly run this to validate: while go test -v -tags integration -count 1 ./input/filestream/... -run TestParsersCStyleLog; do :; done

You can also run the entire set of filestream tests but it will take longer for each run.

Related issues

@cmacknz cmacknz added flaky-test Unstable or unreliable test cases. Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels May 28, 2022
@cmacknz cmacknz requested review from belimawr and kvch May 28, 2022 00:09
@cmacknz cmacknz requested a review from a team as a code owner May 28, 2022 00:09
@cmacknz cmacknz self-assigned this May 28, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 28, 2022
@cmacknz cmacknz changed the title WIP fix flaky filestream tests. Fix for flaky filestream parser tests May 28, 2022
@cmacknz
Attempt to fix flaky filestream parser tests.
bfb560d 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.
@cmacknz cmacknz force-pushed the fix-flaky-filebeat-crawler-tests branch from bb24676 to bfb560d Compare May 28, 2022 00:13
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 28, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-31T15:34:13.238+0000

  • Duration: 73 min 50 sec

Test stats 🧪

Test Results
Failed 0
Passed 6258
Skipped 737
Total 6995

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@cmacknz
Copy link
Member Author

cmacknz commented May 30, 2022

/test

@cmacknz
Copy link
Member Author

cmacknz commented May 31, 2022

@Mergifyio update

@mergify
Copy link
Contributor

mergify bot commented May 31, 2022

update

✅ Branch has been successfully updated

@cmacknz cmacknz merged commit b502a78 into elastic:main May 31, 2022
@cmacknz cmacknz deleted the fix-flaky-filebeat-crawler-tests branch May 31, 2022 19:40
@cmacknz cmacknz added backport-v8.3.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify labels Jun 6, 2022
@mergify mergify bot mentioned this pull request Jun 6, 2022
Merged
mergify bot pushed a commit that referenced this pull request Jun 6, 2022
@cmacknz @mergify
Attempt to fix flaky filestream parser tests. (#31773)
ceff280 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)
mergify bot pushed a commit that referenced this pull request Jun 6, 2022
@cmacknz @mergify
Attempt to fix flaky filestream parser tests. (#31773)
dd6f61f 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)
@mergify mergify bot mentioned this pull request Jun 6, 2022
Merged
cmacknz added a commit that referenced this pull request Jun 6, 2022
@mergify @cmacknz
Attempt to fix flaky filestream parser tests. (#31773) (#31842)
27f60c9 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)

Co-authored-by: Craig MacKenzie <[email protected]>
cmacknz added a commit that referenced this pull request Jun 6, 2022
@mergify @cmacknz
Attempt to fix flaky filestream parser tests. (#31773) (#31841)
3376681 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)

Co-authored-by: Craig MacKenzie <[email protected]>
@cmacknz cmacknz added the backport-7.17 Automated backport to the 7.17 branch with mergify label Jul 6, 2022
@cmacknz cmacknz mentioned this pull request Jul 6, 2022
Closed
@mergify mergify bot mentioned this pull request Jul 6, 2022
Merged
mergify bot pushed a commit that referenced this pull request Jul 6, 2022
@cmacknz @mergify
Attempt to fix flaky filestream parser tests. (#31773)
0e45815 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)
cmacknz added a commit that referenced this pull request Jul 6, 2022
@mergify @cmacknz
Attempt to fix flaky filestream parser tests. (#31773) (#32234)
db11ce9 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
(cherry picked from commit b502a78)

Co-authored-by: Craig MacKenzie <[email protected]>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@cmacknz @mergify @chrisberkhout
Attempt to fix flaky filestream parser tests. (#31773)
4c651a1 
Poll on the registry having the expected state instead of just looking
once after the expected number of events were propagated through the
pipeline. Fixes the flakiness that could be reproduced locally.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kvch kvch kvch approved these changes

@belimawr belimawr belimawr approved these changes

Assignees

@cmacknz cmacknz

Labels
backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.2.0 Automated backport with mergify backport-v8.3.0 Automated backport with mergify flaky-test Unstable or unreliable test cases. Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Meta] Filestream Go integration tests are flaky
4 participants
@cmacknz @elasticmachine @belimawr @kvch