Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.2](backport #31464) {,x-pack/}winlogbeat: resurrect tests for event processing #31491

Merged
merged 3 commits into from
May 3, 2022

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented May 3, 2022

This is an automatic backport of pull request #31464 done by Mergify.
Cherry-pick of f22abe3 has failed:

On branch mergify/bp/8.2/pr-31464
Your branch is up to date with 'origin/8.2'.

You are currently cherry-picking commit f22abe3b60.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   winlogbeat/eventlog/wineventlog.go
	new file:   x-pack/winlogbeat/module/powershell/test/powershell_windows_test.go
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/400.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/403.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/4103.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/4104.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/4105.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/4106.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/600.evtx.golden.json
	modified:   x-pack/winlogbeat/module/powershell/test/testdata/800.evtx.golden.json
	new file:   x-pack/winlogbeat/module/security/test/security_windows_test.go
	modified:   x-pack/winlogbeat/module/security/test/testdata/1100.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/1102.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/1104.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/1105.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4670_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4706_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4707_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4713_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4716_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4717_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4718_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4719.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4719_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4739_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4741.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4742.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4743.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4744.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4745.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4746.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4747.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4748.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4749.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4750.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4751.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4752.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4753.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4759.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4760.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4761.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4762.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4763.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4817_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4902_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4904_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4905_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4906_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4907_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/4908_WindowsSrv2016.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4673.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4674.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4697.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4698.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4699.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4700.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4701.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4702.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4768.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4769.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4770.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4771.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4776.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4778.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012_4779.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2012r2-logon.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016-4672.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016-logoff.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4720_Account_Created.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4722_Account_Enabled.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4723_Password_Change.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4724_Password_Reset.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4725_Account_Disabled.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4726_Account_Deleted.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4727.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4728.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4729.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4730.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4731.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4732.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4733.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4734.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4735.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4737.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4738_Account_Changed.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4740_Account_Locked_Out.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4754.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4755.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4756.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4757.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4758.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4764.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4767_Account_Unlocked.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4781_Account_Renamed.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4798.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4799.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2016_4964.evtx.golden.json
	modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4689_Process_Exited.evtx.golden.json
	new file:   x-pack/winlogbeat/module/sysmon/test/sysmon_windows_test.go
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-10.2-dns.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-11-filedelete.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-11-filedeletedetected.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-11-registry.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-12-loadimage.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-12-processcreate.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-13-clipboardchange.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-13-processtampering.evtx.golden.json
	modified:   x-pack/winlogbeat/module/sysmon/test/testdata/sysmon-9.01.evtx.golden.json

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
	both modified:   x-pack/winlogbeat/module/testing_windows.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.

(cherry picked from commit f22abe3)

# Conflicts:
#	x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
#	x-pack/winlogbeat/module/testing_windows.go
@mergify mergify bot requested a review from a team as a code owner May 3, 2022 01:20
@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels May 3, 2022
@mergify mergify bot assigned efd6 May 3, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 3, 2022
@botelastic
Copy link

botelastic bot commented May 3, 2022

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented May 3, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-03T02:05:20.704+0000

  • Duration: 47 min 59 sec

Test stats 🧪

Test Results
Failed 0
Passed 930
Skipped 9
Total 939

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@efd6 efd6 merged commit 7b90207 into 8.2 May 3, 2022
@mergify mergify bot deleted the mergify/bp/8.2/pr-31464 branch May 3, 2022 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants