[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

jpdjere · 2022-12-05T18:34:12Z

Addresses: #130966
Based on: #135127

Summary

This PR deprecates the Sidecar SO of type siem-detection-engine-rule-execution-info in favour of storing Rule Execution Logging data within the rule itself, making use of the work previously done in the Alerting Framework:

Work done:

Pass execution statuses and metrics from rule executors to the Framework: through the use of RuleMonitoringService and RuleResultService from within the rule execution log client for executor. x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts
Fetch execution statuses and metrics from rules themselves instead of the sidecar siem-detection-engine-rule-execution-info saved objects: through the use of the new function createRuleExecutionSummary in x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/create_rule_execution_summary.ts, which extracts last execution information from the rule itself.
Remove the siem-detection-engine-rule-execution-info saved objects type from the codebase. Mark it as deleted in Kibana Core: added siem-detection-engine-rule-execution-info to packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/unused_types.ts; and got rid of the related Saved Object client.
Make sure to keep backward compatibility in the Detection API endpoints and rule execution events we write into the Event Log: API compatibility is maintained. No breaking changes.

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

x-pack/plugins/alerting/server/lib/last_run_status.ts

elasticmachine · 2023-01-22T13:55:17Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2023-01-22T13:55:17Z

Pinging @elastic/security-solution (Team: SecuritySolution)

banderror

Awesome, thanks for the fixes @maximpn! 🚀
Just one nit here for your consideration: #147035 (comment)

…on rules

…-fix'

kibana-ci · 2023-01-27T14:40:22Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b54f645

Failed CI Steps

Security Solution Tests #3

Test Failures

[job] [logs] Security Solution Tests #3 / Timelines Creates a timeline by clicking untitled timeline from bottom bar can be added notes

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	12.8MB	12.8MB	+364.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`siem-detection-engine-rule-execution-info`	11	-	-11

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`securitySolution`	386	384	-2

History

💔 Build #103329 failed 8f0a58c
💔 Build #103312 failed 01b52fe
💛 Build #102091 was flaky 24c831c
💚 Build #101639 succeeded ef3d69c
💔 Build #101615 failed 9d60a59

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn @jpdjere

## Summary It fixes a problem of exporting `execution_summary` field while exporting detection rules which was introduce in #147035. Presence of that field make importing of just exported rule failing. Tests to cover this fix will come in a separate PR.

…150097) ## Summary It fixes a problem of exporting `execution_summary` field while exporting detection rules which was introduce in elastic#147035. Presence of that field make importing of just exported rule failing. Tests to cover this fix will come in a separate PR.

…ead of saved object (elastic#147035) **Addresses:** elastic#130966 **Based on:** elastic#135127 ## Summary This PR deprecates the Sidecar SO of type `siem-detection-engine-rule-execution-info` in favour of storing Rule Execution Logging data within the rule itself, making use of the work previously done in the Alerting Framework: - elastic#140882 - elastic#147278 Work done: - **Pass execution statuses and metrics from rule executors to the Framework:** through the use of `RuleMonitoringService` and `RuleResultService` from within the rule execution log client for executor. `x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts` - **Fetch execution statuses and metrics from rules themselves instead of the sidecar `siem-detection-engine-rule-execution-info` saved objects**: through the use of the new function `createRuleExecutionSummary` in `x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/create_rule_execution_summary.ts`, which extracts last execution information from the rule itself. - **Remove the siem-detection-engine-rule-execution-info saved objects type from the codebase. Mark it as deleted in Kibana Core:** added `siem-detection-engine-rule-execution-info` to `packages/core/saved-objects/core-saved-objects-migration-server-internal/src/core/unused_types.ts`; and got rid of the related Saved Object client. - **Make sure to keep backward compatibility in the Detection API endpoints and rule execution events we write into the Event Log**: API compatibility is maintained. No breaking changes. ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…150097) ## Summary It fixes a problem of exporting `execution_summary` field while exporting detection rules which was introduce in elastic#147035. Presence of that field make importing of just exported rule failing. Tests to cover this fix will come in a separate PR.

jpdjere force-pushed the expand-rule-monitoring-security-solution branch from 931279d to c0407b1 Compare December 5, 2022 19:21

jpdjere added the ci:no-auto-commit Disable auto-committing changes on CI label Dec 6, 2022

jpdjere force-pushed the expand-rule-monitoring-security-solution branch from fb7c12f to 604ec4a Compare December 6, 2022 11:29

jpdjere mentioned this pull request Dec 7, 2022

[ResponseOps] API for reading/writing rule execution summary #135127

Closed

banderror assigned jpdjere Dec 12, 2022

jpdjere force-pushed the expand-rule-monitoring-security-solution branch from c09ed3e to 1ca9b7f Compare December 14, 2022 12:34

jpdjere force-pushed the expand-rule-monitoring-security-solution branch 2 times, most recently from 31a7159 to 9718858 Compare December 22, 2022 15:01

jpdjere commented Dec 29, 2022

View reviewed changes

x-pack/plugins/alerting/server/lib/last_run_status.ts Outdated Show resolved Hide resolved

jpdjere changed the title ~~[Security Solution] Expand RuleMonitoringService~~ [Security Solution] Write and read Rule Execution Logs from rule instead of saved object Dec 29, 2022

jpdjere force-pushed the expand-rule-monitoring-security-solution branch from 7239852 to 2928e87 Compare December 29, 2022 16:29

jpdjere removed the ci:no-auto-commit Disable auto-committing changes on CI label Dec 29, 2022

maximpn force-pushed the expand-rule-monitoring-security-solution branch from 8318023 to 95dc24d Compare January 16, 2023 16:31

banderror assigned maximpn Jan 19, 2023

maximpn force-pushed the expand-rule-monitoring-security-solution branch from a38fcd4 to 9d60a59 Compare January 20, 2023 14:28

maximpn marked this pull request as ready for review January 22, 2023 13:53

maximpn requested review from a team as code owners January 22, 2023 13:53

maximpn requested a review from spong January 22, 2023 13:53

maximpn added backport:skip This commit does not require backporting release_note:feature Makes this part of the condensed release notes labels Jan 22, 2023

jpdjere and others added 14 commits January 27, 2023 11:51

Fix test

fc35960

Lint fix

67cbc12

set always outcome message

1e4a351

remove message type prefixes

ccbec38

revert last_run_status changes

bf65057

do not add an end line period

29a9073

take into account rule's 'running' status

ba28aae

fix unit tests

eec24ed

fix functional tests

79b66a6

remove TBC acronym

ed7965d

remove unused function

d319580

add invariant checks for ruleMonitoringService and ruleResultService

2b441c9

improve function's naming

960b788

rollback StatusChangeArgsWithMessage to StatusChangeArgs

01b52fe

maximpn force-pushed the expand-rule-monitoring-security-solution branch from 24c831c to 01b52fe Compare January 27, 2023 11:30

banderror approved these changes Jan 27, 2023

View reviewed changes

make ruleMonitoringService and ruleResultService required for detecti…

8f0a58c

…on rules

maximpn force-pushed the expand-rule-monitoring-security-solution branch from 53466b1 to 8f0a58c Compare January 27, 2023 12:29

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

b54f645

…-fix'

maximpn merged commit 6cc9855 into elastic:main Jan 27, 2023

banderror mentioned this pull request Jan 27, 2023

[Security Solution] Pass rule execution statuses and metrics to Alerting Framework #130966

Closed

4 tasks

maximpn mentioned this pull request Feb 1, 2023

[Security Solution] Avoid exporting execution_summary field #150097

Merged

spong mentioned this pull request Feb 2, 2023

[Security Solution][Platform] - Add connectors to import/export API #148703

Merged

7 tasks

spong mentioned this pull request Apr 24, 2023

[Security Solution] Failed rule execution shows 0 for index duration even though alerts were written #155672

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

jpdjere commented Dec 5, 2022 •

edited by banderror

Loading

elasticmachine commented Jan 22, 2023

elasticmachine commented Jan 22, 2023

banderror left a comment

kibana-ci commented Jan 27, 2023

References to deprecated APIs

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

[Security Solution] Write and read Rule Execution Logs from rule instead of saved object #147035

Conversation

jpdjere commented Dec 5, 2022 • edited by banderror Loading

Summary

Checklist

elasticmachine commented Jan 22, 2023

elasticmachine commented Jan 22, 2023

banderror left a comment

Choose a reason for hiding this comment

kibana-ci commented Jan 27, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Saved Objects .kibana field count

References to deprecated APIs

History

jpdjere commented Dec 5, 2022 •

edited by banderror

Loading