-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Allow exporting of prebuilt rules via the API #194498
Conversation
These mainly fail because we don't include prebuilt rules in export.
We're just threading the feature flag through, here; nothing fancy. It's not great, but it'll get things green while we think about a refactor.
Tests are green (after some small modification), although this should also affect the bulk export route, which I've not added tests for. Will do that next. Then: on to refactoring?
And adds an additional test for the equivalent bulk_actions call.
/ci |
/ci |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @rylnd |
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM thanks @rylnd! Tested the following scenarios using CURL commands:
- Single rule export of custom rule
- Single rule export of non-customized prebuilt rule
- Single rule export of customized prebuilt rule
- Bulk rule export of each individual rule type
- Bulk rule export of non-customized prebuilt rules and custom rules
- Bulk rule export of customized prebuilt rule, non-customized prebuilt rule, and custom rule
if (prebuiltRulesCustomizationEnabled) { | ||
rulesCount = await getRulesCount({ | ||
rulesClient, | ||
filter: '', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does an empty filter string just mean fetch everything?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep! It's a required argument and we typically call it with a predefined filter for mutable rules, but as this PR is extending that functionality this is a consequence of that. I could make it optional and default to ''
if it's not specified, if you prefer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nah, it's a bit wonky looking but easy enough to understand what it's doing - I was just curious.
💚 Build Succeeded
Metrics [docs]
History
cc @rylnd |
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/11354645633 |
…astic#194498) ## Summary This PR introduces the backend functionality necessary to export prebuilt rules via our existing export APIs: 1. Export Rules - POST /rules/_export 2. Bulk Actions - POST /rules/_bulk_action The [Prebuilt Rule Customization RFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md) goes into detail, and the export-specific issue is described [here](elastic#180167 (comment)). ## Steps to Review 1. Enable the Feature Flag: `prebuiltRulesCustomizationEnabled` 1. Install the prebuilt rules package via fleet 1. Install some prebuilt rules, and obtain a prebuilt rule's `rule_id`, e.g. `ac8805f6-1e08-406c-962e-3937057fa86f` 1. Export the rule via the export route, e.g. (in Dev Tools): POST kbn:api/detection_engine/rules/_export Note that you may need to use the CURL equivalent for these requests, as the dev console does not seem to handle file responses: curl --location --request POST 'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson' \ --header 'kbn-xsrf: true' \ --header 'elastic-api-version: 2023-10-31' \ --header 'Authorization: Basic waefoijawoefiajweo==' 1. Export the rule via bulk actions, e.g. (in Dev Tools): POST kbn:api/detection_engine/rules/_bulk_action { "action": "export" } 1. Observe that the exported rules' fields are correct, especially `rule_source` and `immutable` (see tests added here for examples). ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit b67bd83)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…PI (#194498) (#196447) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Allow exporting of prebuilt rules via the API (#194498)](#194498) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ryland Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-15T21:26:25Z","message":"[Security Solution] Allow exporting of prebuilt rules via the API (#194498)\n\n## Summary\r\n\r\nThis PR introduces the backend functionality necessary to export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export Rules - POST /rules/_export \r\n2. Bulk Actions - POST /rules/_bulk_action \r\n\r\nThe [Prebuilt Rule Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes into detail, and the export-specific issue is described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n## Steps to Review\r\n1. Enable the Feature Flag: `prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules package via fleet \r\n1. Install some prebuilt rules, and obtain a prebuilt rule's `rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the export route, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to use the CURL equivalent for these requests, as\r\nthe dev console does not seem to handle file responses:\r\n\r\ncurl --location --request POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n --header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version: 2023-10-31' \\\r\n --header 'Authorization: Basic waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\": \"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are correct, especially\r\n`rule_source` and `immutable` (see tests added here for examples).\r\n\r\n### Checklist\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","v8.16.0","backport:version"],"title":"[Security Solution] Allow exporting of prebuilt rules via the API","number":194498,"url":"https://github.com/elastic/kibana/pull/194498","mergeCommit":{"message":"[Security Solution] Allow exporting of prebuilt rules via the API (#194498)\n\n## Summary\r\n\r\nThis PR introduces the backend functionality necessary to export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export Rules - POST /rules/_export \r\n2. Bulk Actions - POST /rules/_bulk_action \r\n\r\nThe [Prebuilt Rule Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes into detail, and the export-specific issue is described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n## Steps to Review\r\n1. Enable the Feature Flag: `prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules package via fleet \r\n1. Install some prebuilt rules, and obtain a prebuilt rule's `rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the export route, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to use the CURL equivalent for these requests, as\r\nthe dev console does not seem to handle file responses:\r\n\r\ncurl --location --request POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n --header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version: 2023-10-31' \\\r\n --header 'Authorization: Basic waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\": \"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are correct, especially\r\n`rule_source` and `immutable` (see tests added here for examples).\r\n\r\n### Checklist\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194498","number":194498,"mergeCommit":{"message":"[Security Solution] Allow exporting of prebuilt rules via the API (#194498)\n\n## Summary\r\n\r\nThis PR introduces the backend functionality necessary to export\r\nprebuilt rules via our existing export APIs:\r\n\r\n1. Export Rules - POST /rules/_export \r\n2. Bulk Actions - POST /rules/_bulk_action \r\n\r\nThe [Prebuilt Rule Customization\r\nRFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes into detail, and the export-specific issue is described\r\n[here](https://github.com/elastic/kibana/issues/180167#issue-2227974379).\r\n\r\n\r\n## Steps to Review\r\n1. Enable the Feature Flag: `prebuiltRulesCustomizationEnabled`\r\n1. Install the prebuilt rules package via fleet \r\n1. Install some prebuilt rules, and obtain a prebuilt rule's `rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n1. Export the rule via the export route, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_export\r\n \r\nNote that you may need to use the CURL equivalent for these requests, as\r\nthe dev console does not seem to handle file responses:\r\n\r\ncurl --location --request POST\r\n'http://localhost:5601/api/detection_engine/rules/_export?exclude_export_details=true&file_name=exported_rules.ndjson'\r\n\\\r\n --header 'kbn-xsrf: true' \\\r\n --header 'elastic-api-version: 2023-10-31' \\\r\n --header 'Authorization: Basic waefoijawoefiajweo=='\r\n\r\n1. Export the rule via bulk actions, e.g. (in Dev Tools):\r\n\r\n POST kbn:api/detection_engine/rules/_bulk_action\r\n {\r\n \"action\": \"export\"\r\n }\r\n \r\n1. Observe that the exported rules' fields are correct, especially\r\n`rule_source` and `immutable` (see tests added here for examples).\r\n\r\n### Checklist\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"b67bd83ea93909d809206b1004c306a11fd8ee3f"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ryland Herrick <[email protected]>
…191116) ## Summary This PR introduces test plans for both [Prebuilt Rule Import](#180168) (corresponding [PR](#190198)) and [Prebuilt Rule Export](#180167) (corresponding [PR](#194498)). Import is considerably more complicated as it is calculating new values (for `rule_source`, `immutable`), while the export work is mainly removing existing restrictions (which allowed only custom rules to be exported). --------- Co-authored-by: Elastic Machine <[email protected]>
…lastic#191116) ## Summary This PR introduces test plans for both [Prebuilt Rule Import](elastic#180168) (corresponding [PR](elastic#190198)) and [Prebuilt Rule Export](elastic#180167) (corresponding [PR](elastic#194498)). Import is considerably more complicated as it is calculating new values (for `rule_source`, `immutable`), while the export work is mainly removing existing restrictions (which allowed only custom rules to be exported). --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit e429849)
…ort (#191116) (#199716) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Test plans for prebuilt rule import and export (#191116)](#191116) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ryland Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-11T21:01:46Z","message":"[Security Solution] Test plans for prebuilt rule import and export (#191116)\n\n## Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt Rule\r\nImport](#180168) (corresponding\r\n[PR](#190198)) and [Prebuilt Rule\r\nExport](#180167) (corresponding\r\n[PR](#194498)). Import is\r\nconsiderably more complicated as it is calculating new values (for\r\n`rule_source`, `immutable`), while the export work is mainly removing\r\nexisting restrictions (which allowed only custom rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.17.0"],"title":"[Security Solution] Test plans for prebuilt rule import and export","number":191116,"url":"https://github.com/elastic/kibana/pull/191116","mergeCommit":{"message":"[Security Solution] Test plans for prebuilt rule import and export (#191116)\n\n## Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt Rule\r\nImport](#180168) (corresponding\r\n[PR](#190198)) and [Prebuilt Rule\r\nExport](#180167) (corresponding\r\n[PR](#194498)). Import is\r\nconsiderably more complicated as it is calculating new values (for\r\n`rule_source`, `immutable`), while the export work is mainly removing\r\nexisting restrictions (which allowed only custom rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191116","number":191116,"mergeCommit":{"message":"[Security Solution] Test plans for prebuilt rule import and export (#191116)\n\n## Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt Rule\r\nImport](#180168) (corresponding\r\n[PR](#190198)) and [Prebuilt Rule\r\nExport](#180167) (corresponding\r\n[PR](#194498)). Import is\r\nconsiderably more complicated as it is calculating new values (for\r\n`rule_source`, `immutable`), while the export work is mainly removing\r\nexisting restrictions (which allowed only custom rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ryland Herrick <[email protected]>
…lastic#191116) ## Summary This PR introduces test plans for both [Prebuilt Rule Import](elastic#180168) (corresponding [PR](elastic#190198)) and [Prebuilt Rule Export](elastic#180167) (corresponding [PR](elastic#194498)). Import is considerably more complicated as it is calculating new values (for `rule_source`, `immutable`), while the export work is mainly removing existing restrictions (which allowed only custom rules to be exported). --------- Co-authored-by: Elastic Machine <[email protected]>
Summary
This PR introduces the backend functionality necessary to export prebuilt rules via our existing export APIs:
The Prebuilt Rule Customization RFC goes into detail, and the export-specific issue is described here.
Steps to Review
Enable the Feature Flag:
prebuiltRulesCustomizationEnabled
Install the prebuilt rules package via fleet
Install some prebuilt rules, and obtain a prebuilt rule's
rule_id
, e.g.ac8805f6-1e08-406c-962e-3937057fa86f
Export the rule via the export route, e.g. (in Dev Tools):
Note that you may need to use the CURL equivalent for these requests, as the dev console does not seem to handle file responses:
Export the rule via bulk actions, e.g. (in Dev Tools):
Observe that the exported rules' fields are correct, especially
rule_source
andimmutable
(see tests added here for examples).Checklist
For maintainers