Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Allow importing of prebuilt rules via the API #190198

Merged
merged 143 commits into from
Oct 16, 2024
Merged

[Security Solution] Allow importing of prebuilt rules via the API #190198

merged 143 commits into from
Oct 16, 2024

Conversation

rylnd
Copy link
Contributor

@rylnd rylnd commented Aug 8, 2024 •
edited by kibanamachine
Loading

Summary

This PR introduces the backend functionality necessary to import prebuilt rules via our existing import API. The RFC goes into detail, and the import-specific issue is described here, but at a high level we're adding two things in this PR:

  1. The logic to calculate the new rule_source field on import, which contains the information about the rule's relationship to existing prebuilt rules.
  2. A new code path for importing rules, which respects the calculated rule_source field.

In order to maintain backwards compatibility with the existing import logic, and because the existing import implementation is hard to modify/extend, I opted to add a top-level switch on the feature flag in the import route itself, which calls either the existing import function (now named importRulesLegacy), or the new function, importRules, which ultimately calls the new DetectionRulesClient method, importRules. Neither knows about the feature flag, and thanks to great suggestions from @xcrzx there are nice, clean boundaries between the import functions and the client methods.

I went down the path of trying to write the new import code by reusing the outer importRules function, but after discussion with the team we decided it was simplest to simply bifurcate the code at that point, so that we have:

  1. The legacy import code, which:
    • only supports custom rules (immutable: false)
    • accepts version as an optional parameter
    • calculates a legacy rule_source value based on the immutable field
  2. The new import code, which
    • Installs the prebuilt rules assets as necessary
    • Accepts both kinds of rules (immutable: true/false)
    • Requires version as a parameter for prebuilt rules
    • Allows version to be optional for custom rules
    • calculates a proper rule_source (and immutable)

Deprecation of immutable

The RFC (and thus I) had initially thought that we'd be deprecating the immutable field as part of this PR/Epic. However, after discussion we have opted to continue supporting immutable until such time as we can drop it, compatibility-wise.

Steps to Review

  1. Enable the Feature Flag: prebuiltRulesCustomizationEnabled

  2. Install the prebuilt rules package via fleet

  3. Create and export a custom rule to serve as a template (or download one:

     curl -L -H 'Authorization: 8eef0fe5d7dfc52b' -o 'rules_export (1).ndjson' https://upload.elastic.co/d/4693e7fe4356ce7bcf7b7d6b72881a9fd189730c61bf5ef47c9930458d746979

    )

  4. Install some prebuilt rules, and obtain a prebuilt rule's rule_id, e.g. ac8805f6-1e08-406c-962e-3937057fa86f

  5. Edit the rules_export.ndjson to contain only the first line, and modify the rule_id with the prebuilt rule's

  6. Import rules_export.ndjson and then retrieve the rule via the Dev Console:

     GET kbn:api/detection_engine/rules?rule_id=ac8805f6-1e08-406c-962e-3937057fa86f

  7. Observe that the rule has the expected rule_source and immutable values

  8. Test other permutations of import by modifying rules_export.ndjson and re-importing; see (the test plan as well as a reference table of scenarios)

Checklist

Delete any items that are not applicable to this PR.

rylnd added 16 commits August 5, 2024 23:06
@rylnd
WIP
f18348e
@rylnd
Allow import endpoint to specify the immutable property
ea65e92 
This will be necessary for importing prebuilt rules without having to
modify their contents.
@rylnd
Assert our requirement of rule_id
179c2b4 
This behavior was already there, we're just pinning it down.
@rylnd
Style: better assertion syntax
6360b05 
Instead of nested uses of objectContaining with toEqual, we can instead
use toMatchObject, which accomplishes the same.
@rylnd
I've decided that we don't need this type
0900106 
Since we only reference it in legacy validation.
@rylnd
Test our allowing of the rule_source param
816fcfe 
Like immutable, it's dropped, but we still support it being sent (so
that users can send their prebuilt rules).
@rylnd
Add some explanatory structure to our tests
70b53c1
@rylnd
Remove our optional field from the schema entirely
7e531bb 
This has the same effect: any value is allowed, but the field is
ultimately ignored in the handling routes.
@rylnd
Style: removing trailing whitespace
41b86b6
@rylnd
Mark our immutable property as optional
aaaf160 
Our rule types may technically have this.
@rylnd
Update rule schemas per RFC
a34fad5
@rylnd
Add an example datetime string to our openAPI schema
616b8d8 
As a user I always appreciate an actual example (for type, formatting,
etc.)
@rylnd
Add our new, optional field to prebuilt asset schema
7a16826 
This will be added to the detection-rules repo as they modify rules
moving forward.
@rylnd
Reject prebuilt rule import while feature flag is disabled
cbeef5c 
In order to prevent users from importing prebuilt rules before the
feature is fully enabled, we need to add additional validation logic to
account for our looser schema validation (which allows any value for
`immutable`).
@rylnd
More comprehensive test
8dd304f 
That actually asserts that our specified rule_source was ignored.
@rylnd
WIP: ensuring prebuilt rules package is installed when importing rules
28fc492 
The logic here can/should be optimized, but this is mostly just
leveraging existing helpers to do this work.
@rylnd rylnd added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area 8.16 candidate labels Aug 8, 2024
@rylnd rylnd self-assigned this Aug 8, 2024
@rylnd
Copy link
Contributor Author

rylnd commented Aug 8, 2024

/ci

@rylnd
Revert making a base rule param (immutable) optional
b5e36aa 
It would be a breaking change to make this param optional. Since we've
discussed keeping it around but deprecating it, we don't really need to
change this type, and need to instead ensure that we create this field
if left unspecified.
@rylnd
Default immutable to false in our importing path
c7fc008 
This change to the RuleToImport schema affects all rules being imported,
as they're validated by this schema. This ensures we always generate a
value for the field, which is needed for backwards compatibility.

Associated unit tests were similarly updated.
@rylnd
Copy link
Contributor Author

rylnd commented Aug 9, 2024

/ci

@rylnd rylnd force-pushed the rylnd/prebuilt_rule_import branch from 3799027 to b236c84 Compare October 9, 2024 01:13
rylnd and others added 9 commits October 9, 2024 21:45
@rylnd @banderror
Update x-pack/plugins/security_solution/server/lib/detection_engine/r…
2838173 
…ule_management/logic/detection_rules_client/converters/normalize_rule_params.ts

Co-authored-by: Georgii Gorbachev <[email protected]>
@rylnd
Add additional test case for preserving multiple errors per rule
86bee40
@rylnd
Removing general types.ts in favor of a declarative interface file
6f570fa 
This also makes it more obvious that the `RuleSpecifier` type should be
placed with the utility functions that expect them as arguments.
@rylnd
Simplify interface of RuleSourceImporter
52e1117 
Setup's single argument doesn't need to be a named key in an object.
@rylnd
Inline our utils.ts functions
e926f4a 
Since a "utility" file is prone to collect random things, and these
functions are all specific to the importer itself, we're now inlining
them into the same file as the class, until such time as they need to be
reused.
@rylnd
Merge branch 'main' into rylnd/prebuilt_rule_import
4f52f6e 
 Conflicts:
	x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/converters/convert_rule_response_to_alerting_rule.ts
	x-pack/plugins/security_solution/server/lib/detection_engine/rule_management/logic/detection_rules_client/converters/type_specific_camel_to_snake.ts
@rylnd
Add proper defaults to rule during import
6062161 
Since `calculateIsCustomized` will fail if the provided rule is not a
valid RuleResponse, we were experiencing failures (uncovered by
integration tests) due to a improper conversion of incoming rules.

I _believe_ that I've fixed this by:

1. Defining an explicit converter utility for this case: RuleToImport ->
   RuleResponse
2. Calling `applyRuleDefaults` within this utility

`applyRuleDefaults` is what, in the normal import code path, adds these
missing/default fields. My assumption is that this will cover any other
issues with conversion to a `RuleResponse`, which will ideally be
validated by the Rules Management team.

The same issue still exists for the previously-used converter utility,
but I'm going to tackle that in a separate commit.
@rylnd
Define RuleToImport mocks with minimal fields
3feb693 
We don't want to specify `language` here since it's not a required
field, and their presence could actually mask bugs with defaulting logic
throughout our test suite.

I believe there were just a few tests that were (unnecessarily)
dependent on those fields; everything else luckily just works.
@rylnd
Fix bug converting a prebuilt rule asset to a rule response
4ef7a39 
If a prebuilt asset is lacking a 'language' field, this conversion
function would previously throw an error due to us not providing a
default for it.

This was fixed by calling `applyRuleDefaults` within this function,
which made some other logic in this function redundant (and so it was
removed).
@rylnd
Copy link
Contributor Author

rylnd commented Oct 11, 2024

/ci

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #61 / saved objects tagging - functional tests visualize integration editing allows to assign tags to an existing visualization

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 20.6MB 20.6MB -96.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 549 551 +2

Total ESLint disabled count

id before after diff
securitySolution 634 636 +2

History

cc @rylnd

@rylnd rylnd merged commit c0b1301 into elastic:main Oct 16, 2024
41 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11374033325

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 16, 2024
@rylnd
[Security Solution] Allow importing of prebuilt rules via the API (el…
7015873 
…astic#190198)

## Summary

This PR introduces the backend functionality necessary to import
prebuilt rules via our existing import API. The
[RFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)
goes into detail, and the import-specific issue is described
[here](elastic#180168), but at a high
level we're adding two things in this PR:

1. The logic to calculate the new `rule_source` field on import, which
contains the information about the rule's relationship to existing
prebuilt rules.
2. A new code path for importing rules, which respects the calculated
`rule_source` field.

In order to maintain backwards compatibility with the existing import
logic, and because the existing import implementation is hard to
modify/extend, I opted to add a top-level switch on the feature flag in
the import route itself, which calls either the existing import function
(now named `importRulesLegacy`), or the new function, `importRules`,
which ultimately calls the new `DetectionRulesClient` method,
`importRules`. Neither knows about the feature flag, and thanks to great
suggestions from @xcrzx there are nice, clean boundaries between the
import functions and the client methods.

I went down the path of trying to write the new import code by reusing
the outer `importRules` function, but after discussion with the team we
decided it was simplest to simply bifurcate the code at that point, so
that we have:

1. The legacy import code, which:
    * only supports custom rules (`immutable: false`)
    * accepts `version` as an optional parameter
* calculates a legacy `rule_source` value based on the `immutable` field
2. The new import code, which
    * Installs the prebuilt rules assets as necessary
    * Accepts both kinds of rules (`immutable: true/false`)
    * Requires `version` as a parameter for _prebuilt_ rules
    * Allows `version` to be optional for custom rules
    * calculates a proper `rule_source` (and `immutable`)

### Deprecation of `immutable`
The RFC (and thus I) had initially thought that we'd be deprecating the
`immutable` field as part of this PR/Epic. However, after
[discussion](elastic#190198 (comment))
we have opted to continue supporting `immutable` until such time as we
can drop it, compatibility-wise.

## Steps to Review
1. Enable the Feature Flag: `prebuiltRulesCustomizationEnabled`
2. Install the prebuilt rules package via fleet
3. Create and export a custom rule to serve as a template (or download
one:

curl -L -H 'Authorization: 8eef0fe5d7dfc52b' -o 'rules_export
(1).ndjson'
https://upload.elastic.co/d/4693e7fe4356ce7bcf7b7d6b72881a9fd189730c61bf5ef47c9930458d746979
    )

4. Install some prebuilt rules, and obtain a prebuilt rule's `rule_id`,
e.g. `ac8805f6-1e08-406c-962e-3937057fa86f`
5. Edit the `rules_export.ndjson` to contain only the first line, and
modify the `rule_id` with the prebuilt rule's
6. Import `rules_export.ndjson` and then retrieve the rule via the Dev
Console:

GET
kbn:api/detection_engine/rules?rule_id=ac8805f6-1e08-406c-962e-3937057fa86f

7. Observe that the rule has the expected `rule_source` and `immutable`
values
8. Test other permutations of import by modifying `rules_export.ndjson`
and re-importing; see ([the test
plan](elastic#191116) as well as a
[reference table of
scenarios](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md#importing-rules))

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Georgii Gorbachev <[email protected]>
(cherry picked from commit c0b1301)
@kibanamachine kibanamachine mentioned this pull request Oct 16, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

@rylnd rylnd deleted the rylnd/prebuilt_rule_import branch October 16, 2024 22:00
kibanamachine added a commit that referenced this pull request Oct 16, 2024
@kibanamachine @rylnd
[8.x] [Security Solution] Allow importing of prebuilt rules via the A…
40e21a7 
…PI (#190198) (#196613)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Allow importing of prebuilt rules via the API
(#190198)](#190198)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ryland
Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-16T21:19:05Z","message":"[Security
Solution] Allow importing of prebuilt rules via the API (#190198)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
import\r\nprebuilt rules via our existing import API.
The\r\n[RFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the import-specific issue is
described\r\n[here](#180168),
but at a high\r\nlevel we're adding two things in this PR:\r\n\r\n1. The
logic to calculate the new `rule_source` field on import,
which\r\ncontains the information about the rule's relationship to
existing\r\nprebuilt rules.\r\n2. A new code path for importing rules,
which respects the calculated\r\n`rule_source` field.\r\n\r\nIn order to
maintain backwards compatibility with the existing import\r\nlogic, and
because the existing import implementation is hard to\r\nmodify/extend,
I opted to add a top-level switch on the feature flag in\r\nthe import
route itself, which calls either the existing import function\r\n(now
named `importRulesLegacy`), or the new function, `importRules`,\r\nwhich
ultimately calls the new `DetectionRulesClient`
method,\r\n`importRules`. Neither knows about the feature flag, and
thanks to great\r\nsuggestions from @xcrzx there are nice, clean
boundaries between the\r\nimport functions and the client
methods.\r\n\r\nI went down the path of trying to write the new import
code by reusing\r\nthe outer `importRules` function, but after
discussion with the team we\r\ndecided it was simplest to simply
bifurcate the code at that point, so\r\nthat we have:\r\n\r\n1. The
legacy import code, which:\r\n * only supports custom rules (`immutable:
false`)\r\n * accepts `version` as an optional parameter\r\n* calculates
a legacy `rule_source` value based on the `immutable` field\r\n2. The
new import code, which\r\n * Installs the prebuilt rules assets as
necessary\r\n * Accepts both kinds of rules (`immutable:
true/false`)\r\n * Requires `version` as a parameter for _prebuilt_
rules\r\n * Allows `version` to be optional for custom rules\r\n *
calculates a proper `rule_source` (and `immutable`)\r\n\r\n\r\n###
Deprecation of `immutable`\r\nThe RFC (and thus I) had initially thought
that we'd be deprecating the\r\n`immutable` field as part of this
PR/Epic. However,
after\r\n[discussion](https://github.com/elastic/kibana/pull/190198#discussion_r1736021749)\r\nwe
have opted to continue supporting `immutable` until such time as
we\r\ncan drop it, compatibility-wise.\r\n\r\n\r\n## Steps to
Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n2. Install the prebuilt rules
package via fleet\r\n3. Create and export a custom rule to serve as a
template (or download\r\none:\r\n\r\ncurl -L -H 'Authorization:
8eef0fe5d7dfc52b' -o
'rules_export\r\n(1).ndjson'\r\nhttps://upload.elastic.co/d/4693e7fe4356ce7bcf7b7d6b72881a9fd189730c61bf5ef47c9930458d746979\r\n
)\r\n \r\n4. Install some prebuilt rules, and obtain a prebuilt rule's
`rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n5. Edit the
`rules_export.ndjson` to contain only the first line, and\r\nmodify the
`rule_id` with the prebuilt rule's\r\n6. Import `rules_export.ndjson`
and then retrieve the rule via the
Dev\r\nConsole:\r\n\r\nGET\r\nkbn:api/detection_engine/rules?rule_id=ac8805f6-1e08-406c-962e-3937057fa86f\r\n\r\n7.
Observe that the rule has the expected `rule_source` and
`immutable`\r\nvalues\r\n8. Test other permutations of import by
modifying `rules_export.ndjson`\r\nand re-importing; see ([the
test\r\nplan](#191116) as well as
a\r\n[reference table
of\r\nscenarios](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md#importing-rules))\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Georgii Gorbachev
<[email protected]>","sha":"c0b1301a21b74e8b643fc0e898b1d982fd85d98e","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0","backport:version"],"title":"[Security Solution] Allow
importing of prebuilt rules via the
API","number":190198,"url":"https://github.com/elastic/kibana/pull/190198","mergeCommit":{"message":"[Security
Solution] Allow importing of prebuilt rules via the API (#190198)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
import\r\nprebuilt rules via our existing import API.
The\r\n[RFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the import-specific issue is
described\r\n[here](#180168),
but at a high\r\nlevel we're adding two things in this PR:\r\n\r\n1. The
logic to calculate the new `rule_source` field on import,
which\r\ncontains the information about the rule's relationship to
existing\r\nprebuilt rules.\r\n2. A new code path for importing rules,
which respects the calculated\r\n`rule_source` field.\r\n\r\nIn order to
maintain backwards compatibility with the existing import\r\nlogic, and
because the existing import implementation is hard to\r\nmodify/extend,
I opted to add a top-level switch on the feature flag in\r\nthe import
route itself, which calls either the existing import function\r\n(now
named `importRulesLegacy`), or the new function, `importRules`,\r\nwhich
ultimately calls the new `DetectionRulesClient`
method,\r\n`importRules`. Neither knows about the feature flag, and
thanks to great\r\nsuggestions from @xcrzx there are nice, clean
boundaries between the\r\nimport functions and the client
methods.\r\n\r\nI went down the path of trying to write the new import
code by reusing\r\nthe outer `importRules` function, but after
discussion with the team we\r\ndecided it was simplest to simply
bifurcate the code at that point, so\r\nthat we have:\r\n\r\n1. The
legacy import code, which:\r\n * only supports custom rules (`immutable:
false`)\r\n * accepts `version` as an optional parameter\r\n* calculates
a legacy `rule_source` value based on the `immutable` field\r\n2. The
new import code, which\r\n * Installs the prebuilt rules assets as
necessary\r\n * Accepts both kinds of rules (`immutable:
true/false`)\r\n * Requires `version` as a parameter for _prebuilt_
rules\r\n * Allows `version` to be optional for custom rules\r\n *
calculates a proper `rule_source` (and `immutable`)\r\n\r\n\r\n###
Deprecation of `immutable`\r\nThe RFC (and thus I) had initially thought
that we'd be deprecating the\r\n`immutable` field as part of this
PR/Epic. However,
after\r\n[discussion](https://github.com/elastic/kibana/pull/190198#discussion_r1736021749)\r\nwe
have opted to continue supporting `immutable` until such time as
we\r\ncan drop it, compatibility-wise.\r\n\r\n\r\n## Steps to
Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n2. Install the prebuilt rules
package via fleet\r\n3. Create and export a custom rule to serve as a
template (or download\r\none:\r\n\r\ncurl -L -H 'Authorization:
8eef0fe5d7dfc52b' -o
'rules_export\r\n(1).ndjson'\r\nhttps://upload.elastic.co/d/4693e7fe4356ce7bcf7b7d6b72881a9fd189730c61bf5ef47c9930458d746979\r\n
)\r\n \r\n4. Install some prebuilt rules, and obtain a prebuilt rule's
`rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n5. Edit the
`rules_export.ndjson` to contain only the first line, and\r\nmodify the
`rule_id` with the prebuilt rule's\r\n6. Import `rules_export.ndjson`
and then retrieve the rule via the
Dev\r\nConsole:\r\n\r\nGET\r\nkbn:api/detection_engine/rules?rule_id=ac8805f6-1e08-406c-962e-3937057fa86f\r\n\r\n7.
Observe that the rule has the expected `rule_source` and
`immutable`\r\nvalues\r\n8. Test other permutations of import by
modifying `rules_export.ndjson`\r\nand re-importing; see ([the
test\r\nplan](#191116) as well as
a\r\n[reference table
of\r\nscenarios](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md#importing-rules))\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Georgii Gorbachev
<[email protected]>","sha":"c0b1301a21b74e8b643fc0e898b1d982fd85d98e"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/190198","number":190198,"mergeCommit":{"message":"[Security
Solution] Allow importing of prebuilt rules via the API (#190198)\n\n##
Summary\r\n\r\nThis PR introduces the backend functionality necessary to
import\r\nprebuilt rules via our existing import API.
The\r\n[RFC](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md)\r\ngoes
into detail, and the import-specific issue is
described\r\n[here](#180168),
but at a high\r\nlevel we're adding two things in this PR:\r\n\r\n1. The
logic to calculate the new `rule_source` field on import,
which\r\ncontains the information about the rule's relationship to
existing\r\nprebuilt rules.\r\n2. A new code path for importing rules,
which respects the calculated\r\n`rule_source` field.\r\n\r\nIn order to
maintain backwards compatibility with the existing import\r\nlogic, and
because the existing import implementation is hard to\r\nmodify/extend,
I opted to add a top-level switch on the feature flag in\r\nthe import
route itself, which calls either the existing import function\r\n(now
named `importRulesLegacy`), or the new function, `importRules`,\r\nwhich
ultimately calls the new `DetectionRulesClient`
method,\r\n`importRules`. Neither knows about the feature flag, and
thanks to great\r\nsuggestions from @xcrzx there are nice, clean
boundaries between the\r\nimport functions and the client
methods.\r\n\r\nI went down the path of trying to write the new import
code by reusing\r\nthe outer `importRules` function, but after
discussion with the team we\r\ndecided it was simplest to simply
bifurcate the code at that point, so\r\nthat we have:\r\n\r\n1. The
legacy import code, which:\r\n * only supports custom rules (`immutable:
false`)\r\n * accepts `version` as an optional parameter\r\n* calculates
a legacy `rule_source` value based on the `immutable` field\r\n2. The
new import code, which\r\n * Installs the prebuilt rules assets as
necessary\r\n * Accepts both kinds of rules (`immutable:
true/false`)\r\n * Requires `version` as a parameter for _prebuilt_
rules\r\n * Allows `version` to be optional for custom rules\r\n *
calculates a proper `rule_source` (and `immutable`)\r\n\r\n\r\n###
Deprecation of `immutable`\r\nThe RFC (and thus I) had initially thought
that we'd be deprecating the\r\n`immutable` field as part of this
PR/Epic. However,
after\r\n[discussion](https://github.com/elastic/kibana/pull/190198#discussion_r1736021749)\r\nwe
have opted to continue supporting `immutable` until such time as
we\r\ncan drop it, compatibility-wise.\r\n\r\n\r\n## Steps to
Review\r\n1. Enable the Feature Flag:
`prebuiltRulesCustomizationEnabled`\r\n2. Install the prebuilt rules
package via fleet\r\n3. Create and export a custom rule to serve as a
template (or download\r\none:\r\n\r\ncurl -L -H 'Authorization:
8eef0fe5d7dfc52b' -o
'rules_export\r\n(1).ndjson'\r\nhttps://upload.elastic.co/d/4693e7fe4356ce7bcf7b7d6b72881a9fd189730c61bf5ef47c9930458d746979\r\n
)\r\n \r\n4. Install some prebuilt rules, and obtain a prebuilt rule's
`rule_id`,\r\ne.g. `ac8805f6-1e08-406c-962e-3937057fa86f`\r\n5. Edit the
`rules_export.ndjson` to contain only the first line, and\r\nmodify the
`rule_id` with the prebuilt rule's\r\n6. Import `rules_export.ndjson`
and then retrieve the rule via the
Dev\r\nConsole:\r\n\r\nGET\r\nkbn:api/detection_engine/rules?rule_id=ac8805f6-1e08-406c-962e-3937057fa86f\r\n\r\n7.
Observe that the rule has the expected `rule_source` and
`immutable`\r\nvalues\r\n8. Test other permutations of import by
modifying `rules_export.ndjson`\r\nand re-importing; see ([the
test\r\nplan](#191116) as well as
a\r\n[reference table
of\r\nscenarios](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md#importing-rules))\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[
]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>\r\nCo-authored-by:
Georgii Gorbachev
<[email protected]>","sha":"c0b1301a21b74e8b643fc0e898b1d982fd85d98e"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ryland Herrick <[email protected]>
@banderror banderror mentioned this pull request Oct 22, 2024
Merged
rylnd added a commit that referenced this pull request Nov 11, 2024
@rylnd @elasticmachine
[Security Solution] Test plans for prebuilt rule import and export (#…
e429849 
…191116)

## Summary

This PR introduces test plans for both [Prebuilt Rule
Import](#180168) (corresponding
[PR](#190198)) and [Prebuilt Rule
Export](#180167) (corresponding
[PR](#194498)). Import is
considerably more complicated as it is calculating new values (for
`rule_source`, `immutable`), while the export work is mainly removing
existing restrictions (which allowed only custom rules to be exported).

---------

Co-authored-by: Elastic Machine <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 11, 2024
@rylnd
[Security Solution] Test plans for prebuilt rule import and export (e…
3c89524 
…lastic#191116)

## Summary

This PR introduces test plans for both [Prebuilt Rule
Import](elastic#180168) (corresponding
[PR](elastic#190198)) and [Prebuilt Rule
Export](elastic#180167) (corresponding
[PR](elastic#194498)). Import is
considerably more complicated as it is calculating new values (for
`rule_source`, `immutable`), while the export work is mainly removing
existing restrictions (which allowed only custom rules to be exported).

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit e429849)
kibanamachine added a commit that referenced this pull request Nov 11, 2024
@kibanamachine @rylnd
[8.x] [Security Solution] Test plans for prebuilt rule import and exp…
48c81df 
…ort (#191116) (#199716)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Test plans for prebuilt rule import and export
(#191116)](#191116)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ryland
Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-11T21:01:46Z","message":"[Security
Solution] Test plans for prebuilt rule import and export (#191116)\n\n##
Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt
Rule\r\nImport](#180168)
(corresponding\r\n[PR](#190198))
and [Prebuilt
Rule\r\nExport](#180167)
(corresponding\r\n[PR](#194498)).
Import is\r\nconsiderably more complicated as it is calculating new
values (for\r\n`rule_source`, `immutable`), while the export work is
mainly removing\r\nexisting restrictions (which allowed only custom
rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic
Machine
<[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.17.0"],"title":"[Security Solution] Test
plans for prebuilt rule import and
export","number":191116,"url":"https://github.com/elastic/kibana/pull/191116","mergeCommit":{"message":"[Security
Solution] Test plans for prebuilt rule import and export (#191116)\n\n##
Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt
Rule\r\nImport](#180168)
(corresponding\r\n[PR](#190198))
and [Prebuilt
Rule\r\nExport](#180167)
(corresponding\r\n[PR](#194498)).
Import is\r\nconsiderably more complicated as it is calculating new
values (for\r\n`rule_source`, `immutable`), while the export work is
mainly removing\r\nexisting restrictions (which allowed only custom
rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic
Machine
<[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191116","number":191116,"mergeCommit":{"message":"[Security
Solution] Test plans for prebuilt rule import and export (#191116)\n\n##
Summary\r\n\r\nThis PR introduces test plans for both [Prebuilt
Rule\r\nImport](#180168)
(corresponding\r\n[PR](#190198))
and [Prebuilt
Rule\r\nExport](#180167)
(corresponding\r\n[PR](#194498)).
Import is\r\nconsiderably more complicated as it is calculating new
values (for\r\n`rule_source`, `immutable`), while the export work is
mainly removing\r\nexisting restrictions (which allowed only custom
rules to be exported).\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic
Machine
<[email protected]>","sha":"e4298492b5e48338396618d51168ea3e8427c103"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ryland Herrick <[email protected]>
tkajtoch pushed a commit to tkajtoch/kibana that referenced this pull request Nov 12, 2024
@rylnd @elasticmachine @tkajtoch
[Security Solution] Test plans for prebuilt rule import and export (e…
5d5733e 
…lastic#191116)

## Summary

This PR introduces test plans for both [Prebuilt Rule
Import](elastic#180168) (corresponding
[PR](elastic#190198)) and [Prebuilt Rule
Export](elastic#180167) (corresponding
[PR](elastic#194498)). Import is
considerably more complicated as it is calculating new values (for
`rule_source`, `immutable`), while the export work is mainly removing
existing restrictions (which allowed only custom rules to be exported).

---------

Co-authored-by: Elastic Machine <[email protected]>
@banderror banderror mentioned this pull request Nov 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joepeeples joepeeples joepeeples left review comments

@approksiu approksiu approksiu left review comments

@yctercero yctercero yctercero left review comments

@banderror banderror banderror left review comments

@xcrzx xcrzx xcrzx approved these changes

@jpdjere jpdjere Awaiting requested review from jpdjere

@maximpn maximpn Awaiting requested review from maximpn

Assignees

@rylnd rylnd

Labels
backport:version Backport to applied version labels Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.16.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@rylnd @elasticmachine @banderror @kibana-ci @kibanamachine @xcrzx @jpdjere @yctercero @approksiu @joepeeples