-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] Extend the POST /upgrade/_perform
API endpoint's contract and functionality
#166376
Comments
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
POST /prebuilt_rules/upgrade/_perform
API endpoint's contract and functionality
POST /prebuilt_rules/upgrade/_perform
API endpoint's contract and functionalityPOST /prebuilt_rules/upgrade/_perform
API endpoint's contract and functionality (DRAFT)
POST /prebuilt_rules/upgrade/_perform
API endpoint's contract and functionality (DRAFT)POST /upgrade/_perform
API endpoint's contract and functionality (DRAFT)
POST /upgrade/_perform
API endpoint's contract and functionality (DRAFT)POST /upgrade/_perform
API endpoint's contract and functionality
…tract migrating to Zod (#189790) Partially addresses (contract change only): #166376 Created in favour of: #189187 (closed) ## Summary - Extends contract as described in the [POC](#144060), migrating from `io-ts` to Zod (search for `Perform rule upgrade`) - Uses new types in endpoint, but functionality remains unchaged. ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
General planThis ticket will be worked on by following the steps:
Step 2 to 6 will be a second, single PR. Implementation plan
|
@jpdjere We talked with @xcrzx yesterday and figured that it would be great, besides writing a test plan, to first create a list of edge cases that the endpoint should handle. I see that you have already added such a list to #166376 (comment), could you please revisit it and add to the ticket description as an AC? One thing we discussed is that any rule type change should yield an |
…0128) ## Summary - Partially addresses #166376 (see step 1 of [plan](#166376 (comment))) - Partially addresses: #190597 - Creates a Map of the fields that are upgradable during the Upgrade workflow, by type. - Creating this Map dynamically, based of BaseCreateProps and TypeSpecificFields, ensures that we don't need to: - manually add rule types to this Map if they are created - manually add or remove any fields if they are added or removed to a specific rule type - manually add or remove any fields if we decide that they should not be part of the upgradable fields. - This Map will be used as part of the `/upgrade/_perform` endpoint handler logic to build the payload of fields that will be upgraded to their different versions (`BASE`, `CURRENT`, `TARGET`, `MERGED`,`RESOLVED`) - Creates `RuleFieldsToUpgrade` Zod schema and `FieldUpgradeSpecifier` type, part of the `/upgrade/_perform` payload, which defines which fields can be upgraded and how. <br> <details> <summary>See output: <b>UPGRADABLE_RULES_FIELDS_BY_TYPE_MAP</b></summary> ```ts new Map([ [ "eql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "language", "index", "data_view_id", "filters", "event_category_override", "tiebreaker_field", "timestamp_field", "alert_suppression" ] ], [ "query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "query", "language" ] ], [ "saved_query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "saved_id", "index", "data_view_id", "filters", "alert_suppression", "query", "language" ] ], [ "threshold", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threshold", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "language" ] ], [ "threat_match", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threat_query", "threat_mapping", "threat_index", "index", "data_view_id", "filters", "saved_id", "threat_filters", "threat_indicator_path", "threat_language", "concurrent_searches", "items_per_search", "alert_suppression", "language" ] ], [ "machine_learning", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "anomaly_threshold", "machine_learning_job_id", "alert_suppression" ] ], [ "new_terms", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "new_terms_fields", "history_window_start", "index", "data_view_id", "filters", "alert_suppression", "language" ] ], [ "esql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "alert_suppression", "type", "language", "query" ] ] ]) ``` </details> <br> ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
I added this to the ticket description.
As part of the preparation PR, the I'll include two runtime checks related to this, which I described in the description above:
These two fields need to be handled with the rest of fields that need special treatment as described here. In this case, the endpoint will always force their update to the |
…stic#190128) ## Summary - Partially addresses elastic#166376 (see step 1 of [plan](elastic#166376 (comment))) - Partially addresses: elastic#190597 - Creates a Map of the fields that are upgradable during the Upgrade workflow, by type. - Creating this Map dynamically, based of BaseCreateProps and TypeSpecificFields, ensures that we don't need to: - manually add rule types to this Map if they are created - manually add or remove any fields if they are added or removed to a specific rule type - manually add or remove any fields if we decide that they should not be part of the upgradable fields. - This Map will be used as part of the `/upgrade/_perform` endpoint handler logic to build the payload of fields that will be upgraded to their different versions (`BASE`, `CURRENT`, `TARGET`, `MERGED`,`RESOLVED`) - Creates `RuleFieldsToUpgrade` Zod schema and `FieldUpgradeSpecifier` type, part of the `/upgrade/_perform` payload, which defines which fields can be upgraded and how. <br> <details> <summary>See output: <b>UPGRADABLE_RULES_FIELDS_BY_TYPE_MAP</b></summary> ```ts new Map([ [ "eql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "language", "index", "data_view_id", "filters", "event_category_override", "tiebreaker_field", "timestamp_field", "alert_suppression" ] ], [ "query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "query", "language" ] ], [ "saved_query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "saved_id", "index", "data_view_id", "filters", "alert_suppression", "query", "language" ] ], [ "threshold", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threshold", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "language" ] ], [ "threat_match", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threat_query", "threat_mapping", "threat_index", "index", "data_view_id", "filters", "saved_id", "threat_filters", "threat_indicator_path", "threat_language", "concurrent_searches", "items_per_search", "alert_suppression", "language" ] ], [ "machine_learning", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "anomaly_threshold", "machine_learning_job_id", "alert_suppression" ] ], [ "new_terms", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "new_terms_fields", "history_window_start", "index", "data_view_id", "filters", "alert_suppression", "language" ] ], [ "esql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "alert_suppression", "type", "language", "query" ] ] ]) ``` </details> <br> ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
…stic#190128) ## Summary - Partially addresses elastic#166376 (see step 1 of [plan](elastic#166376 (comment))) - Partially addresses: elastic#190597 - Creates a Map of the fields that are upgradable during the Upgrade workflow, by type. - Creating this Map dynamically, based of BaseCreateProps and TypeSpecificFields, ensures that we don't need to: - manually add rule types to this Map if they are created - manually add or remove any fields if they are added or removed to a specific rule type - manually add or remove any fields if we decide that they should not be part of the upgradable fields. - This Map will be used as part of the `/upgrade/_perform` endpoint handler logic to build the payload of fields that will be upgraded to their different versions (`BASE`, `CURRENT`, `TARGET`, `MERGED`,`RESOLVED`) - Creates `RuleFieldsToUpgrade` Zod schema and `FieldUpgradeSpecifier` type, part of the `/upgrade/_perform` payload, which defines which fields can be upgraded and how. <br> <details> <summary>See output: <b>UPGRADABLE_RULES_FIELDS_BY_TYPE_MAP</b></summary> ```ts new Map([ [ "eql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "language", "index", "data_view_id", "filters", "event_category_override", "tiebreaker_field", "timestamp_field", "alert_suppression" ] ], [ "query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "query", "language" ] ], [ "saved_query", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "saved_id", "index", "data_view_id", "filters", "alert_suppression", "query", "language" ] ], [ "threshold", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threshold", "index", "data_view_id", "filters", "saved_id", "alert_suppression", "language" ] ], [ "threat_match", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "threat_query", "threat_mapping", "threat_index", "index", "data_view_id", "filters", "saved_id", "threat_filters", "threat_indicator_path", "threat_language", "concurrent_searches", "items_per_search", "alert_suppression", "language" ] ], [ "machine_learning", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "anomaly_threshold", "machine_learning_job_id", "alert_suppression" ] ], [ "new_terms", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "type", "query", "new_terms_fields", "history_window_start", "index", "data_view_id", "filters", "alert_suppression", "language" ] ], [ "esql", [ "name", "description", "risk_score", "severity", "rule_name_override", "timestamp_override", "timestamp_override_fallback_disabled", "timeline_id", "timeline_title", "license", "note", "building_block_type", "investigation_fields", "version", "tags", "enabled", "risk_score_mapping", "severity_mapping", "interval", "from", "to", "exceptions_list", "author", "false_positives", "references", "max_signals", "threat", "setup", "related_integrations", "required_fields", "alert_suppression", "type", "language", "query" ] ] ]) ``` </details> <br> ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Summary
In #148184 we implemented a basic version of the
upgrade/_perform
API endpoint.We need to enhance this endpoint so it can work with prebuilt rules customized by users and resolve conflicts between user customizations and updates from Elastic in the target version.
Acceptance criteria
MERGED
version for rule upgrades.If the
MERGED
version is selected, the diffs are recalculated and the rule fields are updated to the result of the diff calculation. This is only possible if all field diffs return aconflict
value of eitherNO
. If any fields returns a value ofNON_SOLVABLE
orSOLVABLE
, reject the request with an error specifying that there are conflicts, and that they must be resolved on a per-field basis.pick_version
isMERGED
.pick_versions
:BASE' | 'CURRENT' | 'TARGET' | 'MERGED' | 'RESOLVED'
(SeeFieldUpgradeRequest
in PoC for details)Handling of special fields
Specific fields need to be handled under the hood based on #186544
Edge cases
pick_version
, at all levels, matchTARGET
. Otherwise, create new error and add to ruleErrors array.targetVersion.type
(for example, EQL) and the user includes in itsfields
object of the request payload any fields which do not match that rule type (in this case, for example, sending inmachine_learning_job_id
as part offields
), throw an error for that rule.NON_SOLVABLE
:MERGED
, and ANY fields return with aNON_SOLVABLE
conflict, reject the whole update for that rule: create new error and add to ruleErrors array.MERGED
, and one or more of the fields return with aNON_SOLVABLE
conflict, BUT those same fields have a specificpick_version
for them in thefields
object which ARE NOTMERGED
. No error should be reported in this case.pick_version
other than MERGED, but any specific field in thefields
object is set to upgrade toMERGED
, and the diff for that fields returns aNON_SOLVABLE
conflict. In that case, create new error and add to ruleErrors array.The text was updated successfully, but these errors were encountered: