[Security Solution] Implement UI for updating prebuilt rule to a new rule type (MVP)

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Design Discussion context: #178211
Design: Figma (internal)
Depends on: #180589, #190482 (not blocked by, but would make sense to address this one first)

Summary

Currently, we allow rule types to be changed by the TRADE team in the next version of a prebuilt rule. For backward compatibility, we will need to continue to support that. The problem is that the regular upgrade UI we will have (#171520) won't be suitable for handling rule type changes because different rule types can have incompatible rule fields.

If a rule has changed its type in the target version, during rule upgrade we should handle it with a dedicated UI for this edge case. Instead of the regular ThreeWayDiff component we should show another component in the same tab.

User story (MVP-focused)

• As a user, I want to be informed that the incoming Elastic update has a rule type change.
• As a user, I want to assess the rule update before applying it and know the implications for my customizations.
• As a user, I want to know what I can do to preserve the current rule version with my customizations.

Acceptance criteria

• Customised rules with the incoming updates that change the rule type can be viewed in the 2-way diff rule flyout.
• The user cannot edit the incoming update, only accept it as a whole.
• User is informed that the rule type is changing, if they update, their customizations will be lost, and they need to take note of those to be able to reapply to the new rule version. Users are informed they can clone the rule if they want to keep the old and new rules.

Designs

TBD @ARWNightingale

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ARWNightingale]

Hey @jpdjere, some quick questions:

1. If the new rule type has fields that are not present in the current version. Are we saying that we still show those but with no suggestion or as deprecated?
2. You mention automatically copy/transfer field values from the current version to the final one. would this be automatically an option in the target version via a user action or be more of a bulk action/higher action?

Thanks :)

[jpdjere]

@ARWNightingale

If the new rule type has fields that are not present in the current version. Are we saying that we still show those but with no suggestion or as deprecated?

New fields (fields only existing in the target version but not in base or current versions) should be shown to the user, with their values set to whatever the target version (the update proposed by Elastic) has. An example, if a query rule is being updated to new_terms, the target version will have a new new_terms_fields which previously didn't exist. If the proposed value for that field is ["A", "B", "C"], then that's what we should show to the user.

You mention automatically copy/transfer field values from the current version to the final one. would this be automatically an option in the target version via a user action or be more of a bulk action/higher action?

This actually refers to a calculation that we would do in the backend: if we detect that the same field exists in the current and in the target rule, we would return as the proposed final version the value that is in the current version: i.e., maintain whatever values the user currently has for the rule, even if the target version has changed. So when the user opens up this upgrade workflow, we will propose to the user that all fields existing in both version will maintain their current values. (Option 1)

Alternatively, we can think of this more like "bulk action" for the whole rule: instead of proposing the current value for the fields, as proposed above, we propose to use the new target version. BUT, we offer this clickable bulk action button which, when clicked, will replace the values of all fields that exist in both versions from their target version to their current value. (Option 2)

To clarify, let's see the example above, using the mechanisms described for Option 1 and Option 2. The name and description field exists in both current and target but might be different, while new_terms_fields exists only in the target version.

Option 1

Field name	Current	Target	Proposed Final/Merged Version
name	My Great Rule	New Terms Fields Rule	My Great Rule
description	This is the rule I created.	This rule has a new type, which is new terms.	This is the rule I created.
new_terms_fields	➖	["A", "B", "C"]	["A", "B", "C"]

Option 2

Field name	Current	Target	Proposed Final Version	Proposed Final Version after clicking on Bulk Action
name	My Great Rule	New Terms Fields Rule	New Terms Fields Rule	My Great Rule
description	This is the rule I created.	This rule has a new type, which is new terms.	This rule has a new type, which is new terms.	This is the rule I created.
new_terms_fields	➖	["A", "B", "C"]	["A", "B", "C"]	["A", "B", "C"]

I rather prefer Option 2. I think it makes more sense for the user that we offer them a calculated Proposed Final version in which the fields that exists in both current and target are updated to whatever Elastic is proposing, BUT giving them and easy, one-click way of restoring their current values into the Final version.

FYI @approksiu @banderror to get your opinions.

[banderror]

Not sure I fully understand your proposal @jpdjere.

I think before we discuss UI patterns like "bulk actions" or anything like that, we should get a high-level understanding of what should be the overall behavior of a "rule upgrade to a new type, considering possible user customization in the current version". What do we allow, what do we not allow. The description suggests a list possibilities, but it needs review from the product side. Maybe we decide that we only allow to update to a target version, and that's it.

I think it would be great to set up a meeting to brainstorm options, based on which @approksiu could write user stories for this feature. I can set up one. What do you think?

@approksiu @ARWNightingale @jpdjere

[approksiu]

Examples of the rule type changes on upgrade:

• https://github.com/elastic/detection-rules/pull/3802/files
• https://github.com/elastic/detection-rules/pull/3745/files

[jpdjere]

Meeting discussion from 18-07-2024

Participants: @approksiu @ARWNightingale @banderror @jpdjere

• Instead of showing ThreeWayDiff component being built, use current existing 2-way diff to show changes
• We will only allow to update to target version, and customizations will be lost
• Inform the user via callout/text/pop-up that:
  • rules type has change
  • rule can only update to target version
  • customizations (except actions, response_actions and exceptions) will be lost.
• Advise that user might want to:
  • Note customizations before updating
  • Clone the rule before updating

[approksiu]

@jpdjere for the incoming update fields - do we allow users to edit them in the MVP?

[jpdjere]

@approksiu Not sure if I'm understanding your question correctly but: no, in this MVP, the process will be "read-only". We'll display the new incoming fields as a diff with the current ones, but with no ability to edit them. Very similar to the current read-only update process.