[ResponseOps] API for reading/writing rule execution summary

[xcrzx]

Based on: #112193, RFC: Consolidating rule statuses for RAC Rule Management / Monitoring (internal)
Related to: #136039

Summary

We need to design and implement an API that allows passing execution outcomes and metrics from rule executors.

As per RFC, the object containing the execution summary could look like the following:

{
  "outcome": "warning",
  "outcome_msg": "outcome description",
  "warning": ["list of execution warnings"],
  "metrics": {
    "total_search_duration_ms": 1234567890,
    "total_indexing_duration_ms": 123456789,
    "total_alerts_detected": 543,
    "total_alerts_created": 100,
    "gap_duration_s": 7
  }
}

A couple of things to mention:

• The summary object is generated on the solution side at the end of rule execution. So we need to find a way to pass it from solution to alerting framework.
• The number of metrics might vary from one rule type to another. Therefore, we should be able to extend it on the solution side if needed.
• Rules should be filterable and sortable by metrics and execution outcome.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[peluja1012]

Implementation ticket here #136039

[jpdjere]

@XavierM

I started work in this PR, that injects the RuleMonitoringService into the security rule wrapper, so as to update the rule monitoring data on the Security Solution side -during rule execution- before the final lastRun data is calculated by the Alerting Framework in the processRunResults method of the Task Runner, and saved to the rule SavedObject.

I expanded the RuleMonitoringService so as to be able to update the outcome, outcomeMsg and warning fields of the new lastRun field during rule execution, but found the following limitation:

The Alerting Framework currently calculates the values for the lastRun property at the end of rule execution, using the lastRunFromState function within processRunResults method of the TaskRunner, just before saving to the Rule Saved Object. I updated the lastRunFromState function so that it also takes the current state of the ruleMonitoringService as an additional parameter, in order to take into account the previous updates that we might have done to outcome, outcomeMsg and warning from the Security Solution side.

However, in the PR as I have it up to now, the lastRunFromState function can still overwrite outcome, outcomeMsg and warning in two particular cases based on the metrics generated by the Alerting Framework (the alert circuit breaker and the actions circuit breaker). It worries us that in these two cases, any previous value for outcome, outcomeMsg and warning set by us on the Security Solution side will be overwritten and could lead us to lose information that we need of what happened during the execution.

We though of two possibilities to counter this issue, but would like your input:

• one simple solution would be for the Framework never to overwrite outcome or warning values that were generated from the Solution side. This would require us to add some additional logic but would completely solve our concerns. But we'd need your input if is possible.
• have warning as an array. This would allow us to have both data generated by the Solution side and the Framework side, but still the outcome would be overwritten by the Framework (unless we do some additional changes to this logic as well, to prioritise the outcome set by the Security Solution).

[banderror]

Addressed in #136039 and #147278