[Security Solution] Kibana OOM and crashing when running indicator match rule

[marshallmain]

Describe the bug:
Kibana's memory usage increases significantly when running a particular indicator match rule with ~520k indicator items. The rule is scheduled to run every 30 minutes, and every 30 minutes Kibana's memory usage increases to the limit (4GB in this case) and it crashes. The system returned to normal once the rule was disabled. The logs indicate that the rule executes for ~12 minutes before Kibana crashes, and during that time the rule does not appear to finish executing.

Kibana/Elasticsearch Stack version:
7.15.1

Steps to reproduce:

1. Enable APM in Kibana. Ensure that captureSpanStackTraces: true is set in the APM config. This will happen by default in 7.15.1 if you set a serverUrl in the APM config.

module.exports = {
  active: true,
};

2. Create an index with ~500k documents to use as an indicator index. I used the es archive x-pack/test/functional/es_archives/filebeat/default, but modified it by removing the _id from each doc in the archive and creating ~80 copies of the data.json file. Using es_archiver on this folder created an index with ~500k docs.
3. Create an index with a single document for the indicator match rule to query.

PUT /test-index
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "host.name": {
        "type": "keyword"
      }
    }
  }
}

POST test-both/_doc
{
  "@timestamp": 1636493980000,
  "host.name": "myHost"
}

4. Create an indicator match rule that uses the filebeat index as the indicator index and test-index as the source index pattern. Activate the rule.
5. Observe as Kibana's memory usage increases. After a few minutes, Kibana should crash with an out of memory error. With the filebeat indicator index described above, Kibana crashed after processing ~300k indicator items and using 4GB of memory.

Current behavior:
Kibana crashes due to running out of memory. It appears that the APM agent may be storing the entire response for every Elasticsearch query within a transaction. Since task manager runs tasks within a transaction, every Elasticsearch query response from a rule execution is being stored in memory.

Without APM enabled, the same rule executes without crashing Kibana.

Expected behavior:
Kibana should not run out of memory, even with APM enabled.

[marshallmain]

Closing as this is not a bug in the indicator match rule logic.