Stage 2 changes for RFC 0021 - Threat enrichments (elastic#1478) (ela…

…stic#1487)

* migrate fields from experimental to beta

* generate artifacts

* edit changelog

* updating field defintions

* regenerate reuses

* align artifacts with master

* update artifacts

* removing experimental threat schema file

CHANGELOG.next.md

@@ -20,9 +20,9 @@ Thanks, you're awesome :-) -->
 * Remove `beta` from `orchestrator` field set. #1417
 * Extend `threat.*` field set beta. #1438
 * Added `event.agent_id_status` field. #1454
-* `threat.enrichments` added to the experimental schema. #1457
 * `process.target` and `process.target.parent` added to experimental schema. #1467
 * Threat indicator fields progress to beta stage. #1471
+* `threat.enrichments` beta fields. #1478
 
 #### Improvements
 

docs/field-details.asciidoc

@@ -289,6 +289,8 @@ The `as` fields are expected to be nested at:
 
 * `source.as`
 
+* `threat.enrichments.as`
+
 
 Note also that the `as` fields are not expected to be used directly at the root of the events.
 
@@ -3032,6 +3034,20 @@ example: `https://mysystem.example.com/alert/5271dedb-f5b0-4218-87f0-4ac4870a38f
 
 |=====
 
+[discrete]
+==== Field Reuse
+
+The `event` fields are expected to be nested at:
+
+
+* `threat.enrichments.event`
+
+
+Note also that the `event` fields may be used directly at the root of the events.
+
+
+
+
 [[ecs-file]]
 === File Fields
 
@@ -3418,6 +3434,14 @@ example: `1001`
 [discrete]
 ==== Field Reuse
 
+The `file` fields are expected to be nested at:
+
+
+* `threat.enrichments.file`
+
+
+Note also that the `file` fields may be used directly at the root of the events.
+
 
 
 
@@ -3688,6 +3712,8 @@ The `geo` fields are expected to be nested at:
 
 * `source.geo`
 
+* `threat.enrichments.geo`
+
 
 Note also that the `geo` fields are not expected to be used directly at the root of the events.
 
@@ -3886,6 +3912,8 @@ The `hash` fields are expected to be nested at:
 
 * `process.hash`
 
+* `threat.enrichments.hash`
+
 
 Note also that the `hash` fields are not expected to be used directly at the root of the events.
 
@@ -7646,6 +7674,114 @@ These fields are for users to classify alerts from all of their sources (e.g. ID
 
 // ===============================================================
 
+|
+[[field-threat-enrichments]]
+<<field-threat-enrichments, threat.enrichments>>
+
+| beta:[ This field is beta and subject to change. ]
+
+A list of associated indicators enriching the event, and the context of that association/enrichment.
+
+type: nested
+
+
+
+
+
+| extended
+
+// ===============================================================
+
+|
+[[field-threat-enrichments-matched-atomic]]
+<<field-threat-enrichments-matched-atomic, threat.enrichments.matched.atomic>>
+
+| beta:[ This field is beta and subject to change. ]
+
+Identifies the atomic indicator value that matched a local environment endpoint or network event.
+
+type: keyword
+
+
+
+example: `bad-domain.com`
+
+| extended
+
+// ===============================================================
+
+|
+[[field-threat-enrichments-matched-field]]
+<<field-threat-enrichments-matched-field, threat.enrichments.matched.field>>
+
+| beta:[ This field is beta and subject to change. ]
+
+Identifies the field of the atomic indicator that matched a local environment endpoint or network event.
+
+type: keyword
+
+
+
+example: `file.hash.sha256`
+
+| extended
+
+// ===============================================================
+
+|
+[[field-threat-enrichments-matched-id]]
+<<field-threat-enrichments-matched-id, threat.enrichments.matched.id>>
+
+| beta:[ This field is beta and subject to change. ]
+
+Identifies the _id of the indicator document enriching the event.
+
+type: keyword
+
+
+
+example: `ff93aee5-86a1-4a61-b0e6-0cdc313d01b5`
+
+| extended
+
+// ===============================================================
+
+|
+[[field-threat-enrichments-matched-index]]
+<<field-threat-enrichments-matched-index, threat.enrichments.matched.index>>
+
+| beta:[ This field is beta and subject to change. ]
+
+Identifies the _index of the indicator document enriching the event.
+
+type: keyword
+
+
+
+example: `filebeat-8.0.0-2021.05.23-000011`
+
+| extended
+
+// ===============================================================
+
+|
+[[field-threat-enrichments-matched-type]]
+<<field-threat-enrichments-matched-type, threat.enrichments.matched.type>>
+
+| beta:[ This field is beta and subject to change. ]
+
+Identifies the type of match that caused the event to be enriched with the given indicator
+
+type: keyword
+
+
+
+example: `indicator_match_rule`
+
+| extended
+
+// ===============================================================
+
 |
 [[field-threat-framework]]
 <<field-threat-framework, threat.framework>>
@@ -8315,6 +8451,81 @@ example: `https://attack.mitre.org/techniques/T1059/001/`
 
 // ===============================================================
 
+|=====
+
+[discrete]
+==== Field Reuse
+
+
+
+
+[[ecs-threat-nestings]]
+[discrete]
+===== Field sets that can be nested under Threat
+
+[options="header"]
+|=====
+| Location | Field Set | Description
+
+// ===============================================================
+
+
+| `threat.enrichments.as.*`
+| <<ecs-as,as>>| beta:[ Reusing the `as` fields in this location is currently considered beta.]
+
+Fields describing an Autonomous System (Internet routing prefix).
+
+// ===============================================================
+
+
+| `threat.enrichments.event.*`
+| <<ecs-event,event>>| beta:[ Reusing the `event` fields in this location is currently considered beta.]
+
+Fields breaking down the event details.
+
+// ===============================================================
+
+
+| `threat.enrichments.file.*`
+| <<ecs-file,file>>| beta:[ Reusing the `file` fields in this location is currently considered beta.]
+
+Fields describing files.
+
+// ===============================================================
+
+
+| `threat.enrichments.geo.*`
+| <<ecs-geo,geo>>| beta:[ Reusing the `geo` fields in this location is currently considered beta.]
+
+Fields describing a location.
+
+// ===============================================================
+
+
+| `threat.enrichments.hash.*`
+| <<ecs-hash,hash>>| beta:[ Reusing the `hash` fields in this location is currently considered beta.]
+
+Hashes, usually file hashes.
+
+// ===============================================================
+
+
+| `threat.enrichments.url.*`
+| <<ecs-url,url>>| beta:[ Reusing the `url` fields in this location is currently considered beta.]
+
+Fields that let you store URLs in various forms.
+
+// ===============================================================
+
+
+| `threat.enrichments.x509.*`
+| <<ecs-x509,x509>>| beta:[ Reusing the `x509` fields in this location is currently considered beta.]
+
+These fields contain x509 certificate metadata.
+
+// ===============================================================
+
+
 |=====
 
 [[ecs-tls]]
@@ -9191,6 +9402,20 @@ type: keyword
 
 |=====
 
+[discrete]
+==== Field Reuse
+
+The `url` fields are expected to be nested at:
+
+
+* `threat.enrichments.url`
+
+
+Note also that the `url` fields may be used directly at the root of the events.
+
+
+
+
 [[ecs-user]]
 === User Fields
 
@@ -10300,6 +10525,8 @@ The `x509` fields are expected to be nested at:
 
 * `file.x509`
 
+* `threat.enrichments.x509`
+
 * `tls.client.x509`
 
 * `tls.server.x509`

experimental/generated/ecs/ecs_flat.yml

@@ -12423,6 +12423,7 @@ tags:
   short: List of keywords used to tag each event.
   type: keyword
 threat.enrichments:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments
   description: A list of associated indicators enriching the event, and the context
     of that association/enrichment.
@@ -14165,6 +14166,7 @@ threat.enrichments.hash.ssdeep:
   short: SSDEEP hash.
   type: keyword
 threat.enrichments.matched.atomic:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments-matched-atomic
   description: Identifies the atomic indicator value that matched a local environment
     endpoint or network event.
@@ -14177,6 +14179,7 @@ threat.enrichments.matched.atomic:
   short: Matched indicator value
   type: keyword
 threat.enrichments.matched.field:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments-matched-field
   description: Identifies the field of the atomic indicator that matched a local environment
     endpoint or network event.
@@ -14189,6 +14192,7 @@ threat.enrichments.matched.field:
   short: Matched indicator field
   type: keyword
 threat.enrichments.matched.id:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments-matched-id
   description: Identifies the _id of the indicator document enriching the event.
   example: ff93aee5-86a1-4a61-b0e6-0cdc313d01b5
@@ -14200,6 +14204,7 @@ threat.enrichments.matched.id:
   short: Matched indicator identifier
   type: keyword
 threat.enrichments.matched.index:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments-matched-index
   description: Identifies the _index of the indicator document enriching the event.
   example: filebeat-8.0.0-2021.05.23-000011
@@ -14211,6 +14216,7 @@ threat.enrichments.matched.index:
   short: Matched indicator index
   type: keyword
 threat.enrichments.matched.type:
+  beta: This field is beta and subject to change.
   dashed_name: threat-enrichments-matched-type
   description: Identifies the type of match that caused the event to be enriched with
     the given indicator