Skip to content

Commit

Permalink
[Security Solution] [Attack discovery] Improves Attack discovery GPT-…
Browse files Browse the repository at this point in the history
…4o output (elastic#187183)

## [Security Solution] [Attack discovery] Improves Attack discovery GPT-4o output

### Summary

This PR improves the output of Attack discoveries generated with `GPT-4o`, as illustrated by the _After_ screenshot below:

#### After

![gpt-4o-after](https://github.com/elastic/kibana/assets/4459398/c6896da2-584b-48a3-a5ba-31b94c06e6e8)

_Above: Attack discoveries generated via `GPT-4o` after the change in this branch_

#### Before

![gpt-4o-before](https://github.com/elastic/kibana/assets/4459398/d878c36a-c926-4f45-b1d5-7867d530df12)

_Above: Attack discoveries generated via `GPT-4o` before the change in this branch_

### Desk testing

1) Generate (using `main`) Attack discoveries via `GPT-4o`, and a few other models, i.e. `Claude 3 Opus`, and `Claude 3.5 Sonnet`

2) Generate Attack discoveries for the same models using the PR branch

**Expected result**

- Significant improvement in the narrative and format of Attack discoveries generated by `GPT-4o`
- Similar / improved output from other models (no regressions)
  • Loading branch information
andrew-goldstein authored Jun 28, 2024
1 parent 72e1d11 commit e64978e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Your output will be parsed and type-checked according to the provided schema ins
Here is the JSON Schema instance your output must adhere to. Include the enclosing markdown codeblock:
\`\`\`json
{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"alertIds\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"The alert IDs that the insight is based on.\"},\"detailsMarkdown\":{\"type\":\"string\",\"description\":\"A detailed insight with markdown that always uses special {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax for field names and values from the source data. Examples of CORRECT syntax (includes field names and values): {{ host.name hostNameValue }} {{ user.name userNameValue }} {{ source.ip sourceIpValue }} Examples of INCORRECT syntax (bad, because the field names are not included): {{ hostNameValue }} {{ userNameValue }} {{ sourceIpValue }}\"},\"entitySummaryMarkdown\":{\"type\":\"string\",\"description\":\"A short (no more than a sentence) summary of the insight featuring only the host.name and user.name fields (when they are applicable), using the same {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax\"},\"mitreAttackTactics\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"An array of MITRE ATT&CK tactic for the insight, using one of the following values: Reconnaissance,Initial Access,Execution,Persistence,Privilege Escalation,Discovery,Lateral Movement,Command and Control,Exfiltration\"},\"summaryMarkdown\":{\"type\":\"string\",\"description\":\"A markdown summary of insight, using the same {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax\"},\"title\":{\"type\":\"string\",\"description\":\"A short, no more than 7 words, title for the insight, NOT formatted with special syntax or markdown. This must be as brief as possible.\"}},\"required\":[\"alertIds\",\"detailsMarkdown\",\"summaryMarkdown\",\"title\"],\"additionalProperties\":false},\"description\":\"Insights with markdown that always uses special {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax for field names and values from the source data. Examples of CORRECT syntax (includes field names and values): {{ host.name hostNameValue }} {{ user.name userNameValue }} {{ source.ip sourceIpValue }} Examples of INCORRECT syntax (bad, because the field names are not included): {{ hostNameValue }} {{ userNameValue }} {{ sourceIpValue }}\",\"$schema\":\"http://json-schema.org/draft-07/schema#\"}
{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"alertIds\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"The alert IDs that the insight is based on.\"},\"detailsMarkdown\":{\"type\":\"string\",\"description\":\"A detailed insight with markdown, where each markdown bullet contains a description of what happened that reads like a story of the attack as it played out and always uses special {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax for field names and values from the source data. Examples of CORRECT syntax (includes field names and values): {{ host.name hostNameValue }} {{ user.name userNameValue }} {{ source.ip sourceIpValue }} Examples of INCORRECT syntax (bad, because the field names are not included): {{ hostNameValue }} {{ userNameValue }} {{ sourceIpValue }}\"},\"entitySummaryMarkdown\":{\"type\":\"string\",\"description\":\"A short (no more than a sentence) summary of the insight featuring only the host.name and user.name fields (when they are applicable), using the same {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax\"},\"mitreAttackTactics\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"An array of MITRE ATT&CK tactic for the insight, using one of the following values: Reconnaissance,Initial Access,Execution,Persistence,Privilege Escalation,Discovery,Lateral Movement,Command and Control,Exfiltration\"},\"summaryMarkdown\":{\"type\":\"string\",\"description\":\"A markdown summary of insight, using the same {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax\"},\"title\":{\"type\":\"string\",\"description\":\"A short, no more than 7 words, title for the insight, NOT formatted with special syntax or markdown. This must be as brief as possible.\"}},\"required\":[\"alertIds\",\"detailsMarkdown\",\"summaryMarkdown\",\"title\"],\"additionalProperties\":false},\"description\":\"Insights with markdown that always uses special {{ field.name fieldValue1 fieldValue2 fieldValueN }} syntax for field names and values from the source data. Examples of CORRECT syntax (includes field names and values): {{ host.name hostNameValue }} {{ user.name userNameValue }} {{ source.ip sourceIpValue }} Examples of INCORRECT syntax (bad, because the field names are not included): {{ hostNameValue }} {{ userNameValue }} {{ sourceIpValue }}\",\"$schema\":\"http://json-schema.org/draft-07/schema#\"}
\`\`\`
`;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ export const getOutputParser = () =>
detailsMarkdown: z
.string()
.describe(
`A detailed insight with markdown that always uses special ${SYNTAX} syntax for field names and values from the source data. ${GOOD_SYNTAX_EXAMPLES} ${BAD_SYNTAX_EXAMPLES}`
`A detailed insight with markdown, where each markdown bullet contains a description of what happened that reads like a story of the attack as it played out and always uses special ${SYNTAX} syntax for field names and values from the source data. ${GOOD_SYNTAX_EXAMPLES} ${BAD_SYNTAX_EXAMPLES}`
),
entitySummaryMarkdown: z
.string()
Expand Down

0 comments on commit e64978e

Please sign in to comment.