x-pack/filebeat/module/oracle - Added authentication messages parsing #35127

leweafan · 2023-04-19T11:02:05Z

What does this PR do?

Adds authentication messages parsing

Why is it important?

Authentication messages important for SIEM.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Filebeat Oracle ingest pipeline can't parse authentication messages #30975

Use cases

Screenshots

Logs

mergify · 2023-04-19T11:02:50Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @leweafan? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-04-19T11:06:57Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-23T23:03:44.402+0000
Duration: 75 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7696
Skipped	753
Total	8449

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-04-19T14:05:42Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

leweafan · 2023-04-19T14:49:44Z

Hello @jamiehynds can you run /test ?

x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

efd6 · 2023-04-19T23:58:46Z

x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.json

Why is this being added?

mergify · 2023-04-20T00:10:56Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b oracle_new upstream/oracle_new
git merge upstream/main
git push upstream oracle_new

….geo/as

Co-authored-by: Dan Kortschak <[email protected]>

leweafan · 2023-04-20T15:26:35Z

@efd6 thanks for help. Seems I've fixed all problems.

efd6 · 2023-04-20T21:51:38Z

x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.json

Please delete this and replace the yml rendering of the pipeline.

What is wrong with json?

Nothing, just that for pipelines in integrations we render as YAML.

This has brought in a lot of diff noise. Ideally only the code that you are changing here should be reflected in the diff.

I think this is the state that you wanted it in, but with only relevant changes. Please check and update.

description: Pipeline for parsing Oracle Audit logs processors: - set: field: event.ingested value: "{{_ingest.timestamp}}" - set: field: event.action value: database_audit - set: field: event.kind value: event - set: field: event.category value: database - set: field: event.type value: access - set: field: event.outcome value: success - grok: field: message patterns: - "%{GREEDYDATA:tmp_timestamp}\\\nLENGTH : '%{GREEDYDATA:LENGTH}'\\\n(?m)%{GREEDYDATA:audit}" - "%{GREEDYDATA:tmp_timestamp}\\\nLENGTH: \"%{GREEDYDATA:LENGTH}\"\\\n(?m)%{GREEDYDATA:auth}" - gsub: field: "auth" pattern: "\"\\s" replacement: "\"|" ignore_missing: true if: ctx.auth != null - kv: field: audit field_split: "\\\n(?=[a-zA-Z])" value_split: ":\\S\\d+\\S(?= ')" trim_value: " '" trim_key: " " prefix: oracle.database_audit. if: ctx.audit != null - grok: field: log.file.path patterns: - "%{BASE10NUM:process.pid}\\_%{BASE10NUM}\\.aud(\\.log)?$" # All field names are uppercase by default, converts them to lowercase - script: source: "ctx.oracle.database_audit = ctx.oracle.database_audit.entrySet().stream().collect(Collectors.toMap(entry -> entry.getKey().toLowerCase(), Map.Entry::getValue));" lang: painless if: ctx.oracle?.database_audit != null # Replace all field names that has spaces in them with _ - script: lang: painless source: "ctx.oracle.database_audit = ctx.oracle?.database_audit.entrySet().stream().collect(Collectors.toMap(e -> e.getKey().replace(' ', '_'), e -> e.getValue()));" if: ctx.oracle?.database_audit != null - gsub: field: "oracle.database_audit.action" pattern: "\\n" replacement: "" if: ctx.oracle?.database_audit != null - gsub: field: "oracle.database_audit.action" pattern: "\\s{2,}" replacement: " " if: ctx.oracle?.database_audit != null - trim: field: "oracle.database_audit.action_number" ignore_missing: true if: ctx.oracle?.database_audit != null # Removes all null values from ctx.* - script: lang: painless if: ctx?.oracle?.database_audit != null source: | void handleMap(Map map) { for (def x : map.values()) { if (x instanceof Map) { handleMap(x); } else if (x instanceof List) { handleList(x); } } map.values().removeIf(v -> v instanceof String && v.isEmpty() == true); } void handleList(List list) { for (def x : list) { if (x instanceof Map) { handleMap(x); } else if (x instanceof List) { handleList(x); } } } handleMap(ctx); - remove: field: - "@timestamp" ignore_missing: true - date: field: tmp_timestamp target_field: "@timestamp" formats: - EEE MMM [ d][dd] HH:mm:ss uuuu XXX - grok: field: tmp_timestamp patterns: - "%{ISO8601_TIMEZONE:event.timezone}$" - rename: field: oracle.database_audit.privilege target_field: user.roles ignore_missing: true - rename: field: LENGTH target_field: oracle.database_audit.length ignore_missing: true - rename: field: oracle.database_audit.client_user target_field: client.user.name ignore_missing: true - rename: field: oracle.database_audit.client_address target_field: client.address ignore_missing: true - rename: field: oracle.database_audit.userhost target_field: server.address ignore_missing: true - rename: field: oracle.database_audit.database_user target_field: server.user.name ignore_missing: true - convert: field: oracle.database_audit.length type: long ignore_missing: true if: ctx.oracle?.database_audit != null - grok: field: client.address patterns: - "(?:%{IP:client.ip}|%{GREEDYDATA:client.domain})" ignore_failure: true ignore_missing: true - grok: field: server.address patterns: - "(?:%{IP:server.ip}|%{GREEDYDATA:server.domain})" ignore_failure: true ignore_missing: true # Renaming certain fields for better data structure - rename: field: oracle.database_audit.sessionid target_field: oracle.database_audit.session_id ignore_missing: true - rename: field: oracle.database_audit.client_terminal target_field: oracle.database_audit.client.terminal ignore_missing: true - rename: field: oracle.database_audit.client_address target_field: oracle.database_audit.client.address ignore_missing: true - rename: field: oracle.database_audit.database_user target_field: oracle.database_audit.database.user ignore_missing: true - rename: field: oracle.database_audit.userhost target_field: oracle.database_audit.database.host ignore_missing: true - rename: field: oracle.database_audit.dbid target_field: oracle.database_audit.database.id ignore_missing: true - rename: field: oracle.database_audit.entry_id target_field: oracle.database_audit.entry.id ignore_missing: true - geoip: field: client.ip target_field: client.geo ignore_missing: true - geoip: target_field: client.as properties: - asn - organization_name ignore_missing: true database_file: GeoLite2-ASN.mmdb field: client.ip - rename: target_field: client.as.number ignore_missing: true field: client.as.asn - rename: field: client.as.organization_name target_field: client.as.organization.name ignore_missing: true - geoip: field: server.ip target_field: server.geo ignore_missing: true - geoip: target_field: server.as properties: - asn - organization_name ignore_missing: true database_file: GeoLite2-ASN.mmdb field: server.ip - rename: target_field: server.as.number ignore_missing: true field: server.as.asn - rename: field: server.as.organization_name target_field: server.as.organization.name ignore_missing: true - geoip: field: source.ip target_field: source.geo ignore_missing: true - geoip: target_field: source.as properties: - asn - organization_name ignore_missing: true database_file: GeoLite2-ASN.mmdb field: source.ip - rename: target_field: source.as.number ignore_missing: true field: source.as.asn - rename: field: source.as.organization_name target_field: source.as.organization.name ignore_missing: true - append: value: "{{source.ip}}" if: ctx.source?.ip != null allow_duplicates: false field: related.ip - append: value: "{{client.ip}}" if: ctx.client?.ip != null allow_duplicates: false field: related.ip - append: value: "{{server.ip}}" if: ctx.server?.ip != null allow_duplicates: false field: related.ip - append: field: related.user value: "{{client.user.name}}" allow_duplicates: false if: ctx.client?.user?.name != null && ctx.client.user.name != '/' - append: field: related.user value: "{{server.user.name}}" allow_duplicates: false if: ctx.server?.user?.name != null && ctx.server.user.name != '/' - append: value: "{{server.domain}}" if: ctx.server?.domain != null allow_duplicates: false field: related.hosts - append: value: "{{client.domain}}" if: ctx.client?.domain != null allow_duplicates: false field: related.hosts - remove: field: - tmp_timestamp - audit - message - auth ignore_missing: true on_failure: - set: field: error.message value: "{{ _ingest.on_failure_message }}"

x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

efd6 · 2023-04-23T22:03:27Z

x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

+      replacement: "\"\\|"
+      ignore_missing: true
+  - kv:
+      field: auth
+      field_split: "\\|"


Are we confident that there is never a | in fields in auth?

Co-authored-by: Dan Kortschak <[email protected]>

efd6 · 2023-04-23T23:03:32Z

/test

efd6

Thanks

…elastic#35127)

…#35127)

added oracle authentication messages parsing

2ac28e9

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 19, 2023

mergify bot assigned leweafan Apr 19, 2023

leweafan added 3 commits April 19, 2023 15:36

added oracle new authentication messages parsing

55ae819

Updated for PR 35127

572f695

added oracle new fields

35766e5

leweafan marked this pull request as ready for review April 19, 2023 14:02

leweafan requested a review from a team as a code owner April 19, 2023 14:02

Merge branch 'main' into oracle_new

ee5d3aa

jamiehynds added the Team:Security-External Integrations label Apr 19, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 19, 2023

leweafan added 2 commits April 19, 2023 19:57

updated oracle database_audit pipeline

8ac9850

Merge branch 'main' into oracle_new

81ce302

efd6 reviewed Apr 19, 2023

View reviewed changes

leweafan and others added 5 commits April 20, 2023 14:20

updated oracle database_audit pipeline - added client.geo/as & server…

4fac406

….geo/as

Update x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

06a1a06

Co-authored-by: Dan Kortschak <[email protected]>

Update x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

ba248b5

Co-authored-by: Dan Kortschak <[email protected]>

updated oracle database_audit pipeline

02405cb

Merge branch 'main' into oracle_new

73e3265

efd6 reviewed Apr 20, 2023

View reviewed changes

leweafan added 4 commits April 21, 2023 01:56

changed oracle database_audit pipeline to yml format

7da1f7f

changed oracle database_audit pipeline to yml format

8b52ab4

updated oracle database_audit pipeline

dcd82cf

Merge branch 'main' into oracle_new

955ea30

efd6 reviewed Apr 23, 2023

View reviewed changes

leweafan and others added 3 commits April 24, 2023 01:42

Update x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

bafbff1

Co-authored-by: Dan Kortschak <[email protected]>

Update x-pack/filebeat/module/oracle/database_audit/ingest/pipeline.yml

4bdb896

Co-authored-by: Dan Kortschak <[email protected]>

Merge branch 'main' into oracle_new

dfa5355

efd6 approved these changes Apr 24, 2023

View reviewed changes

efd6 merged commit 185d823 into elastic:main Apr 24, 2023

leweafan added a commit to leweafan/beats that referenced this pull request Apr 28, 2023

x-pack/filebeat/module/oracle - Added authentication messages parsing (…

fce4e05

…elastic#35127)

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

x-pack/filebeat/module/oracle - Added authentication messages parsing (…

83a6b92

…#35127)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/module/oracle - Added authentication messages parsing #35127

x-pack/filebeat/module/oracle - Added authentication messages parsing #35127

leweafan commented Apr 19, 2023

mergify bot commented Apr 19, 2023

elasticmachine commented Apr 19, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 19, 2023

leweafan commented Apr 19, 2023

efd6 Apr 19, 2023

mergify bot commented Apr 20, 2023

leweafan commented Apr 20, 2023

efd6 Apr 20, 2023

leweafan Apr 20, 2023

efd6 Apr 20, 2023

leweafan Apr 20, 2023

efd6 Apr 21, 2023

efd6 Apr 21, 2023

leweafan Apr 21, 2023

efd6 Apr 23, 2023

leweafan Apr 23, 2023

efd6 commented Apr 23, 2023

efd6 left a comment

x-pack/filebeat/module/oracle - Added authentication messages parsing #35127

x-pack/filebeat/module/oracle - Added authentication messages parsing #35127

Conversation

leweafan commented Apr 19, 2023

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Apr 19, 2023

elasticmachine commented Apr 19, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Apr 19, 2023

leweafan commented Apr 19, 2023

Choose a reason for hiding this comment

mergify bot commented Apr 20, 2023

leweafan commented Apr 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 commented Apr 23, 2023

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 19, 2023 •

edited by jenkins-beats-ci bot

Loading