Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Automatic Import] Add support for handling unstructured syslog samples #192817

Merged
merged 5 commits into from
Sep 17, 2024
Merged

[Automatic Import] Add support for handling unstructured syslog samples #192817

merged 5 commits into from
Sep 17, 2024

Conversation

bhapas
Copy link
Contributor

@bhapas bhapas commented Sep 13, 2024 •
edited
Loading

Release note

Adds a feature to add support for handling syslogs with unsupported message body.

Summary

This PR handles the unstructured syslog samples in Automatic Import.

Examples of unstructured samples would be:

<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8
<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5
<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3
video1656541084.mp4

Checklist

For maintainers

@bhapas bhapas added release_note:enhancement enhancement New value added to drive a business result 8.16 candidate Team:Security-Scalability Team label for Security Integrations Scalability Team labels Sep 13, 2024
@bhapas bhapas self-assigned this Sep 13, 2024
@bhapas bhapas marked this pull request as ready for review September 16, 2024 09:28
@bhapas bhapas requested a review from a team as a code owner September 16, 2024 09:28
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-scalability (Team:Security-Scalability)

@bhapas bhapas added backport:prev-major Backport to (8.x, 8.15) the previous major branch and all later branches still in development and removed 8.16 candidate labels Sep 16, 2024
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @bhapas

@bhapas bhapas merged commit 77fe423 into elastic:main Sep 17, 2024
20 checks passed
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 17, 2024
@bhapas
[Automatic Import] Add support for handling unstructured syslog sampl…
72b60fd 
…es (elastic#192817)

## Summary

This PR handles the `unstructured` syslog samples in Automatic Import.

Examples of unstructured samples would be:

```
<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8
<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5
<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3
```

https://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 77fe423)
@kibanamachine kibanamachine mentioned this pull request Sep 17, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 17, 2024
@bhapas
[Automatic Import] Add support for handling unstructured syslog sampl…
44952cf 
…es (elastic#192817)

## Summary

This PR handles the `unstructured` syslog samples in Automatic Import.

Examples of unstructured samples would be:

```
<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8
<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5
<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3
```

https://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 77fe423)
@kibanamachine kibanamachine mentioned this pull request Sep 17, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.15
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Sep 17, 2024
@kibanamachine @bhapas
[8.x] [Automatic Import] Add support for handling unstructured syslog…
146ae69 
… samples (#192817) (#193159)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Add support for handling unstructured syslog
samples (#192817)](#192817)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-17T12:28:01Z","message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","enhancement","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Add support for handling unstructured syslog
samples","number":192817,"url":"https://github.com/elastic/kibana/pull/192817","mergeCommit":{"message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192817","number":192817,"mergeCommit":{"message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <[email protected]>
kibanamachine added a commit that referenced this pull request Sep 17, 2024
@kibanamachine @bhapas
[8.15] [Automatic Import] Add support for handling unstructured syslo…
74d5d30 
…g samples (#192817) (#193158)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Automatic Import] Add support for handling unstructured syslog
samples (#192817)](#192817)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-17T12:28:01Z","message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","enhancement","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Add support for handling unstructured syslog
samples","number":192817,"url":"https://github.com/elastic/kibana/pull/192817","mergeCommit":{"message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192817","number":192817,"mergeCommit":{"message":"[Automatic
Import] Add support for handling unstructured syslog samples
(#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog
samples in Automatic Import.\r\n\r\nExamples of unstructured samples
would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root'
failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su
root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine
su: 'su root' failed for otheruser on
/dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <[email protected]>
markov00 pushed a commit to markov00/kibana that referenced this pull request Sep 18, 2024
@bhapas @markov00
[Automatic Import] Add support for handling unstructured syslog sampl…
cd37704 
…es (elastic#192817)

## Summary

This PR handles the `unstructured` syslog samples in Automatic Import.

Examples of unstructured samples would be:

```
<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8
<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5
<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3
```


https://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@bhapas bhapas mentioned this pull request Sep 26, 2024
Closed
@zez3
Copy link

zez3 commented Sep 28, 2024

Hey @bhapas

Are aware if there is any plan to support beat processors as alternative to elasticsearch processors?

@bhapas
Copy link
Contributor Author

bhapas commented Sep 30, 2024

Hey @bhapas

Are aware if there is any plan to support beat processors as alternative to elasticsearch processors?

@zez3 Integrations support beat processors in the integration configuration. But Automatic Import currently generates Elastic Agent based integrations which always go through elasticsearch ingest pipelines for writing docs into ES.

Do you have any usecase that you are looking to solve?

@zez3
Copy link

zez3 commented Sep 30, 2024 •
edited
Loading

Do you have any usecase that you are looking to solve?

Yes dns beats processor are not available on the elasticsearch processors. We ingest syslogs and the log.source.ip is reverse resolved to get the hostname

@bhapas
Copy link
Contributor Author

bhapas commented Sep 30, 2024

Yes dns beats processor are not available on the elasticsearch processors. We ingest syslogs and the log.source.ip is resolved to get the hostname

@zez3 You can use https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html in your integration to achieve this.

@zez3
Copy link

zez3 commented Oct 1, 2024

@zez3 You can use https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html in your integration to achieve this.

This is exactly what I do.
My argument was that perhaps the automatic import should support(optional) to use the beats processors

@bhapas
Copy link
Contributor Author

bhapas commented Oct 1, 2024

@zez3 This is not something in the near product roadmap. But I shall take this in to check the possibility of having this in roadmap. Thanks for raising.

@zez3
Copy link

zez3 commented Oct 1, 2024

@jamiehynds ^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kgeller kgeller kgeller approved these changes

@haetamoudi haetamoudi haetamoudi approved these changes

Assignees

@bhapas bhapas

Labels
backport:prev-major Backport to (8.x, 8.15) the previous major branch and all later branches still in development enhancement New value added to drive a business result Feature:AutomaticImport release_note:enhancement Team:Security-Scalability Team label for Security Integrations Scalability Team v8.15.2 v8.16.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@bhapas @elasticmachine @kibana-ci @kibanamachine @zez3 @kgeller @haetamoudi