Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add datastream to collect pod logs in kubernetes integration #1523

Merged
merged 8 commits into from
Aug 24, 2021
Merged

Add datastream to collect pod logs in kubernetes integration #1523

merged 8 commits into from
Aug 24, 2021

Conversation

MichaelKatsoulis
Copy link
Contributor

@MichaelKatsoulis MichaelKatsoulis commented Aug 12, 2021 •
edited
Loading

What does this PR do?

This PR adds a new data stream in kubernetes integration to collect pods logs.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

How to test this PR locally

Adding Kubernetes integration to a policy or just kube-pod-logs integration automatically
starts collecting pods logs.

Related issues

Screenshots

integrations-view
container-logs-integration
configure-integration

nginx-logs

@MichaelKatsoulis MichaelKatsoulis marked this pull request as draft August 12, 2021 10:50
@MichaelKatsoulis MichaelKatsoulis added Team:Integrations Label for the Integrations team enhancement New feature or request labels Aug 12, 2021
@elasticmachine
Copy link

elasticmachine commented Aug 12, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-24T09:48:20.074+0000

  • Duration: 31 min 8 sec

  • Commit: f0f8e23

Test stats 🧪

Test Results
Failed 0
Passed 113
Skipped 0
Total 113

Trends 🧪

Image of Build Times

Image of Tests

ChrsMark
ChrsMark reviewed Aug 23, 2021
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me and it should be a fair equivalent to https://github.com/elastic/beats/blob/3e2dcbea48e7a008ad981a69b0ad8d274e71350a/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L216-L230.

My only concern is about naming: Is kube-pod-logs ok or it should by just pod-logs (or even container-logs)? I'm afraid that by kube-pod-logs one could thing that we refer to logs from pods of the control plane like controller etc. Product team @mukeshelastic @akshay-saraswat can provide some guidance here?

@mukeshelastic
Copy link

I recommend container-logs because that is literally what we have.. Let's drop kube from the name because what @ChrsMark mentioned as potential confusion between control plane logs.

@MichaelKatsoulis MichaelKatsoulis marked this pull request as ready for review August 24, 2021 08:53
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@MichaelKatsoulis
Copy link
Contributor Author

@ChrsMark , @mukeshelastic I changed the name to container-logs. It is ready for review now!

ChrsMark
ChrsMark requested changes Aug 24, 2021
View reviewed changes
packages/kubernetes/_dev/build/docs/README.md Outdated
@@ -78,6 +78,11 @@ These datasets are not enabled by default.
Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on
the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets.

#### kube-pod-logs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems out dated

packages/kubernetes/docs/README.md Outdated
@@ -78,6 +78,11 @@ These datasets are not enabled by default.
Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on
the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets.

#### kube-pod-logs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems out dated too

ChrsMark
ChrsMark approved these changes Aug 24, 2021
View reviewed changes
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@MichaelKatsoulis MichaelKatsoulis merged commit 81fe759 into elastic:master Aug 24, 2021
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
@MichaelKatsoulis
Add datastream to collect pod logs in kubernetes integration (elastic…
a474b54 
…#1523)

* Add new container-logs datastream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

@jsoriano jsoriano Awaiting requested review from jsoriano

Assignees
No one assigned
Labels
enhancement New feature or request Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add data stream to collect pods logs in kubernetes integration
4 participants
@MichaelKatsoulis @elasticmachine @mukeshelastic @ChrsMark