Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[filebeat][gcs] - Added support for new features and removed partial save mechanism #36713

Merged
merged 13 commits into from
Oct 3, 2023
Merged

[filebeat][gcs] - Added support for new features and removed partial save mechanism #36713

merged 13 commits into from
Oct 3, 2023

Conversation

ShourieG
Copy link
Contributor

@ShourieG ShourieG commented Oct 2, 2023

Type of change

  • Enhancement
  • Docs

Proposed commit message

Added support for the following features as part of a planned enhancement update:

  1. File selectors support similar to aws-s3 input, but only supports regex matching atm.
  2. A date filter for blobs based on unix epoch (seconds) format.
  3. Expand event list from field similar to aws-s3 input.
  4. Removed partial save mechanism for now due to concurrency issues.

These additions help make the input more robust for further use.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@ShourieG ShourieG requested a review from a team as a code owner October 2, 2023 07:37
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 2, 2023
@mergify
Copy link
Contributor

mergify bot commented Oct 2, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@ShourieG ShourieG added Team:Security-External Integrations and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 2, 2023
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@ShourieG ShourieG added Filebeat Filebeat 8.11-candidate backport-skip Skip notification from the automated backport with mergify labels Oct 2, 2023
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 2, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Duration: 74 min 50 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

bhapas
bhapas reviewed Oct 2, 2023
View reviewed changes
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/config.go Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/scheduler.go Outdated Show resolved Hide resolved
@ShourieG ShourieG enabled auto-merge (squash) October 2, 2023 10:51
@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 2, 2023

@bhapas could you approve if all else looks ok. Addressed all the comments, and have set the pr to auto-merge as I will be going out now.

@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 2, 2023

@bhapas is it ok for an approval now ?

@bhapas
Copy link
Contributor

bhapas commented Oct 2, 2023

@bhapas is it ok for an approval now ?

@ShourieG LGTM. But wanted a second review from an experienced dev as this is a big change

@ShourieG ShourieG disabled auto-merge October 2, 2023 14:56
@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 2, 2023

@bhapas I also agree, but I'm worried it might miss FF. Can we keep it approved, while I look for a senior dev to review this. It similar to the azure blob pr which was reviewed by Dan and Andrew.

@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 2, 2023

@andrewkroh, could you give a quick review for this PR, trying to meet FF for tomorrow.

bhapas
bhapas approved these changes Oct 2, 2023
View reviewed changes
Copy link
Contributor

@bhapas bhapas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

efd6
efd6 reviewed Oct 2, 2023
View reviewed changes
CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-gcs.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/config.go Show resolved Hide resolved
x-pack/filebeat/input/gcs/input.go Outdated Show resolved Hide resolved
@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 3, 2023

@efd6, Have resolved the PR suggestions.

efd6
efd6 approved these changes Oct 3, 2023
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comments are informational only; no action is required.

x-pack/filebeat/input/gcs/scheduler.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Show resolved Hide resolved
x-pack/filebeat/input/gcs/job.go Outdated Show resolved Hide resolved
@ShourieG ShourieG enabled auto-merge (squash) October 3, 2023 09:20
@ShourieG ShourieG merged commit 8818aca into elastic:main Oct 3, 2023
8 checks passed
@ShourieG ShourieG deleted the gcs/feature branch October 3, 2023 10:51
@ShourieG ShourieG mentioned this pull request Oct 10, 2023
Closed
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@ShourieG @Scholar-Li
[filebeat][gcs] - Added support for new features and removed partial …
830d2bc 
…save mechanism (elastic#36713)

* removed partial save mechanism

* added support for timestamp filtering, expand event list & fileselectors. Updated docs

* resolved linter issues

* updated changelog

* fixed asciidoc errors

* addressed PR comments

* removed redundant nil check for fileselector regex

* addressed PR suggestions

* updated comments and resolved pr suggestions

* updated logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 approved these changes

@bhapas bhapas bhapas approved these changes

Assignees

@ShourieG ShourieG

Labels
8.11-candidate backport-skip Skip notification from the automated backport with mergify enhancement Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improvements to the gcs input
4 participants
@ShourieG @elasticmachine @bhapas @efd6