Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat AWS Module (S3 Access Log) Grok Pattern Issue #26913

Closed
jsealover opened this issue Jul 15, 2021 · 3 comments · Fixed by #26940
Closed

Filebeat AWS Module (S3 Access Log) Grok Pattern Issue #26913

jsealover opened this issue Jul 15, 2021 · 3 comments · Fixed by #26940
Labels
backport-v7.15.0 Automated backport with mergify bug Filebeat Filebeat

Comments

@jsealover
Copy link

jsealover commented Jul 15, 2021

PUT requests in AWS S3 access logs may include a hyphen in place of an IP address, so the %{IP:aws.s3access.remote_ip} grok pattern for the message field defined in filebeat-7.13.3-aws-s3access-pipeline results in the following error: “Provided Grok expressions do not match field value”.

Modifying the pattern to (?:-|%{IP:aws.s3access.remote_ip}) resolves the issue.

  • Version: 7.13.3
  • Operating System: Amazon Linux
  • Discuss Forum URL:
  • Steps to Reproduce:
POST _ingest/pipeline/filebeat-7.13.3-aws-s3access-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": """67797214d75628047d9c76b18a78cded1a4b069b71f2a9d5a53649c38da8770b flow-log-test [14/Jul/2021:18:57:31 +0000] - svc:delivery.logs.amazonaws.com MVGXZXEVN3IG9S24 REST.PUT.OBJECT AWSLogs/000000000000/vpcflowlogs/us-gov-east-1/2021/07/13/000000000000_vpcflowlogs_us-gov-east-1_fl-_20210713T1855Z_f12aa632.log.gz "PUT /AWSLogs/000000000000/vpcflowlogs/us-gov-east-1/2021/07/13/000000000000_vpcflowlogs_us-gov-east-1_fl-0e7c13bf00cf15bfe_20210713T1855Z_f12aa632.log.gz HTTP/1.1" 200 - - 773 103 13 "-" "-" - 02SxwfXpO5UysN0GsKGa3uGDQ6E/W7+Hwo/luRH8p1VEexULoe66RCM+nja0dEq2JqLrtgjocvVRRkVt4= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader flow-log-test.s3.us-gov-west-1.amazonaws.com TLSv1.2 -
        """
      }
    }
  ]
}
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 15, 2021
@jsealover jsealover added the Filebeat Filebeat label Jul 15, 2021
@legoguy1000
Copy link
Contributor

That's an easy fix. I'll add it to the my list tomorrow.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 16, 2021
legoguy1000 added a commit to legoguy1000/beats that referenced this issue Jul 16, 2021
@legoguy1000
Copy link
Contributor

PR Submitted

@andrewkroh andrewkroh added the backport-v7.15.0 Automated backport with mergify label Jul 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.15.0 Automated backport with mergify bug Filebeat Filebeat
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants