Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Filebeat AWS module cloudtrail fileset not skip the digest log #20972

Closed
sophiaxu8 opened this issue Sep 4, 2020 · 3 comments
Closed
Assignees
Labels

Comments

@sophiaxu8
Copy link
Contributor

  • Version: 7.9.0
  • Operating System:
  • Discuss Forum URL:
  • Steps to Reproduce:

Currently the path of cloudtrail is as follows:
source_vpc -> s3 -> SNS -> SQS -> filebeat -> elasticsearch

There are some errors related to the cloudtrail digest log in filebeat log:

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:562 key 'Records' not found

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:464 decodeJSONWithKey failed for 'AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz' from S3 bucket 'XXXX-cloudtrail-logs': key 'Records' not found

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:396 createEventsFromS3Info failed processing file from s3 bucket "XXXX-cloudtrail-logs" with name "AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz": decodeJSONWithKey failed for 'AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz' from S3 bucket 'XXXX-cloudtrail-logs': key 'Records' not found

As mentioned in the document for filebeat AWS module cloudtrail fileset :

The cloudtrail fileset does not read the CloudTrail Digest files that are delivered to the S3 bucket when Log File Integrity is turned on, it only reads the CloudTrail logs.

But filebeat log shows it try to parse the digest log and failed, the expect behaviour is to skip the digest log.

Here is the filebeat.yml config:

setup.dashboards.enabled: false
setup.template.enabled: false
setup.ilm.enabled: false
setup.template.name: ${INDEX}
setup.template.pattern: ${INDEX}-*

monitoring:
  enabled: yes
  elasticsearch:
    hosts:
      - ${ES_SERVER}
    password: ${BEATS_PASSWORD}
    username: ${BEATS_USERNAME}

filebeat.modules:
  - module: aws
    elb:
      enabled: false
    s3access:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    vpcflow:
      enabled: false
    cloudtrail:
      enabled: true
      var.queue_url: ${QUEUE_URL:}
      var.role_arn: ${ROLE_ARN:}
      var.visibility_timeout: ${VISIBILITY_TIMEOUT:300s}

processors:
  - add_fields:
      target: ''
      fields:
        labels.owners:
          - marty.chong
          - platsec
        labels.description: Cloudtrail logs

logging.level: ${LOGGING_LEVEL:info}

output.elasticsearch:
  index: ${INDEX}
  hosts: '${ES_SERVER}'
  username: '${ES_USERNAME}'
  password: '${ES_PASSWORD}'
  ssl.enabled: ${SSL_ENABLED:true}
  ssl.verification_mode: ${SSL_VERIFICATION_MODE:full}
~                                                      
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2020
@leehinman
Copy link
Contributor

Adding Prefix AWSLogs/$AWS_ACCOUNT_ID/CloudTrail/ to the S3 notification will make it so only notifications for the CloudTrail logs are sent to the SQS queue and processed by Filebeat.

For some this may not work because there are too many AWS_ACCOUNT_IDs that are writing to the same S3 bucket.

@leehinman leehinman self-assigned this Sep 4, 2020
@leehinman
Copy link
Contributor

closed by #21086

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants