Skip to content

Commit

Permalink
Update filestream reader offset when line is skipped (elastic#23417)
Browse files Browse the repository at this point in the history
This PR adds two previously missing offset updates to the `filestream` reader when a line is skipped.

The offset could be incorrect if Filebeat skips the line if the line should not be published because of user configuration in `export_line` or `import_line`

If the offset is not updated in the reader, the state information of newer published events become incorrect. This might lead to duplicated events if Filebeat is restarted.

(cherry picked from commit e5cd64f)
  • Loading branch information
kvch committed Jan 21, 2021
1 parent a89dacc commit 1861631
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Filebeat*

- Update `filestream` reader offset when a line is skipped. {pull}23417[23417]
- cisco/asa fileset: Fix parsing of 302021 message code. {pull}14519[14519]
- Fix filebeat azure dashboards, event category should be `Alert`. {pull}14668[14668]
- Fixed dashboard for Cisco ASA Firewall. {issue}15420[15420] {pull}15553[15553]
Expand Down
4 changes: 2 additions & 2 deletions filebeat/input/filestream/input.go
Original file line number Diff line number Diff line change
Expand Up @@ -318,13 +318,13 @@ func (inp *filestream) readFromSource(
return nil
}

s.Offset += int64(message.Bytes)

if message.IsEmpty() || inp.isDroppedLine(log, string(message.Content)) {
continue
}

event := inp.eventFromMessage(message, path)
s.Offset += int64(message.Bytes)

if err := p.Publish(event, s); err != nil {
return err
}
Expand Down

0 comments on commit 1861631

Please sign in to comment.