Create fsnotify watcher only when starting file_integrity module #19505

vjsamuel · 2020-06-30T08:01:14Z

Bug

What does this PR do?

It makes the creation of an FS notify watcher lazy. It is only set up when the metricset starts up.

Why is it important?

When FIM module is used with autodiscover, the Check function is called too many times where fsnotify watchers end up leaking and keeps increasing the number of file handles. It eventually causes auditbeat OOM.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

elasticmachine · 2020-06-30T08:01:16Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-06-30T08:01:16Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-06-30T08:06:17Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [andrewkroh commented: jenkins, run tests]
Start Time: 2020-07-06T12:38:51.962+0000
Duration: 40 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	393
Skipped	68
Total	461

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 28 sec
- Start Time: 2020-07-06T13:09:29.776+0000
- log

elasticmachine · 2020-06-30T08:33:26Z

Pinging @elastic/siem (Team:SIEM)

exekias · 2020-06-30T08:33:51Z

auditbeat/module/file_integrity/eventreader_fsnotify.go

@@ -39,19 +39,19 @@ type reader struct {

 // NewEventReader creates a new EventProducer backed by fsnotify.
 func NewEventReader(c Config) (EventProducer, error) {
-	watcher, err := monitor.New(c.Recursive)


I wonder if this was here to detect config mistakes early, @andrewkroh probably you can confirm

I don't think much validation is gained with this particular instantiation from a config validation perspective. So moving it into Start should be OK.

andrewkroh

It sounds like this problem could affect any Metricbeat or Auditbeat module. The CheckConfig function instantiates modules (and hence metricsets), but it never invokes the optionally implemented mb.Closer on the metricsets that it creates. I expect that this produces a leak in more places than Auditbeat FIM.

If this correct then I think the best solution would be to fix the root of the problem and ensure that the CheckConfig implementation in Metricbeat always closes the MetricSets that it creates. Perhaps something like

diff --git a/metricbeat/mb/module/factory.go b/metricbeat/mb/module/factory.go
index d5b342afe..d35ef24e3 100644
--- a/metricbeat/mb/module/factory.go
+++ b/metricbeat/mb/module/factory.go
@@ -75,10 +75,13 @@ func (r *Factory) Create(p beat.PipelineConnector, c *common.Config) (cfgfile.Ru
 
 // CheckConfig checks if a config is valid or not
 func (r *Factory) CheckConfig(config *common.Config) error {
-       _, err := NewWrapper(config, mb.Registry, r.options...)
+       wrapper, err := NewWrapper(config, mb.Registry, r.options...)
        if err != nil {
                return err
        }
+       for _, ms := range wrapper.MetricSets() {
+               ms.close()
+       }
 
        return nil
 }

andrewkroh · 2020-06-30T19:53:38Z

I think there is also a potential resource leak within NewWrapper from initMetricSets when an error occurs. This needs to make sure that any MetricSets instantiated prior to the error are closed().

vjsamuel · 2020-07-01T05:49:15Z

ideally we need to add a Check(*common.Config) error to the metricset interface. This will allow modules to optionally do an unmarshal of the config and do Validate() on the config if the config implements the same.

andrewkroh

I'm OK with this change. But the root cause still needs to be addressed otherwise other modules could be leaking resources with every CheckConfig.

andrewkroh · 2020-07-01T23:21:56Z

auditbeat/module/file_integrity/eventreader_fsnotify.go

@@ -39,19 +39,19 @@ type reader struct {

 // NewEventReader creates a new EventProducer backed by fsnotify.
 func NewEventReader(c Config) (EventProducer, error) {
-	watcher, err := monitor.New(c.Recursive)


I don't think much validation is gained with this particular instantiation from a config validation perspective. So moving it into Start should be OK.

andrewkroh · 2020-07-01T23:28:07Z

jenkins, run tests

andrewkroh · 2020-07-06T12:38:21Z

jenkins, run tests

…stic#19505) * Create fsnotify watcher only when starting file_integrity module * Add close to Start() so that failed starts don't create resource leaks

vjsamuel requested a review from a team as a code owner June 30, 2020 08:01

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 30, 2020

exekias added enhancement Team:SIEM labels Jun 30, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 30, 2020

exekias reviewed Jun 30, 2020

View reviewed changes

vjsamuel force-pushed the fix_watcher_leak branch from ac701a4 to c0c3de7 Compare June 30, 2020 15:52

andrewkroh reviewed Jun 30, 2020

View reviewed changes

vjsamuel force-pushed the fix_watcher_leak branch from c0c3de7 to 042a502 Compare July 1, 2020 05:46

vjsamuel added 2 commits July 1, 2020 16:15

Create fsnotify watcher only when starting file_integrity module

ae44ee5

Add close to Start() so that failed starts don't create resource leaks

f293c68

vjsamuel force-pushed the fix_watcher_leak branch from 042a502 to f293c68 Compare July 1, 2020 23:15

andrewkroh reviewed Jul 1, 2020

View reviewed changes

andrewkroh approved these changes Jul 6, 2020

View reviewed changes

exekias merged commit de897b0 into elastic:master Jul 7, 2020

vjsamuel deleted the fix_watcher_leak branch July 8, 2020 05:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create fsnotify watcher only when starting file_integrity module #19505

Create fsnotify watcher only when starting file_integrity module #19505

vjsamuel commented Jun 30, 2020 •

edited

Loading

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jun 30, 2020

exekias Jun 30, 2020

andrewkroh Jul 1, 2020

andrewkroh left a comment •

edited

Loading

andrewkroh commented Jun 30, 2020 •

edited

Loading

vjsamuel commented Jul 1, 2020

andrewkroh left a comment

andrewkroh Jul 1, 2020

andrewkroh commented Jul 1, 2020

andrewkroh commented Jul 6, 2020

Create fsnotify watcher only when starting file_integrity module #19505

Create fsnotify watcher only when starting file_integrity module #19505

Conversation

vjsamuel commented Jun 30, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

elasticmachine commented Jun 30, 2020

exekias Jun 30, 2020

Choose a reason for hiding this comment

andrewkroh Jul 1, 2020

Choose a reason for hiding this comment

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

andrewkroh commented Jun 30, 2020 • edited Loading

vjsamuel commented Jul 1, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Jul 1, 2020

Choose a reason for hiding this comment

andrewkroh commented Jul 1, 2020

andrewkroh commented Jul 6, 2020

vjsamuel commented Jun 30, 2020 •

edited

Loading

elasticmachine commented Jun 30, 2020 •

edited

Loading

andrewkroh left a comment •

edited

Loading

andrewkroh commented Jun 30, 2020 •

edited

Loading