Filebeat modules merge behaviour does weird things

[philippkahr]

• Version: 7.14.1
• Operating System: macOS

Expected behavior

1. I explicitly configure one dataset from a module (e.g. Cisco, dataset: asa)
2. Filebeat ignores all other datasets from a module. (e.g. Cisco, dataset: iOS, nexus, amp, ...)

Current behavior and steps to reproduce

1. Copy the following filebeat.yml

filebeat.yml

```yml filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false filebeat.modules: - module: cisco asa: enabled: true setup.template.settings: index.number_of_shards: 1 output.elasticsearch: hosts: ["localhost:9200"] ```

2. Check if Cisco.yml.disabled exists inside the modules.d directory
3. Start filebeat -e
4. Observe that all datasets e.g. ios, nexus and more are started. Especially check for the amp dataset, as this does some HTTP requests that fail.

Log

```text 2021-09-13T11:56:53.206+0200 INFO instance/beat.go:665 Home path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64] Config path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64] Data path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data] Logs path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/logs] 2021-09-13T11:56:53.249+0200 INFO instance/beat.go:673 Beat ID: fe6edae4-ec39-4b37-8fe3-8de4921dd814 2021-09-13T11:56:53.250+0200 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "data": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data", "home": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "logs": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/logs"}, "type": "filebeat", "uuid": "fe6edae4-ec39-4b37-8fe3-8de4921dd814"}}} 2021-09-13T11:56:53.250+0200 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "703d589a09cfdbfd7f84c1d990b50b6b7f62ac29", "libbeat": "7.14.1", "time": "2021-08-26T09:12:57.000Z", "version": "7.14.1"}}} 2021-09-13T11:56:53.250+0200 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":12,"version":"go1.16.6"}}} 2021-09-13T11:56:53.251+0200 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-01T08:14:37.525318+02:00","name":"Philipps-Elastic-Macbook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::aede:48ff:fe00:1122/64","fe80::431:f3ad:c430:145c/64","192.168.0.121/24","fe80::a043:60ff:feb1:4efe/64","fe80::a043:60ff:feb1:4efe/64","fe80::a964:2842:34be:262d/64","fe80::48b0:580a:68d7:daba/64","fe80::df3a:2a98:ae06:857a/64","fe80::47c3:26a8:c91b:26cd/64","fe80::a4ed:6921:380f:3305/64","fe80::673:a070:759e:7394/64","fe80::6e:334e:1664:aae2/64","169.254.112.88/16"],"kernel_version":"20.6.0","mac":["ac:de:48:00:11:22","aa:66:5a:03:6a:54","88:66:5a:03:6a:54","a2:43:60:b1:4e:fe","a2:43:60:b1:4e:fe","82:87:18:83:94:01","82:87:18:83:94:00","82:87:18:83:94:05","82:87:18:83:94:04","82:87:18:83:94:01","f2:79:60:d8:fb:c7"],"os":{"type":"macos","family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.16","major":10,"minor":16,"patch":0,"build":"20G95"},"timezone":"CEST","timezone_offset_sec":7200,"id":"029BBEE6-1ACC-565A-ACDC-7373F4F07AAC"}}} 2021-09-13T11:56:53.252+0200 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"cwd": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "exe": "./filebeat", "name": "filebeat", "pid": 32720, "ppid": 32243, "start_time": "2021-09-13T11:56:51.720+0200"}}} 2021-09-13T11:56:53.252+0200 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.14.1 2021-09-13T11:56:53.252+0200 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.14.1' as ILM is enabled. 2021-09-13T11:56:53.253+0200 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200 2021-09-13T11:56:53.254+0200 INFO [publisher] pipeline/module.go:113 Beat name: Philipps-Elastic-Macbook-Pro.local 2021-09-13T11:56:53.265+0200 INFO beater/filebeat.go:117 Enabled modules/filesets: cisco (amp, asa, ftd, ios, meraki, nexus, umbrella), () 2021-09-13T11:56:53.273+0200 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s 2021-09-13T11:56:53.273+0200 INFO instance/beat.go:473 filebeat start running. 2021-09-13T11:56:53.314+0200 INFO memlog/store.go:119 Loading data file of '/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data/registry/filebeat' succeeded. Active transaction id=0 2021-09-13T11:56:53.314+0200 INFO memlog/store.go:124 Finished loading transaction log file for '/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data/registry/filebeat'. Active transaction id=0 2021-09-13T11:56:53.319+0200 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0 2021-09-13T11:56:53.320+0200 INFO [crawler] beater/crawler.go:71 Loading Inputs: 7 2021-09-13T11:56:53.435+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.435+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.435+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.435+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.436+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.436+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.436+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 16891225278044354029) 2021-09-13T11:56:53.436+0200 INFO udp/input.go:99 Starting UDP input 2021-09-13T11:56:53.436+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T11:56:53.436+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 2783938500818514675) 2021-09-13T11:56:53.436+0200 INFO [input.aws-s3] compat/compat.go:111 Input aws-s3 starting {"id": "26A2895109D232F3"} 2021-09-13T11:56:53.437+0200 ERROR [input.aws-s3] awss3/input.go:95 getRegionFromQueueURL failed: QueueURL is not in format: https://sqs.{REGION_ENDPOINT}.{ENDPOINT}/{ACCOUNT_NUMBER}/{QUEUE_NAME} {"id": "26A2895109D232F3", "queue_url": ""} 2021-09-13T11:56:53.437+0200 ERROR [input.aws-s3] compat/compat.go:122 Input 'aws-s3' failed with: getRegionFromQueueURL failed: QueueURL is not in format: https://sqs.{REGION_ENDPOINT}.{ENDPOINT}/{ACCOUNT_NUMBER}/{QUEUE_NAME} {"id": "26A2895109D232F3"} 2021-09-13T11:56:53.439+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 12134056745465751135) 2021-09-13T11:56:53.439+0200 INFO [input.httpjson-cursor] compat/compat.go:111 Input httpjson-cursor starting {"id": "A864D4522DEB265F"} 2021-09-13T11:56:53.439+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 11000608890507230499) 2021-09-13T11:56:53.439+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 1392379934275647788) 2021-09-13T11:56:53.439+0200 INFO udp/input.go:99 Starting UDP input 2021-09-13T11:56:53.439+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T11:56:53.439+0200 INFO udp/input.go:99 Starting UDP input 2021-09-13T11:56:53.439+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T11:56:53.441+0200 INFO [input.httpjson-cursor] v2/input.go:112 Process another repeated request. {"id": "A864D4522DEB265F", "input_source": "https://api.amp.cisco.com/v1/events?offset=0&limit=300", "input_url": "https://api.amp.cisco.com/v1/events?offset=0&limit=300"} 2021-09-13T11:56:53.444+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 1817941144496107359) 2021-09-13T11:56:53.444+0200 INFO [syslog] syslog/input.go:147 Starting Syslog input {"protocol": "udp"} 2021-09-13T11:56:53.444+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 WARN [cfgwarn] registered_domain/registered_domain.go:61 BETA: The registered_domain processor is beta. 2021-09-13T11:56:53.472+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 17274751607468830697) 2021-09-13T11:56:53.473+0200 INFO udp/input.go:99 Starting UDP input 2021-09-13T11:56:53.473+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T11:56:53.473+0200 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 7 2021-09-13T11:56:53.473+0200 INFO cfgfile/reload.go:164 Config reloader started 2021-09-13T11:56:53.474+0200 INFO cfgfile/reload.go:224 Loading of config files completed. 2021-09-13T11:56:54.458+0200 ERROR [input.httpjson-cursor] v2/input.go:115 Error while processing http request: failed to execute http client.Do: failed to execute http client.Do: http request was unsuccessful with a status code 401 {"id": "A864D4522DEB265F", "input_source": "https://api.amp.cisco.com/v1/events?offset=0&limit=300", "input_url": "https://api.amp.cisco.com/v1/events?offset=0&limit=300"} ^C2021-09-13T11:57:03.638+0200 INFO beater/filebeat.go:515 Stopping filebeat 2021-09-13T11:57:03.638+0200 INFO beater/crawler.go:148 Stopping Crawler 2021-09-13T11:57:03.638+0200 INFO beater/crawler.go:158 Stopping 7 inputs 2021-09-13T11:57:03.638+0200 INFO cfgfile/reload.go:227 Dynamic config reloader stopped 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 1817941144496107359 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 2783938500818514675 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 11000608890507230499 2021-09-13T11:57:03.638+0200 INFO input/input.go:136 input ticker stopped 2021-09-13T11:57:03.638+0200 INFO [input.aws-s3] compat/compat.go:132 Input 'aws-s3' stopped {"id": "26A2895109D232F3"} 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 17274751607468830697 2021-09-13T11:57:03.638+0200 INFO [syslog] syslog/input.go:167 Stopping Syslog input 2021-09-13T11:57:03.638+0200 INFO input/input.go:136 input ticker stopped 2021-09-13T11:57:03.638+0200 INFO input/input.go:136 input ticker stopped 2021-09-13T11:57:03.638+0200 INFO udp/input.go:114 Stopping UDP input 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 12134056745465751135 2021-09-13T11:57:03.638+0200 INFO [udp] dgram/handler.go:73 Connection has been closed {"address": "localhost:9541"} 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 16891225278044354029 2021-09-13T11:57:03.638+0200 INFO [input.httpjson-cursor] compat/compat.go:132 Input 'httpjson-cursor' stopped {"id": "A864D4522DEB265F"} 2021-09-13T11:57:03.638+0200 INFO udp/input.go:114 Stopping UDP input 2021-09-13T11:57:03.638+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 1392379934275647788 2021-09-13T11:57:03.638+0200 INFO input/input.go:136 input ticker stopped 2021-09-13T11:57:03.638+0200 INFO [input.httpjson-cursor] v2/input.go:131 Input stopped because context was cancelled with: context canceled {"id": "A864D4522DEB265F", "input_source": "https://api.amp.cisco.com/v1/events?offset=0&limit=300", "input_url": "https://api.amp.cisco.com/v1/events?offset=0&limit=300"} 2021-09-13T11:57:03.638+0200 INFO udp/input.go:114 Stopping UDP input 2021-09-13T11:57:03.638+0200 INFO input/input.go:136 input ticker stopped 2021-09-13T11:57:03.638+0200 INFO [udp] dgram/handler.go:73 Connection has been closed {"address": "localhost:9001"} 2021-09-13T11:57:03.638+0200 INFO [udp] dgram/handler.go:73 Connection has been closed {"address": "localhost:9002"} 2021-09-13T11:57:03.638+0200 INFO [input.httpjson-cursor] compat/compat.go:124 Input 'httpjson-cursor' stopped {"id": "A864D4522DEB265F"} 2021-09-13T11:57:03.638+0200 INFO udp/input.go:114 Stopping UDP input 2021-09-13T11:57:03.638+0200 INFO [udp] dgram/handler.go:73 Connection has been closed {"address": "localhost:9528"} 2021-09-13T11:57:03.638+0200 INFO [udp] dgram/handler.go:73 Connection has been closed {"address": "localhost:9003"} 2021-09-13T11:57:03.638+0200 INFO beater/crawler.go:178 Crawler stopped 2021-09-13T11:57:03.638+0200 INFO [registrar] registrar/registrar.go:132 Stopping Registrar 2021-09-13T11:57:03.638+0200 INFO [registrar] registrar/registrar.go:166 Ending Registrar 2021-09-13T11:57:03.638+0200 INFO [registrar] registrar/registrar.go:137 Registrar stopped ```

5. Now modify the filebeat.yml again to that

filebeat.yml with modules disabled

```yml filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false filebeat.modules: - module: cisco asa: enabled: true ftd: enabled: false ios: enabled: false nexus: enabled: false meraki: enabled: false umbrella: enabled: false amp: enabled: false setup.template.settings: index.number_of_shards: 1 output.elasticsearch: hosts: ["localhost:9200"] ```

6. See that the logs do not contain any failed http requests or module loading anymore.

logs

``` 2021-09-13T12:00:58.698+0200 INFO instance/beat.go:665 Home path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64] Config path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64] Data path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data] Logs path: [/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/logs] 2021-09-13T12:00:58.699+0200 INFO instance/beat.go:673 Beat ID: fe6edae4-ec39-4b37-8fe3-8de4921dd814 2021-09-13T12:00:58.700+0200 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "data": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data", "home": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "logs": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/logs"}, "type": "filebeat", "uuid": "fe6edae4-ec39-4b37-8fe3-8de4921dd814"}}} 2021-09-13T12:00:58.701+0200 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "703d589a09cfdbfd7f84c1d990b50b6b7f62ac29", "libbeat": "7.14.1", "time": "2021-08-26T09:12:57.000Z", "version": "7.14.1"}}} 2021-09-13T12:00:58.701+0200 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":12,"version":"go1.16.6"}}} 2021-09-13T12:00:58.701+0200 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-01T08:14:37.525318+02:00","name":"Philipps-Elastic-Macbook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::aede:48ff:fe00:1122/64","fe80::431:f3ad:c430:145c/64","192.168.0.121/24","fe80::a043:60ff:feb1:4efe/64","fe80::a043:60ff:feb1:4efe/64","fe80::a964:2842:34be:262d/64","fe80::48b0:580a:68d7:daba/64","fe80::df3a:2a98:ae06:857a/64","fe80::47c3:26a8:c91b:26cd/64","fe80::a4ed:6921:380f:3305/64","fe80::673:a070:759e:7394/64","fe80::6e:334e:1664:aae2/64","169.254.112.88/16"],"kernel_version":"20.6.0","mac":["ac:de:48:00:11:22","aa:66:5a:03:6a:54","88:66:5a:03:6a:54","a2:43:60:b1:4e:fe","a2:43:60:b1:4e:fe","82:87:18:83:94:01","82:87:18:83:94:00","82:87:18:83:94:05","82:87:18:83:94:04","82:87:18:83:94:01","f2:79:60:d8:fb:c7"],"os":{"type":"macos","family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.16","major":10,"minor":16,"patch":0,"build":"20G95"},"timezone":"CEST","timezone_offset_sec":7200,"id":"029BBEE6-1ACC-565A-ACDC-7373F4F07AAC"}}} 2021-09-13T12:00:58.701+0200 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"cwd": "/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64", "exe": "./filebeat", "name": "filebeat", "pid": 33057, "ppid": 32243, "start_time": "2021-09-13T12:00:58.267+0200"}}} 2021-09-13T12:00:58.701+0200 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.14.1 2021-09-13T12:00:58.701+0200 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.14.1' as ILM is enabled. 2021-09-13T12:00:58.703+0200 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200 2021-09-13T12:00:58.706+0200 INFO [publisher] pipeline/module.go:113 Beat name: Philipps-Elastic-Macbook-Pro.local 2021-09-13T12:00:58.709+0200 INFO beater/filebeat.go:117 Enabled modules/filesets: cisco (asa), () 2021-09-13T12:00:58.713+0200 INFO instance/beat.go:473 filebeat start running. 2021-09-13T12:00:58.713+0200 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s 2021-09-13T12:00:58.714+0200 INFO memlog/store.go:119 Loading data file of '/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data/registry/filebeat' succeeded. Active transaction id=0 2021-09-13T12:00:58.715+0200 INFO memlog/store.go:124 Finished loading transaction log file for '/Users/philippkahr/Downloads/filebeat-7.14.1-darwin-x86_64/data/registry/filebeat'. Active transaction id=1 2021-09-13T12:00:58.718+0200 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0 2021-09-13T12:00:58.719+0200 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1 2021-09-13T12:00:58.719+0200 INFO [crawler] beater/crawler.go:141 Starting input (ID: 11000608890507230499) 2021-09-13T12:00:58.720+0200 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1 2021-09-13T12:00:58.720+0200 INFO udp/input.go:99 Starting UDP input 2021-09-13T12:00:58.720+0200 INFO [UDP] dgram/server.go:99 Started listening for UDP connection 2021-09-13T12:00:58.720+0200 INFO cfgfile/reload.go:164 Config reloader started 2021-09-13T12:00:58.721+0200 INFO cfgfile/reload.go:224 Loading of config files completed. ```

Related issues

It sort of relates to : #27802 as I think the underlying go logic is the same.

[botelastic]

This issue doesn't have a Team:<team> label.

[philippkahr]

beats/filebeat/fileset/modules.go

Line 118 in 23e4403

func NewModuleRegistry(moduleConfigs []*common.Config, beatInfo beat.Info, init bool) (*ModuleRegistry, error) {

@kvch I looked into this, but I could not figure out on how to properly rewrite that go function to deal with the setup command issue and this issue, without breaking everything.

[jsoriano]

Hey @philippkahr,

Thanks for your feedback. This issue is already being addressed in #27526 and will hopefully be fixed in 8.0.

I am closing this because it has been already reported in #17256 and #11943. Feel free to comment in one of these issues or reopen if you think that something else needs to be addressed.