[metricbeat] meraki module with device_health metricset #40836

tommyers-elastic · 2024-09-15T22:41:47Z

Proposed commit message

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

…c associations. This commit removes a couple of data points from the original code, namely: - network appliance port settings - these settings are not defined per-device, but per-network, and as such don't fit into the current naming taxonomy. - site-to-site VPN settings - similarly to the above, these are not per-device attibutes. In addition there is a limitation included on including only a single metric bucket (as returned from the API) per metricbeat collection loop. There needs to be some better logging/config validation on this, which will come later; for now, just run with a maximum 5 minute collection interval. This commit includes the field mappings too, but it currently causes events to get dropped due to the index field limit getting hit (max 10k fields). You can get around this by setting 'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't include all these mappings in the metricbeat module, but for now they are here to allow testing with standalone metricbeat.

…d latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent.

x-pack/metricbeat/module/meraki/device_health/_meta/fields.yml

…dex template

ishleenk17 · 2024-09-17T10:05:26Z

The lint error is due to implicit memory aliasing within for loop.
Something like this might help resolving the error ?

portCopy := port
switchports = append(switchports, &switchport{port: &portCopy})

tommyers-elastic · 2024-09-17T10:35:41Z

The lint error is due to implicit memory aliasing within for loop. Something like this might help resolving the error ?
portCopy := port
switchports = append(switchports, &switchport{port: &portCopy})

ah yeh thanks, i thought i fixed all these already. this issue actually has dissapeared in go1.22. i was gonna just put a global ignore rule for the linter error, but figured it might be easier to just guard against in the code. maybe i'll do both.

…o1.22+

tommyers-elastic · 2024-09-17T11:32:54Z

the more of these i fix the more the linter seems to find. since there's not really any need to do it, i reverted the fixes and added the exclude rule for the linkter.

…ng that change and putting back the annoying index loop code

go.mod

Co-authored-by: Craig MacKenzie <[email protected]>

cmacknz

Approving go.mod changes.

* inital meraki module and metricsets * initial refactor for single meraki metricset device_health * added tunnel support aka VPN support by Device * adding interfaces aka switch ports and switch port status * processing review comments * refactored for comments * fixing default metricset * Removing unused variables and adding text to required variables * add go module deps * fixing for loop and starting on fields yml * remove %d from loop condition * fixing review comments and loss latency * update dashboard-api lib to version with MIT licensed ratelimiter * remove unused fields in metricset struct * remove device details for now * Major refactor to improve overall readability and device status/metric associations. This commit removes a couple of data points from the original code, namely: - network appliance port settings - these settings are not defined per-device, but per-network, and as such don't fit into the current naming taxonomy. - site-to-site VPN settings - similarly to the above, these are not per-device attibutes. In addition there is a limitation included on including only a single metric bucket (as returned from the API) per metricbeat collection loop. There needs to be some better logging/config validation on this, which will come later; for now, just run with a maximum 5 minute collection interval. This commit includes the field mappings too, but it currently causes events to get dropped due to the index field limit getting hit (max 10k fields). You can get around this by setting 'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't include all these mappings in the metricbeat module, but for now they are here to allow testing with standalone metricbeat. * Add maximum collection interval check to avoid missing uplink loss and latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent. * revert accidental changes to reference config * add example in data.json * review comments: license headers, missing/incorrect mapping * result of 'go mod tidy' ... trying to unblock the CI builds * fix linter issues * update NOTICE.txt * add release tag to fields.yml, update docs * review feedback: keyword -> dat * commit results of 'make update' in x-pack metricbeat for CI * remove mappings to avoid hitting field limits in global metricbeat index template * add one field back to please the docs CI * more linter errors guarding against shared loop var pointers * fix error introduced in previous commit * add exception for golangci-lint G601 which is no longer required in go1.22+ * remove the guards against taking addresses of loop vars * i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code * Update go.mod to match toolchain version to .go-version Co-authored-by: Craig MacKenzie <[email protected]> --------- Co-authored-by: Dan Hiebert <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit ae71c67) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/metricbeat/metricbeat.reference.yml

* inital meraki module and metricsets * initial refactor for single meraki metricset device_health * added tunnel support aka VPN support by Device * adding interfaces aka switch ports and switch port status * processing review comments * refactored for comments * fixing default metricset * Removing unused variables and adding text to required variables * add go module deps * fixing for loop and starting on fields yml * remove %d from loop condition * fixing review comments and loss latency * update dashboard-api lib to version with MIT licensed ratelimiter * remove unused fields in metricset struct * remove device details for now * Major refactor to improve overall readability and device status/metric associations. This commit removes a couple of data points from the original code, namely: - network appliance port settings - these settings are not defined per-device, but per-network, and as such don't fit into the current naming taxonomy. - site-to-site VPN settings - similarly to the above, these are not per-device attibutes. In addition there is a limitation included on including only a single metric bucket (as returned from the API) per metricbeat collection loop. There needs to be some better logging/config validation on this, which will come later; for now, just run with a maximum 5 minute collection interval. This commit includes the field mappings too, but it currently causes events to get dropped due to the index field limit getting hit (max 10k fields). You can get around this by setting 'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't include all these mappings in the metricbeat module, but for now they are here to allow testing with standalone metricbeat. * Add maximum collection interval check to avoid missing uplink loss and latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent. * revert accidental changes to reference config * add example in data.json * review comments: license headers, missing/incorrect mapping * result of 'go mod tidy' ... trying to unblock the CI builds * fix linter issues * update NOTICE.txt * add release tag to fields.yml, update docs * review feedback: keyword -> dat * commit results of 'make update' in x-pack metricbeat for CI * remove mappings to avoid hitting field limits in global metricbeat index template * add one field back to please the docs CI * more linter errors guarding against shared loop var pointers * fix error introduced in previous commit * add exception for golangci-lint G601 which is no longer required in go1.22+ * remove the guards against taking addresses of loop vars * i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code * Update go.mod to match toolchain version to .go-version Co-authored-by: Craig MacKenzie <[email protected]> --------- Co-authored-by: Dan Hiebert <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit ae71c67)

) * inital meraki module and metricsets * initial refactor for single meraki metricset device_health * added tunnel support aka VPN support by Device * adding interfaces aka switch ports and switch port status * processing review comments * refactored for comments * fixing default metricset * Removing unused variables and adding text to required variables * add go module deps * fixing for loop and starting on fields yml * remove %d from loop condition * fixing review comments and loss latency * update dashboard-api lib to version with MIT licensed ratelimiter * remove unused fields in metricset struct * remove device details for now * Major refactor to improve overall readability and device status/metric associations. This commit removes a couple of data points from the original code, namely: - network appliance port settings - these settings are not defined per-device, but per-network, and as such don't fit into the current naming taxonomy. - site-to-site VPN settings - similarly to the above, these are not per-device attibutes. In addition there is a limitation included on including only a single metric bucket (as returned from the API) per metricbeat collection loop. There needs to be some better logging/config validation on this, which will come later; for now, just run with a maximum 5 minute collection interval. This commit includes the field mappings too, but it currently causes events to get dropped due to the index field limit getting hit (max 10k fields). You can get around this by setting 'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't include all these mappings in the metricbeat module, but for now they are here to allow testing with standalone metricbeat. * Add maximum collection interval check to avoid missing uplink loss and latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent. * revert accidental changes to reference config * add example in data.json * review comments: license headers, missing/incorrect mapping * result of 'go mod tidy' ... trying to unblock the CI builds * fix linter issues * update NOTICE.txt * add release tag to fields.yml, update docs * review feedback: keyword -> dat * commit results of 'make update' in x-pack metricbeat for CI * remove mappings to avoid hitting field limits in global metricbeat index template * add one field back to please the docs CI * more linter errors guarding against shared loop var pointers * fix error introduced in previous commit * add exception for golangci-lint G601 which is no longer required in go1.22+ * remove the guards against taking addresses of loop vars * i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code * Update go.mod to match toolchain version to .go-version Co-authored-by: Craig MacKenzie <[email protected]> --------- Co-authored-by: Dan Hiebert <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit ae71c67) Co-authored-by: Tom Myers <[email protected]>

… metricset (#40862) * [metricbeat] meraki module with device_health metricset (#40836) * inital meraki module and metricsets * initial refactor for single meraki metricset device_health * added tunnel support aka VPN support by Device * adding interfaces aka switch ports and switch port status * processing review comments * refactored for comments * fixing default metricset * Removing unused variables and adding text to required variables * add go module deps * fixing for loop and starting on fields yml * remove %d from loop condition * fixing review comments and loss latency * update dashboard-api lib to version with MIT licensed ratelimiter * remove unused fields in metricset struct * remove device details for now * Major refactor to improve overall readability and device status/metric associations. This commit removes a couple of data points from the original code, namely: - network appliance port settings - these settings are not defined per-device, but per-network, and as such don't fit into the current naming taxonomy. - site-to-site VPN settings - similarly to the above, these are not per-device attibutes. In addition there is a limitation included on including only a single metric bucket (as returned from the API) per metricbeat collection loop. There needs to be some better logging/config validation on this, which will come later; for now, just run with a maximum 5 minute collection interval. This commit includes the field mappings too, but it currently causes events to get dropped due to the index field limit getting hit (max 10k fields). You can get around this by setting 'setup.template.settings.index.mapping.total_fields.limit' to a higher value. We maybe can't include all these mappings in the metricbeat module, but for now they are here to allow testing with standalone metricbeat. * Add maximum collection interval check to avoid missing uplink loss and latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent. * revert accidental changes to reference config * add example in data.json * review comments: license headers, missing/incorrect mapping * result of 'go mod tidy' ... trying to unblock the CI builds * fix linter issues * update NOTICE.txt * add release tag to fields.yml, update docs * review feedback: keyword -> dat * commit results of 'make update' in x-pack metricbeat for CI * remove mappings to avoid hitting field limits in global metricbeat index template * add one field back to please the docs CI * more linter errors guarding against shared loop var pointers * fix error introduced in previous commit * add exception for golangci-lint G601 which is no longer required in go1.22+ * remove the guards against taking addresses of loop vars * i can't get the golanglint-ci config to get picked up, so i'm reverting that change and putting back the annoying index loop code * Update go.mod to match toolchain version to .go-version Co-authored-by: Craig MacKenzie <[email protected]> --------- Co-authored-by: Dan Hiebert <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit ae71c67) # Conflicts: # NOTICE.txt # go.mod # go.sum # x-pack/metricbeat/metricbeat.reference.yml * fix go.mod merge conflict * fix merge errors in NOTICE.txt * fix merge errors in metricbeat.reference.yml * add result of running 'mage update' --------- Co-authored-by: Tom Myers <[email protected]>

DanH-Semplicity and others added 25 commits August 30, 2024 23:14

inital meraki module and metricsets

d6e356a

Merge branch 'main' into dans_final_factor

c048f77

Merge branch 'elastic:main' into dans_final_factor

2986d09

initial refactor for single meraki metricset device_health

9fe235e

added tunnel support aka VPN support by Device

53ae668

adding interfaces aka switch ports and switch port status

c122fc8

Merge branch 'elastic:main' into dans_final_factor

729094b

processing review comments

0c5b27f

refactored for comments

0ffeed9

Merge branch 'elastic:main' into dans_final_factor

a9f809b

fixing default metricset

9223215

Removing unused variables and adding text to required variables

d2e5848

add go module deps

d438a84

Merge branch 'elastic:main' into dans_final_factor

07819cd

fixing for loop and starting on fields yml

998936f

remove %d from loop condition

c1f3dd1

fixing review comments and loss latency

bd41dd0

update dashboard-api lib to version with MIT licensed ratelimiter

d99353f

remove unused fields in metricset struct

0d5138a

remove device details for now

88f5905

Add maximum collection interval check to avoid missing uplink loss an…

28109af

…d latency metrics. Remove 'percent' units, because the data does not fit the 0-1 elastic interpretation of percent.

revert accidental changes to reference config

6effa2d

Merge branch 'main' into meraki_metrics_tom_refactor

990b20f

add example in data.json

c0379ee

tommyers-elastic requested review from gpop63, agithomas and ishleenk17 September 15, 2024 22:41

tommyers-elastic requested review from a team as code owners September 15, 2024 22:41

tommyers-elastic added 2 commits September 16, 2024 15:36

fix linter issues

80cef99

update NOTICE.txt

6037396

agithomas reviewed Sep 16, 2024

View reviewed changes

x-pack/metricbeat/module/meraki/device_health/_meta/fields.yml Outdated Show resolved Hide resolved

agithomas reviewed Sep 16, 2024

View reviewed changes

x-pack/metricbeat/module/meraki/device_health/_meta/fields.yml Outdated Show resolved Hide resolved

tommyers-elastic added 5 commits September 16, 2024 20:39

add release tag to fields.yml, update docs

c220397

review feedback: keyword -> dat

122cd4b

commit results of 'make update' in x-pack metricbeat for CI

7bd4e06

remove mappings to avoid hitting field limits in global metricbeat in…

c7406cc

…dex template

add one field back to please the docs CI

8a97051

tommyers-elastic added 4 commits September 17, 2024 11:43

more linter errors guarding against shared loop var pointers

dd1f33d

fix error introduced in previous commit

6a666d2

add exception for golangci-lint G601 which is no longer required in g…

3f1e4b2

…o1.22+

remove the guards against taking addresses of loop vars

bebc839

i can't get the golanglint-ci config to get picked up, so i'm reverti…

cb4590b

…ng that change and putting back the annoying index loop code

tommyers-elastic added the backport-8.15 Automated backport to the 8.15 branch with mergify label Sep 17, 2024

cmacknz reviewed Sep 17, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Update go.mod to match toolchain version to .go-version

b0b26e2

Co-authored-by: Craig MacKenzie <[email protected]>

tommyers-elastic requested a review from cmacknz September 17, 2024 13:42

cmacknz approved these changes Sep 17, 2024

View reviewed changes

tommyers-elastic merged commit ae71c67 into elastic:main Sep 17, 2024
125 checks passed

This was referenced Sep 17, 2024

[8.15](backport #40836) [metricbeat] meraki module with device_health metricset #40862

Merged

[8.x](backport #40836) [metricbeat] meraki module with device_health metricset #40863

Merged

v1v mentioned this pull request Nov 7, 2024

[Automation] Bump Golang version to 1.22.9 #41553

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricbeat] meraki module with device_health metricset #40836

[metricbeat] meraki module with device_health metricset #40836

tommyers-elastic commented Sep 15, 2024 •

edited

Loading

ishleenk17 commented Sep 17, 2024

tommyers-elastic commented Sep 17, 2024

tommyers-elastic commented Sep 17, 2024

cmacknz left a comment

[metricbeat] meraki module with device_health metricset #40836

[metricbeat] meraki module with device_health metricset #40836

Conversation

tommyers-elastic commented Sep 15, 2024 • edited Loading

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

ishleenk17 commented Sep 17, 2024

tommyers-elastic commented Sep 17, 2024

tommyers-elastic commented Sep 17, 2024

cmacknz left a comment

Choose a reason for hiding this comment

tommyers-elastic commented Sep 15, 2024 •

edited

Loading