[Filebeat] Add Pensando DFW Module (#21063) (#24045)

* Add Pensando module init * explicitly define the ECS version per testing * updates to docs from make update * updates for pensando module * updates to documentation and db screenshot * add dashboard export to repo * update to add pensando beat * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <[email protected]> * Update pipeline.yml Condensed all "remove" fields to 1 list of fields. * Update pipeline.yml Do not remove the payload_raw field. * Update filebeat/module/pensando/_meta/docs.asciidoc Co-authored-by: Andrew Kroh <[email protected]> * Update config.yml Added syslog_host and syslog_port values as suggested. * Update docs.asciidoc Added documentation for syslog_host and syslog_port as suggested. * Update pipeline.yml Removing payload_raw - this and json are, essentially, the same field and no longer needed after parsing. * Update pipeline.yml Changed checks if values are != null to use the filebeat specific ignore_empty_value: true instead. * Remove set of event.module Remove the set param for event.module. Filebeat should add this automatically. * Apply suggestions from code review Co-authored-by: Andrew Kroh <[email protected]> * Update test.log * Use convert instead of set for some fields Changed ECS sets for IP addresses and ports to converts of type ip and integer respectively. * Updates for geoip and autonomous system * add pensando dfw fields * fixes from make -C filebeat update * fixes for filebeat check * make update changes * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <[email protected]> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <[email protected]> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <[email protected]> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <[email protected]> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <[email protected]> * remove old json file * ran tests * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <[email protected]> * gen after run of 'mage -v pythonIntegTest' * Update fields.yml * mage fmt update request Co-authored-by: Marc Guasch <[email protected]> Co-authored-by: Andrew Kroh <[email protected]> (cherry picked from commit 4194408) Co-authored-by: Edward Arcuri <[email protected]>
elastic · Feb 16, 2021 · 73af5b5 · 73af5b5
1 parent a6d108f
commit 73af5b5
Show file tree

Hide file tree

Showing 21 changed files with 2,152 additions and 0 deletions.
diff --git a/CHANGELOG-developer.next.asciidoc b/CHANGELOG-developer.next.asciidoc
@@ -96,6 +96,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
 - Update Go version to 1.14.7. {pull}20508[20508]
 - Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
 - Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
+- Add Pensando distributed firewall module. {pull}21063[21063]
 - Update ecszap to v0.3.0 for using ECS 1.6.0 in logs {pull}22267[22267]
 - Add support for customized monitoring API. {pull}22605[22605]
 

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -69,6 +69,7 @@ grouped in the following categories:
 * <<exported-fields-oracle>>
 * <<exported-fields-osquery>>
 * <<exported-fields-panw>>
+* <<exported-fields-pensando>>
 * <<exported-fields-postgresql>>
 * <<exported-fields-process>>
 * <<exported-fields-proofpoint>>
@@ -105825,6 +105826,147 @@ Specifies the sub type of the log
 
 --
 
+[[exported-fields-pensando]]
+== Pensando fields
+
+pensando Module
+
+
+
+[float]
+=== pensando
+
+Fields from Pensando logs.
+
+
+
+[float]
+=== dfw
+
+Fields for Pensando DFW
+
+
+
+*`pensando.dfw.action`*::
++
+--
+Action on the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.app_id`*::
++
+--
+Application ID 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.destination_address`*::
++
+--
+Address of destination. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.destination_port`*::
++
+--
+Port of destination. 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.direction`*::
++
+--
+Direction of the flow 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.protocol`*::
++
+--
+Protocol of the flow 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.rule_id`*::
++
+--
+Rule ID that was matched. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.session_id`*::
++
+--
+Session ID of the flow 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.session_state`*::
++
+--
+Session state of the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.source_address`*::
++
+--
+Source address of the flow. 
+
+
+type: keyword
+
+--
+
+*`pensando.dfw.source_port`*::
++
+--
+Source port of the flow. 
+
+
+type: integer
+
+--
+
+*`pensando.dfw.timestamp`*::
++
+--
+Timestamp of the log. 
+
+
+type: date
+
+--
+
 [[exported-fields-postgresql]]
 == PostgreSQL fields
 

diff --git a/filebeat/docs/images/filebeat-pensando-dfw.png b/filebeat/docs/images/filebeat-pensando-dfw.png
diff --git a/filebeat/docs/modules/pensando.asciidoc b/filebeat/docs/modules/pensando.asciidoc
@@ -0,0 +1,69 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-pensando]]
+:modulename: pensando
+:has-dashboards: true
+
+== pensando module
+
+The +{modulename}+ module parses distributed firewall logs created by the
+http://pensando.io/[Pensando] distributed services card (DSC).
+
+
+include::../include/what-happens.asciidoc[]
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+The Pensando module has been tested with 1.12.0-E-54 and later.
+
+include::../include/configuring-intro.asciidoc[]
+The following example shows how to set parameters in the +modules.d/{modulename}.yml+
+file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
+
+["source","yaml",subs="attributes"]
+-----
+- module: pensando
+  access:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: [9001]
+-----
+:fileset_ex: dfw
+
+include::../include/config-option-intro.asciidoc[]
+
+TODO: document the variables from each fileset. If you're describing a variable
+that's common to other modules, you can reuse shared descriptions by including
+the relevant file. For example:
+
+[float]
+==== `dfw` log fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard. For example:
+
+[role="screenshot"]
+image::./images/filebeat-pensando-dfw.png[]
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-pensando,exported fields>> section.
+
diff --git a/filebeat/docs/modules_list.asciidoc b/filebeat/docs/modules_list.asciidoc
@@ -50,6 +50,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-oracle>>
   * <<filebeat-module-osquery>>
   * <<filebeat-module-panw>>
+  * <<filebeat-module-pensando>>
   * <<filebeat-module-postgresql>>
   * <<filebeat-module-proofpoint>>
   * <<filebeat-module-rabbitmq>>
@@ -121,6 +122,7 @@ include::modules/okta.asciidoc[]
 include::modules/oracle.asciidoc[]
 include::modules/osquery.asciidoc[]
 include::modules/panw.asciidoc[]
+include::modules/pensando.asciidoc[]
 include::modules/postgresql.asciidoc[]
 include::modules/proofpoint.asciidoc[]
 include::modules/rabbitmq.asciidoc[]

diff --git a/filebeat/filebeat.reference.yml b/filebeat/filebeat.reference.yml
@@ -335,6 +335,18 @@ filebeat.modules:
     # of the document. The default is true.
     #var.use_namespace: true
 
+#------------------------------- Pensando Module -------------------------------
+- module: pensando
+# Firewall logs
+  dfw:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9001
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:
+
 #------------------------------ PostgreSQL Module ------------------------------
 #- module: postgresql
   # Logs

diff --git a/filebeat/include/list.go b/filebeat/include/list.go
diff --git a/filebeat/module/pensando/_meta/config.yml b/filebeat/module/pensando/_meta/config.yml
@@ -0,0 +1,10 @@
+- module: pensando
+# Firewall logs
+  dfw:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: 9001
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:
diff --git a/filebeat/module/pensando/_meta/docs.asciidoc b/filebeat/module/pensando/_meta/docs.asciidoc
@@ -0,0 +1,56 @@
+:modulename: pensando
+:has-dashboards: true
+
+== pensando module
+
+The +{modulename}+ module parses distributed firewall logs created by the
+http://pensando.io/[Pensando] distributed services card (DSC).
+
+
+include::../include/what-happens.asciidoc[]
+
+include::../include/gs-link.asciidoc[]
+
+[float]
+=== Compatibility
+
+The Pensando module has been tested with 1.12.0-E-54 and later.
+
+include::../include/configuring-intro.asciidoc[]
+The following example shows how to set parameters in the +modules.d/{modulename}.yml+
+file to listen for firewall logs sent from the Pensando DSC(s) on port 5514 (default is 9001):
+
+["source","yaml",subs="attributes"]
+-----
+- module: pensando
+  access:
+    enabled: true
+    var.syslog_host: 0.0.0.0
+    var.syslog_port: [9001]
+-----
+:fileset_ex: dfw
+
+include::../include/config-option-intro.asciidoc[]
+
+TODO: document the variables from each fileset. If you're describing a variable
+that's common to other modules, you can reuse shared descriptions by including
+the relevant file. For example:
+
+[float]
+==== `dfw` log fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard. For example:
+
+[role="screenshot"]
+image::./images/filebeat-pensando-dfw.png[]
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:
diff --git a/filebeat/module/pensando/_meta/fields.yml b/filebeat/module/pensando/_meta/fields.yml
@@ -0,0 +1,10 @@
+- key: pensando
+  title: Pensando 
+  description: >
+    pensando Module
+  fields:
+    - name: pensando
+      type: group
+      description: >
+        Fields from Pensando logs.
+      fields: