Skip to content

Commit

Permalink
sophosxg-module initial release (elastic#19591)
Browse files Browse the repository at this point in the history
What does this PR do?

This PR Introduces the SophosXG filebeat module. Focusing currently on SophosXG Firewall, but should include other Sophos products as separate PR's later on.
Why is it important?

Adding more supported products to the filebeat portfolio.

Co-authored-by: StefanSa <[email protected]>
Co-authored-by: P1llus <[email protected]>
  • Loading branch information
3 people authored Jul 14, 2020
1 parent 1487527 commit 4f36c92
Show file tree
Hide file tree
Showing 46 changed files with 12,099 additions and 0 deletions.
1,881 changes: 1,881 additions & 0 deletions filebeat/docs/fields.asciidoc

Large diffs are not rendered by default.

159 changes: 159 additions & 0 deletions filebeat/docs/modules/sophosxg.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
////
This file is generated! See scripts/docs_collector.py
////

[[filebeat-module-sophosxg]]
[role="xpack"]

:modulename: sophosxg
:has-dashboards: false

== SophosXG module

This is a module for SophosXG SFOS logs sent in the syslog format.

To configure a remote syslog destination, please reference the https://community.sophos.com/kb/en-us/123184[SophosXG/SFOS Documentation].

The syslog format choosen should be `Default`.

include::../include/gs-link.asciidoc[]

[float]
=== Compatibility

This module has been tested against SFOS version 17.5.x and 18.0.x.
Versions above this are expected to work but have not been tested.

include::../include/configuring-intro.asciidoc[]

:fileset_ex: firewall

include::../include/config-option-intro.asciidoc[]

[float]
==== `firewall` fileset settings

[source,yaml]
----
- module: sophosxg
firewall:
enabled: true
var.input: udp
var.syslog_host: 0.0.0.0
var.syslog_port: 9005
var.host_name: firewall.localgroup.local
----

include::../include/var-paths.asciidoc[]

*`var.input`*::

The input to use, can be either the value `tcp`, `udp` or `file`.

*`var.syslog_host`*::

The interface to listen to all syslog traffic. Defaults to localhost.
Set to 0.0.0.0 to bind to all available interfaces.

*`var.syslog_port`*::

The port to listen for syslog traffic. Defaults to 9005.

*`var.host_name`*::

Host name / Observer name, since SophosXG does not provide this in the syslog file.
Default to `firewall.localgroup.local`


[float]
==== SophosXG ECS fields

This is a list of FortiOS fields that are mapped to ECS.

[options="header"]
|==============================================================
| SophosXG Fields | ECS Fields |
| application | network.protocol |
| classification | rule.category |
| device_id | observer.serial_number |
| domainname | url.domain |
| dst_host | destination.address |
| dst_int | observer.egress.interface.name |
| dstzonetype | observer.egress.zone |
| dst_ip | destination.ip |
| destinationip | destination.ip |
| dst_mac | destination.mac |
| dstname | destination.address |
| dst_port | destination.port |
| dst_domainname | url.domain |
| duration | event.duration |
| filename | file.name |
| filetype | file.extension |
| file_size | file.size |
| file_path | file.directory |
| fw_rule_id | rule.id |
| from_email_address | source.user.email |
| httpstatus | http.response.status_code |
| in_interface | observer.ingress.interface.name|
| log_id | event.code |
| log_subtype | event.action |
| message | message |
| method | http.request.method |
| policy_type | rule.ruleset |
| protocol | network.transport |
| recv_bytes | destination.bytes |
| recv_pkts | destination.packets |
| referer | http.request.referrer |
| sent_bytes | source.bytes |
| sent_pkts | source.packets |
| sha1sum | file.hash.sha1 |
| srczonetype | observer.ingress.zone |
| src_ip | source.ip |
| src_domainname | url.domain |
| sourceip | source.ip |
| src_mac | source.mac |
| src_port | source.port |
| status_code | http.response.status_code |
| time_zone | event.timezone |
| to_email_address | destination.user.email |
| tran_dst_ip | destination.nat.ip |
| tran_dst_port | destination.nat.port |
| tran_src_ip | source.nat.ip |
| tran_src_port | source.nat.port |
| url | url.original |
| user_agent | user_agent.original |
| useragent | user_agent.original |
| user_gp | source.user.group |
| user_name | source.user.name |
| ws_protocol | http.version |
|==============================================================

:has-dashboards!:

:fileset_ex!:

[float]
[[dynamic-script-compilations]]
=== Dynamic Script Compilations

The `sophosxg` filesets are based on Elasticsearch ingest pipelines and
make extensive use of script processors and painless conditions. This can cause
the pipelines to fail loading the first time the module is used, due to exceeding
the maximum script compilation limits. It is recommended to tune the following
parameters on your Elasticsearch cluster:

- {ref}/circuit-breaker.html#script-compilation-circuit-breaker[script.max_compilations_rate]:
Increase to at least `100/5m`.

- {ref}/modules-scripting-using.html#modules-scripting-using-caching[script.cache.max_size]:
Increase to at least `300` if using both filesets or other script-heavy modules.

:modulename!:


[float]
=== Fields

For a description of each field in the module, see the
<<exported-fields-sophosxg,exported fields>> section.

2 changes: 2 additions & 0 deletions filebeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ This file is generated! See scripts/docs_collector.py
* <<filebeat-module-redis>>
* <<filebeat-module-santa>>
* <<filebeat-module-sonicwall>>
* <<filebeat-module-sophosxg>>
* <<filebeat-module-squid>>
* <<filebeat-module-suricata>>
* <<filebeat-module-system>>
Expand Down Expand Up @@ -120,6 +121,7 @@ include::modules/rapid7.asciidoc[]
include::modules/redis.asciidoc[]
include::modules/santa.asciidoc[]
include::modules/sonicwall.asciidoc[]
include::modules/sophosxg.asciidoc[]
include::modules/squid.asciidoc[]
include::modules/suricata.asciidoc[]
include::modules/system.asciidoc[]
Expand Down
24 changes: 24 additions & 0 deletions x-pack/filebeat/filebeat.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1335,6 +1335,30 @@ filebeat.modules:
# "+02:00" for GMT+02:00
# var.tz_offset: local

#------------------------------- Sophosxg Module -------------------------------
- module: sophosxg
firewall:
enabled: true

# Set which input to use between tcp, udp (default) or file.
#var.input: udp

# The interface to listen to syslog traffic. Defaults to
# localhost. Set to 0.0.0.0 to bind to all available interfaces.
#var.syslog_host: localhost

# The port to listen for syslog traffic. Defaults to 9004.
#var.syslog_port: 9005

# firewall default hostanme
#var.default_host_name: firewall.localgroup.local

# known firewalls
#var.known_devices:
# "device1_serialnumber": "a.host.local"
# "device2_serialnumber": "b.host.local"


#-------------------------------- Squid Module --------------------------------
- module: squid
log:
Expand Down
1 change: 1 addition & 0 deletions x-pack/filebeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 22 additions & 0 deletions x-pack/filebeat/module/sophosxg/_meta/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
- module: sophosxg
firewall:
enabled: true

# Set which input to use between tcp, udp (default) or file.
#var.input: udp

# The interface to listen to syslog traffic. Defaults to
# localhost. Set to 0.0.0.0 to bind to all available interfaces.
#var.syslog_host: localhost

# The port to listen for syslog traffic. Defaults to 9004.
#var.syslog_port: 9005

# firewall default hostanme
#var.default_host_name: firewall.localgroup.local

# known firewalls
#var.known_devices:
# "device1_serialnumber": "a.host.local"
# "device2_serialnumber": "b.host.local"

146 changes: 146 additions & 0 deletions x-pack/filebeat/module/sophosxg/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
[role="xpack"]

:modulename: sophosxg
:has-dashboards: false

== SophosXG module

This is a module for SophosXG SFOS logs sent in the syslog format.

To configure a remote syslog destination, please reference the https://community.sophos.com/kb/en-us/123184[SophosXG/SFOS Documentation].

The syslog format choosen should be `Default`.

include::../include/gs-link.asciidoc[]

[float]
=== Compatibility

This module has been tested against SFOS version 17.5.x and 18.0.x.
Versions above this are expected to work but have not been tested.

include::../include/configuring-intro.asciidoc[]

:fileset_ex: firewall

include::../include/config-option-intro.asciidoc[]

[float]
==== `firewall` fileset settings

[source,yaml]
----
- module: sophosxg
firewall:
enabled: true
var.input: udp
var.syslog_host: 0.0.0.0
var.syslog_port: 9005
var.host_name: firewall.localgroup.local
----

include::../include/var-paths.asciidoc[]

*`var.input`*::

The input to use, can be either the value `tcp`, `udp` or `file`.

*`var.syslog_host`*::

The interface to listen to all syslog traffic. Defaults to localhost.
Set to 0.0.0.0 to bind to all available interfaces.

*`var.syslog_port`*::

The port to listen for syslog traffic. Defaults to 9005.

*`var.host_name`*::

Host name / Observer name, since SophosXG does not provide this in the syslog file.
Default to `firewall.localgroup.local`


[float]
==== SophosXG ECS fields

This is a list of FortiOS fields that are mapped to ECS.

[options="header"]
|==============================================================
| SophosXG Fields | ECS Fields |
| application | network.protocol |
| classification | rule.category |
| device_id | observer.serial_number |
| domainname | url.domain |
| dst_host | destination.address |
| dst_int | observer.egress.interface.name |
| dstzonetype | observer.egress.zone |
| dst_ip | destination.ip |
| destinationip | destination.ip |
| dst_mac | destination.mac |
| dstname | destination.address |
| dst_port | destination.port |
| dst_domainname | url.domain |
| duration | event.duration |
| filename | file.name |
| filetype | file.extension |
| file_size | file.size |
| file_path | file.directory |
| fw_rule_id | rule.id |
| from_email_address | source.user.email |
| httpstatus | http.response.status_code |
| in_interface | observer.ingress.interface.name|
| log_id | event.code |
| log_subtype | event.action |
| message | message |
| method | http.request.method |
| policy_type | rule.ruleset |
| protocol | network.transport |
| recv_bytes | destination.bytes |
| recv_pkts | destination.packets |
| referer | http.request.referrer |
| sent_bytes | source.bytes |
| sent_pkts | source.packets |
| sha1sum | file.hash.sha1 |
| srczonetype | observer.ingress.zone |
| src_ip | source.ip |
| src_domainname | url.domain |
| sourceip | source.ip |
| src_mac | source.mac |
| src_port | source.port |
| status_code | http.response.status_code |
| time_zone | event.timezone |
| to_email_address | destination.user.email |
| tran_dst_ip | destination.nat.ip |
| tran_dst_port | destination.nat.port |
| tran_src_ip | source.nat.ip |
| tran_src_port | source.nat.port |
| url | url.original |
| user_agent | user_agent.original |
| useragent | user_agent.original |
| user_gp | source.user.group |
| user_name | source.user.name |
| ws_protocol | http.version |
|==============================================================

:has-dashboards!:

:fileset_ex!:

[float]
[[dynamic-script-compilations]]
=== Dynamic Script Compilations

The `sophosxg` filesets are based on Elasticsearch ingest pipelines and
make extensive use of script processors and painless conditions. This can cause
the pipelines to fail loading the first time the module is used, due to exceeding
the maximum script compilation limits. It is recommended to tune the following
parameters on your Elasticsearch cluster:

- {ref}/circuit-breaker.html#script-compilation-circuit-breaker[script.max_compilations_rate]:
Increase to at least `100/5m`.

- {ref}/modules-scripting-using.html#modules-scripting-using-caching[script.cache.max_size]:
Increase to at least `300` if using both filesets or other script-heavy modules.

:modulename!:
9 changes: 9 additions & 0 deletions x-pack/filebeat/module/sophosxg/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- key: sophosxg
title: "sophosxg"
description: >
sophosxg Module
fields:
- name: sophosxg
type: group
description: >
fields:
Loading

0 comments on commit 4f36c92

Please sign in to comment.