Filebeat Juniper SRX Pipeline issue - Trimmed source address

[novaksam]

• Version: 8.8.2
• Operating System: Centos 8 Stream

For the example log: <14>1 2023-08-08T14:28:00.778-05:00 Route1- RT_FLOW - RT_FLOW_SESSION_DENY [source-address="192.168.1.1" source-port="39017" destination-address="8.8.4.4" destination-port="53" connection-tag="0" service-name="junos-dns-udp" protocol-id="17" icmp-type="0" policy-name="dns_deny_outbound" source-zone-name="trust" destination-zone-name="untrust" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No" reason="Denied by policy" session-id="85905209174" application-category="N/A" application-sub-category="N/A" application-risk="-1" application-characteristics="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]

The initial grok processor on the juniper-srx-pipeline pipeline strips out the source address.

"grok": {
        "field": "message",
        "patterns": [
          "^<%{POSINT:syslog_pri}>(\\d{1,3}\\s)?(?:%{TIMESTAMP_ISO8601:_temp_.raw_date})\\s%{SYSLOGHOST:syslog_hostname}\\s%{PROG:syslog_program}\\s(?:%{POSINT:syslog_pid}|-)?\\s%{WORD:log_type}\\s\\[.+?\\s%{GREEDYDATA:log.original}\\]$"
        ]
      }

Snippet of event.original:

"original": "source-port=\"39017\" destination-address=\"8.8.4.4\"

Modifying .+?\\s${GREEDYDATA to \\s?${GREEDYDATA allows the source.address to be correctly collected and enriched

"grok": {
        "field": "message",
        "patterns": [
          "^<%{POSINT:syslog_pri}>(\\d{1,3}\\s)?(?:%{TIMESTAMP_ISO8601:_temp_.raw_date})\\s%{SYSLOGHOST:syslog_hostname}\\s%{PROG:syslog_program}\\s(?:%{POSINT:syslog_pid}|-)?\\s%{WORD:log_type}\\s\\[\\s?%{GREEDYDATA:log.original}\\]$"
        ]
      }

Snippet of event.original:

"original": "source-address=\"192.168.1.1\" source-port=\"39017\"

[novaksam]

beats/x-pack/filebeat/module/juniper/srx/ingest/pipeline.yml

Line 8 in e194656

- '^<%{POSINT:syslog_pri}>(\d{1,3}\s)?(?:%{TIMESTAMP_ISO8601:_temp_.raw_date})\s%{SYSLOGHOST:syslog_hostname}\s%{PROG:syslog_program}\s(?:%{POSINT:syslog_pid}|-)?\s%{WORD:log_type}\s\[.+?\s%{GREEDYDATA:log.original}\]$'

[jamiehynds]

Thanks for reporting this @novaksam. We recently made some changes to our Elastic Agent integration with SRX, which may be a better fit for you. Are you in a position to using our SRX elastic agent integration or are you tied to Beats?

@kcreddy could you check if our SRX pipeline also strips out the source address or is this limited to the Beats pipeline?

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[novaksam]

@jamiehynds I'm unable to use the builtin security features for elastic stack because we utilize Searchguard for authentication and encryption; I have considerably too much data (+33TB, ~12Mo retention) and too many use cases to be looking at purchasing a subscription.

I've looked at using the agent + fleet with a separate stack, but I needed to get things back up and running before I went to vacation. I do a decent amount of custom enrichment in logstash, and some of my beat configurations aren't super straight forward (going to different indexes based on content, many different event subscriptions in winlogbeat, etc) and I just haven't had the time to figure out if it's possible with the agents.

Just wanted to report this finding as I just started using the pipeline and noticed I wasn't getting the source address :)

[kcreddy]

@kcreddy could you check if our SRX pipeline also strips out the source address or is this limited to the Beats pipeline?

Hey @jamiehynds, Dan's PR should fix the beats pipeline. This should align with structured traffic data handling of current SRX ingest pipeline.