rsyslogd display "sendto() error: Network is unreachable" when management VRF is enabled

[irene-pan1202]

Description

rsyslogd display "sendto() error: Network is unreachable" when management VRF is enabled, and eth0 does not have any syslog packet.

Steps to reproduce the issue:

1. Run command sudo config vrf add mgmt.
2. Check /var/log/syslog and rsyslog status

Describe the results you received:

Oct 27 13:36:37.331705 DUT4 INFO rsyslogd: action 'action-0-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
Oct 27 13:36:37.332482 DUT4 ERR rsyslogd: omfwd/udp: socket 9: sendto() error: Network is unreachable [v8.1901.0 try https://www.rsyslog.com/e/2354 ]
Oct 27 13:36:37.332767 DUT4 ERR rsyslogd: omfwd: socket 9: error 101 sending via udp: Network is unreachable [v8.1901.0 try https://www.rsyslog.com/e/2354 ]
Oct 27 13:36:37.332985 DUT4 WARNING rsyslogd: action 'action-0-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try https://www.rsyslog.com/e/2007 ]

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

[lguohan]

@padmanarayana , can you comment on this issue?

[anshuv-mfst]

1. Please provide "show tech support" output
2. Please which release is this seen.

[irene-pan1202]

@anshuv-mfst Should the rsyslog service be started using "ip vrf exec mgmt" like NTP service?

root@as7726-32x-1:~# systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset:
Active: active (running) since Thu 2020-11-12 02:49:49 UTC; 5min ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 20369 (rsyslogd)
Tasks: 5 (limit: 4915)
Memory: 5.2M
CGroup: /system.slice/rsyslog.service
└─20369 /usr/sbin/rsyslogd -n

root@as7726-32x-1:~# systemctl status ntp
● ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enab
Active: active (running) since Thu 2020-11-12 02:42:56 UTC; 12min ago
Docs: man:ntpd(8)
Process: 20046 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status
Tasks: 2 (limit: 4915)
Memory: 2.5M
CGroup: /system.slice/ntp.service
└─vrf
└─mgmt
└─20057 /usr/sbin/ntpd -p /var/run/ntpd.pid -x -u 105:110

[padmanarayana]

As per design (https://github.com/Azure/SONiC/blob/master/doc/mgmt/sonic_stretch_management_vrf_design.md), syslog is currently not supported with mVRF.

[bsun-sudo]

syslog is not yet supported with mVRF in the community code, thus this error when mgmt VRF is configured.

[archidote]

in my case that was a problem with udp traffic.

so in the bottm of my client configuration file, I have added the following line :

. @@192.168.0.101:514

@@ = TCP
@ = UDP

I hope that it will help you

[bluecmd]

@tisteagle and I made a patch that enables rsyslog for VRF management, fixing this issue.

The intention is to submit a PR with the fix, but if somebody else needs it you can apply the following patch.

diff --git a/files/image_config/rsyslog/rsyslog.conf.j2 b/files/image_config/rsyslog/rsyslog.conf.j2
index 37410293a..f863d8497 100644
--- a/files/image_config/rsyslog/rsyslog.conf.j2
+++ b/files/image_config/rsyslog/rsyslog.conf.j2
@@ -38,11 +38,21 @@ $UDPServerRun 514
 
 # Define a custom template
 $template SONiCFileFormat,"%timegenerated%.%timegenerated:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
+# Use RFC3164 compatible format for remote syslog
+$template SONiCRemoteFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
 $ActionFileDefaultTemplate SONiCFileFormat
 
 #Set remote syslog server
 {% for server in SYSLOG_SERVER %}
-*.* @[{{ server }}]:514;SONiCFileFormat
+*.* action(type="omfwd"
+  template="SONiCRemoteFormat"
+  target="{{ server }}"
+  port="{{SYSLOG_SERVER[server].get("port", 514)}}"
+  protocol="{{SYSLOG_SERVER[server].get("protocol", "udp")}}"
+{% if 'vrf' in SYSLOG_SERVER[server] %}
+  device="{{SYSLOG_SERVER[server]["vrf"]}}"
+{% endif %}
+  )
 {% endfor %}
 
 #

You can then manually add a vrf property to a syslog server in config_db, like this:

{
    "SYSLOG_SERVER": {
        "10.198.0.12": {
            "vrf": "mgmt"
        }
    }
}

[nazariig]

@irene-pan1202 FYI: sonic-net/SONiC#1002

[bluecmd]

@nazariig Nice! Confirmed to work fine for us using 202211. Thanks!

I believe this issue can be closed

[nazariig]

@nazariig Nice! Confirmed to work fine for us using 202211. Thanks!

I believe this issue can be closed

@bluecmd thanks a lot for confirmation. Closing the ticket