zebra crash was observed in sonic-mgmt reboot tests

[ysmanman]

Description

We observed the following zebra crash SIGSEGV during sonic-mgmt platform reboot tests. The crash was seen with 202205 image.

(gdb) bt
#0  0x00007f69878d6fe1 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f69879c589c in core_handler (signo=11, siginfo=0x7ffe482a28b0, context=<optimized out>) at lib/sigevent.c:262
#2  <signal handler called>
#3  zebra_vxlan_macvlan_up (ifp=ifp@entry=0x559fcbb07d00) at zebra/zebra_vxlan.c:4710
#4  0x0000559fcb0e2508 in if_up (ifp=ifp@entry=0x559fcbb07d00) at zebra/interface.c:1071
#5  0x0000559fcb0dcfa9 in netlink_link_change (h=0x7ffe482a33d0, ns_id=0, startup=<optimized out>) at zebra/if_netlink.c:1962
#6  0x0000559fcb0e7eed in netlink_parse_info (filter=filter@entry=0x559fcb0e6a30 <netlink_information_fetch>,
    nl=nl@entry=0x559fcbb00048, zns=zns@entry=0x7ffe482ab420, count=count@entry=5, startup=startup@entry=false)
    at zebra/kernel_netlink.c:1007
#7  0x0000559fcb0e8225 in kernel_read (thread=<optimized out>) at zebra/kernel_netlink.c:416
#8  0x00007f69879d749d in thread_call (thread=thread@entry=0x7ffe482ab550) at lib/thread.c:1958
#9  0x00007f69879914a8 in frr_run (master=0x559fcb8efdc0) at lib/libfrr.c:1184
#10 0x0000559fcb0d5fc6 in main (argc=9, argv=0x7ffe482ab958) at zebra/main.c:471

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[ysmanman]

core file zebra.1699483945.44.0.core.gz

[ysmanman]

Add @arlakshm and @kenneth-arista for visibility.

[arlakshm]

debugged the zebra crash seems like we are hitting this issue on the midplane port. zebra: Lookup linked interface in link netns by leonshaw · Pull Request #10580 · FRRouting/frr (github.com)

Bt:
root@15a49dce4015:/# gdb --core=/core/1113/zebra.1699483945.44.0.core /usr/lib/frr/zebra
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/frr/zebra...
Reading symbols from /usr/lib/debug/.build-id/37/7089e60ccdf7c929253f511aa58c36389eb14e.debug...

warning: Can't open file /var/tmp/frr/zebra.44/logbuf.71 during file-backed mapping note processing

warning: Can't open file /var/tmp/frr/zebra.44/logbuf.70 during file-backed mapping note processing

warning: Can't open file /var/tmp/frr/zebra.44/logbuf.62 during file-backed mapping note processing

warning: Can't open file /var/tmp/frr/zebra.44/logbuf.55 during file-backed mapping note processing

warning: Can't open file /var/tmp/frr/zebra.44/logbuf.54 during file-backed mapping note processing
[New LWP 44]
[New LWP 55]
[New LWP 71]
[New LWP 54]
[New LWP 70]
[New LWP 62]
[New LWP 53]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M fpm -M snmp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f69878d6fe1 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7f69873777c0 (LWP 44))]
(gdb) bt
#0 0x00007f69878d6fe1 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f69879c589c in core_handler (signo=11, siginfo=0x7ffe482a28b0, context=) at lib/sigevent.c:262
#2
#3 zebra_vxlan_macvlan_up (ifp=ifp@entry=0x559fcbb07d00) at zebra/zebra_vxlan.c:4710
#4 0x0000559fcb0e2508 in if_up (ifp=ifp@entry=0x559fcbb07d00) at zebra/interface.c:1071
#5 0x0000559fcb0dcfa9 in netlink_link_change (h=0x7ffe482a33d0, ns_id=0, startup=) at zebra/if_netlink.c:1962
#6 0x0000559fcb0e7eed in netlink_parse_info (filter=filter@entry=0x559fcb0e6a30 <netlink_information_fetch>,
nl=nl@entry=0x559fcbb00048, zns=zns@entry=0x7ffe482ab420, count=count@entry=5, startup=startup@entry=false)
at zebra/kernel_netlink.c:1007
#7 0x0000559fcb0e8225 in kernel_read (thread=) at zebra/kernel_netlink.c:416
#8 0x00007f69879d749d in thread_call (thread=thread@entry=0x7ffe482ab550) at lib/thread.c:1958
#9 0x00007f69879914a8 in frr_run (master=0x559fcb8efdc0) at lib/libfrr.c:1184
#10 0x0000559fcb0d5fc6 in main (argc=9, argv=0x7ffe482ab958) at zebra/main.c:471
(gdb) f 3
#3 zebra_vxlan_macvlan_up (ifp=ifp@entry=0x559fcbb07d00) at zebra/zebra_vxlan.c:4710
4710 zebra/zebra_vxlan.c: No such file or directory.
(gdb) p zif->link
$1 = (struct interface *) 0x0
(gdb) p ifp->name
$4 = "eth1", '\000' <repeats 15 times>

[rlhui]

not seen recently