Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PortChannels may lose IP address after config reload #5291

Closed
wangxin opened this issue Sep 1, 2020 · 7 comments
Closed

PortChannels may lose IP address after config reload #5291

wangxin opened this issue Sep 1, 2020 · 7 comments
Assignees
@judyjoseph
Labels
Issue for 201911

Comments

@wangxin
Copy link
Contributor

wangxin commented Sep 1, 2020

Description

Steps to reproduce the issue:

  1. SONiC switch is running the t1-lag topology.
  2. Simply run "sudo config reload -y".
  3. Check bgp neighbors, check interface IP addresses

Describe the results you received:
After config reload, one or multiple PortChannels may lose their IP addresses.

In below console log, PortChannel0023 lost its IP address 10.0.0.28.

admin@str-msn2700-02:~$ show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 24
RIB entries 47, using 8648 bytes of memory
Peers 24, using 490 KiB of memory
Peer groups 4, using 256 bytes of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   NeighborName
10.0.0.33       4      64001      40      75        0    0    0 00:35:06            1   ARISTA01T0
10.0.0.35       4      64002      39      74        0    0    0 00:34:59            1   ARISTA02T0
10.0.0.37       4      64003      40      79        0    0    0 00:35:10            1   ARISTA03T0
10.0.0.39       4      64004      39      73        0    0    0 00:34:56            1   ARISTA04T0
10.0.0.41       4      64005      39      74        0    0    0 00:34:58            1   ARISTA05T0
10.0.0.43       4      64006      40      77        0    0    0 00:35:08            1   ARISTA06T0
10.0.0.45       4      64007      40      75        0    0    0 00:35:02            1   ARISTA07T0
10.0.0.47       4      64008      40      75        0    0    0 00:35:04            1   ARISTA08T0
10.0.0.49       4      64009      39      73        0    0    0 00:34:56            1   ARISTA09T0
10.0.0.51       4      64010      39      73        0    0    0 00:34:54            1   ARISTA10T0
10.0.0.53       4      64011      39      73        0    0    0 00:34:55            1   ARISTA11T0
10.0.0.55       4      64012      40      79        0    0    0 00:35:11            1   ARISTA12T0
10.0.0.57       4      64013      39      73        0    0    0 00:34:52            1   ARISTA13T0
10.0.0.59       4      64014      40      75        0    0    0 00:35:04            1   ARISTA14T0
10.0.0.61       4      64015      39      74        0    0    0 00:34:58            1   ARISTA15T0
10.0.0.63       4      64016      39      73        0    0    0 00:34:53            1   ARISTA16T0
10.0.0.1        4      65200      40      72        0    0    0 00:35:08            1   ARISTA01T2
10.0.0.5        4      65200      39      71        0    0    0 00:34:54            1   ARISTA03T2
10.0.0.9        4      65200      40      72        0    0    0 00:35:04            1   ARISTA05T2
10.0.0.13       4      65200      40      72        0    0    0 00:35:04            1   ARISTA07T2
10.0.0.17       4      65200      40      72        0    0    0 00:35:07            1   ARISTA09T2
10.0.0.21       4      65200      40      72        0    0    0 00:35:04            1   ARISTA11T2
10.0.0.25       4      65200      40      75        0    0    0 00:35:11            1   ARISTA13T2
10.0.0.29       4      65200       0       0        0    0    0    never       Active   ARISTA15T2

Total number of neighbors 24
admin@str-msn2700-02:~$ show int status
      Interface            Lanes    Speed    MTU    FEC    Alias             Vlan    Oper    Admin             Type    Asym PFC
---------------  ---------------  -------  -----  -----  -------  ---------------  ------  -------  ---------------  ----------
      Ethernet0          0,1,2,3      40G   9100    N/A     etp1  PortChannel0002      up       up   QSFP+ or later         off
      Ethernet4          4,5,6,7      40G   9100    N/A     etp2  PortChannel0002      up       up   QSFP+ or later         off
      Ethernet8        8,9,10,11      40G   9100    N/A     etp3  PortChannel0005      up       up   QSFP+ or later         off
     Ethernet12      12,13,14,15      40G   9100    N/A     etp4  PortChannel0005      up       up  QSFP28 or later         off
     Ethernet16      16,17,18,19      40G   9100    N/A     etp5  PortChannel0008      up       up   QSFP+ or later         off
     Ethernet20      20,21,22,23      40G   9100    N/A     etp6  PortChannel0008      up       up   QSFP+ or later         off
     Ethernet24      24,25,26,27      40G   9100    N/A     etp7  PortChannel0011      up       up   QSFP+ or later         off
     Ethernet28      28,29,30,31      40G   9100    N/A     etp8  PortChannel0011      up       up   QSFP+ or later         off
     Ethernet32      32,33,34,35      40G   9100    N/A     etp9  PortChannel0014      up       up   QSFP+ or later         off
     Ethernet36      36,37,38,39      40G   9100    N/A    etp10  PortChannel0014      up       up   QSFP+ or later         off
     Ethernet40      40,41,42,43      40G   9100    N/A    etp11  PortChannel0017      up       up   QSFP+ or later         off
     Ethernet44      44,45,46,47      40G   9100    N/A    etp12  PortChannel0017      up       up   QSFP+ or later         off
     Ethernet48      48,49,50,51      40G   9100    N/A    etp13  PortChannel0020      up       up   QSFP+ or later         off
     Ethernet52      52,53,54,55      40G   9100    N/A    etp14  PortChannel0020      up       up   QSFP+ or later         off
     Ethernet56      56,57,58,59      40G   9100    N/A    etp15  PortChannel0023      up       up   QSFP+ or later         off
     Ethernet60      60,61,62,63      40G   9100    N/A    etp16  PortChannel0023      up       up   QSFP+ or later         off
     Ethernet64      64,65,66,67      40G   9100    N/A    etp17           routed      up       up   QSFP+ or later         off
     Ethernet68      68,69,70,71      40G   9100    N/A    etp18           routed      up       up   QSFP+ or later         off
     Ethernet72      72,73,74,75      40G   9100    N/A    etp19           routed      up       up   QSFP+ or later         off
     Ethernet76      76,77,78,79      40G   9100    N/A    etp20           routed      up       up   QSFP+ or later         off
     Ethernet80      80,81,82,83      40G   9100    N/A    etp21           routed      up       up   QSFP+ or later         off
     Ethernet84      84,85,86,87      40G   9100    N/A    etp22           routed      up       up   QSFP+ or later         off
     Ethernet88      88,89,90,91      40G   9100    N/A    etp23           routed      up       up   QSFP+ or later         off
     Ethernet92      92,93,94,95      40G   9100    N/A    etp24           routed      up       up   QSFP+ or later         off
     Ethernet96      96,97,98,99      40G   9100    N/A    etp25           routed      up       up   QSFP+ or later         off
    Ethernet100  100,101,102,103      40G   9100    N/A    etp26           routed      up       up   QSFP+ or later         off
    Ethernet104  104,105,106,107      40G   9100    N/A    etp27           routed      up       up   QSFP+ or later         off
    Ethernet108  108,109,110,111      40G   9100    N/A    etp28           routed      up       up   QSFP+ or later         off
    Ethernet112  112,113,114,115      40G   9100    N/A    etp29           routed      up       up   QSFP+ or later         off
    Ethernet116  116,117,118,119      40G   9100    N/A    etp30           routed      up       up   QSFP+ or later         off
    Ethernet120  120,121,122,123      40G   9100    N/A    etp31           routed      up       up   QSFP+ or later         off
    Ethernet124  124,125,126,127      40G   9100    N/A    etp32           routed      up       up   QSFP+ or later         off
PortChannel0002              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0005              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0008              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0011              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0014              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0017              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0020              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
PortChannel0023              N/A      80G   9100    N/A      N/A           routed      up       up              N/A         N/A
admin@str-msn2700-02:~$ show ip interface
Interface        Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
---------------  --------  -------------------  ------------  --------------  -------------
Ethernet64                 10.0.0.32/31         up/up         ARISTA01T0      10.0.0.33
Ethernet68                 10.0.0.34/31         up/up         ARISTA02T0      10.0.0.35
Ethernet72                 10.0.0.36/31         up/up         ARISTA03T0      10.0.0.37
Ethernet76                 10.0.0.38/31         up/up         ARISTA04T0      10.0.0.39
Ethernet80                 10.0.0.40/31         up/up         ARISTA05T0      10.0.0.41
Ethernet84                 10.0.0.42/31         up/up         ARISTA06T0      10.0.0.43
Ethernet88                 10.0.0.44/31         up/up         ARISTA07T0      10.0.0.45
Ethernet92                 10.0.0.46/31         up/up         ARISTA08T0      10.0.0.47
Ethernet96                 10.0.0.48/31         up/up         ARISTA09T0      10.0.0.49
Ethernet100                10.0.0.50/31         up/up         ARISTA10T0      10.0.0.51
Ethernet104                10.0.0.52/31         up/up         ARISTA11T0      10.0.0.53
Ethernet108                10.0.0.54/31         up/up         ARISTA12T0      10.0.0.55
Ethernet112                10.0.0.56/31         up/up         ARISTA13T0      10.0.0.57
Ethernet116                10.0.0.58/31         up/up         ARISTA14T0      10.0.0.59
Ethernet120                10.0.0.60/31         up/up         ARISTA15T0      10.0.0.61
Ethernet124                10.0.0.62/31         up/up         ARISTA16T0      10.0.0.63
Loopback0                  10.1.0.32/32         up/up         N/A             N/A
PortChannel0002            10.0.0.0/31          up/up         ARISTA01T2      10.0.0.1
PortChannel0005            10.0.0.4/31          up/up         ARISTA03T2      10.0.0.5
PortChannel0008            10.0.0.8/31          up/up         ARISTA05T2      10.0.0.9
PortChannel0011            10.0.0.12/31         up/up         ARISTA07T2      10.0.0.13
PortChannel0014            10.0.0.16/31         up/up         ARISTA09T2      10.0.0.17
PortChannel0017            10.0.0.20/31         up/up         ARISTA11T2      10.0.0.21
PortChannel0020            10.0.0.24/31         up/up         ARISTA13T2      10.0.0.25
docker0                    240.127.1.1/24       up/down       N/A             N/A
eth0                       10.3.147.45/23       up/up         N/A             N/A
lo                         127.0.0.1/8          up/up         N/A             N/A
admin@str-msn2700-02:~$ sudo ifconfig PortChannel0023
PortChannel0023: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9100
        inet6 fe80::7efe:90ff:fee8:10c0  prefixlen 64  scopeid 0x20<link>
        ether 7c:fe:90:e8:10:c0  txqueuelen 1000  (Ethernet)
        RX packets 496  bytes 34778 (33.9 KiB)
        RX errors 0  dropped 144  overruns 0  frame 0
        TX packets 4  bytes 520 (520.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

admin@str-msn2700-02:~$ sudo ifconfig PortChannel0020
PortChannel0020: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9100
        inet 10.0.0.24  netmask 255.255.255.254  broadcast 0.0.0.0
        inet6 fe80::7efe:90ff:fee8:10c0  prefixlen 64  scopeid 0x20<link>
        inet6 fc00::31  prefixlen 126  scopeid 0x0<global>
        ether 7c:fe:90:e8:10:c0  txqueuelen 1000  (Ethernet)
        RX packets 325  bytes 25064 (24.4 KiB)
        RX errors 0  dropped 144  overruns 0  frame 0
        TX packets 207  bytes 24076 (23.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Describe the results you expected:
PortChannel should not lose IP address.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

admin@str-msn2700-02:~$ show version

SONiC Software Version: SONiC.20191130.47
Distribution: Debian 9.13
Kernel: 4.9.0-11-2-amd64
Build commit: 272885b17
Build date: Fri Aug 28 01:36:16 UTC 2020
Built by: sonicbld@jenkins-slave-phx-2

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700
ASIC: mellanox
Serial Number: MT1552X08318
Uptime: 08:24:34 up  1:04,  1 user,  load average: 1.23, 1.42, 1.52

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-mlnx          20191130.47         d708c45aa851        392MB
docker-syncd-mlnx          latest              d708c45aa851        392MB
docker-sonic-telemetry     20191130.47         c64e3a90f987        353MB
docker-sonic-telemetry     latest              c64e3a90f987        353MB
docker-dhcp-relay          20191130.47         f32d0caa3625        299MB
docker-dhcp-relay          latest              f32d0caa3625        299MB
docker-router-advertiser   20191130.47         78246290cd83        289MB
docker-router-advertiser   latest              78246290cd83        289MB
docker-platform-monitor    20191130.47         7b73c4916338        659MB
docker-platform-monitor    latest              7b73c4916338        659MB
docker-fpm-frr             20191130.47         92ddcd4e9dd1        334MB
docker-fpm-frr             latest              92ddcd4e9dd1        334MB
docker-lldp-sv2            20191130.47         016f9d6c50f9        311MB
docker-lldp-sv2            latest              016f9d6c50f9        311MB
docker-sonic-restapi       20191130.47         774b93dd044b        301MB
docker-sonic-restapi       latest              774b93dd044b        301MB
docker-database            20191130.47         75ea94296731        289MB
docker-database            latest              75ea94296731        289MB
docker-acms                20191130.47         596de4dbeb69        182MB
docker-acms                latest              596de4dbeb69        182MB
docker-teamd               20191130.47         905e788a0862        314MB
docker-teamd               latest              905e788a0862        314MB
docker-snmp-sv2            20191130.47         4d07b3e20e8b        347MB
docker-snmp-sv2            latest              4d07b3e20e8b        347MB
docker-orchagent           20191130.47         cb88b9dcba5a        333MB
docker-orchagent           latest              cb88b9dcba5a        333MB
k8s.gcr.io/pause           3.2                 80d28bedfe5d        683kB

**Attach debug file `sudo generate_dump`:**

sonic_dump_str-msn2700-02_20200901_080506.tar.gz
@judyjoseph
Copy link
Contributor

judyjoseph commented Sep 2, 2020 •
edited
Loading

@rlhui This could be related to the issue for which this PR is raised sonic-net/sonic-swss#1407. Will follow up.
@wangxin Could you confirm if you are seeing this behavior very recently and not seen before ?

@wangxin
Copy link
Contributor Author

wangxin commented Sep 2, 2020

@judyjoseph This issue is observed on build SONiC.20191130.45 - 47. Build SONiC.20191130.44 does not have this issue.

@anshuv-mfst
Copy link

@judyjoseph to triage this issue.

@judyjoseph
Copy link
Contributor

Not able to reproduce the IP address issue. But the PR which could be the root cause sonic-net/sonic-swss#1407 is merged, will plan to include this in the next 201911 build.

@rlhui
Copy link
Contributor

rlhui commented Sep 4, 2020

@judyjoseph , @wangxin - as Xin can reproduce it, do we have a private build that has this change, for Xin to verify if this PR can address the issue? Thanks.

@judyjoseph
Copy link
Contributor

judyjoseph commented Sep 10, 2020 •
edited
Loading

@wangxin Since the next image is available already, could you try the build SONiC.20191130.48, instead of the private build which we discussed earlier. The PR is already merged in.

@wangxin
Copy link
Contributor Author

wangxin commented Sep 14, 2020

Tested build SONiC.20191130.49. Performed sudo config reload -y 10 times. This issue cannot be reproduced. I am closing it.

@wangxin wangxin closed this as completed Sep 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@judyjoseph judyjoseph

Labels
Issue for 201911
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wangxin @rlhui @judyjoseph @anshuv-mfst