Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cold-boot: teammgrd PORTCHANNEL configuration flushed by swss docker start script swss.sh #2606

Closed
jipanyang opened this issue Feb 26, 2019 · 4 comments

Comments

@jipanyang
Copy link
Collaborator

Description
Upon cold boot, teamd service starts immediately after updategraph.service, teammgrd reads the portchannel configuration and push configurations like min_links, fallback, mtu down to appDB.

SWSS server starts after updategraph.service and interfaces-config.service, upon cold start, it will flushes appDB. All the configuration done teammgrd previously will be lost.

teammgrd started at Feb 26 05:11:52.768030

Feb 26 05:11:52.768030 vlab-01 NOTICE teamd#teammgrd: :- main:25: --- Starting teammrgd ---
 866 Feb 26 05:11:52.768030 vlab-01 NOTICE teamd#teammgrd: :- loadRedisScript:28: lua script loaded, sha: 88270a7c5c90583e56425aca8af8a4b8c39fe757
 867 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd Using team device "PortChannel0001".
 868 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd
 869 Feb 26 05:11:52.868304 vlab-01 INFO teamd#supervisord: teammgrd Using PID file "/var/run/teamd/PortChannel0001.pid"
teamd.log:Feb 26 05:11:52.939019 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0001 admin status to up
teamd.log:Feb 26 05:11:52.939019 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0001 MTU to 9100
teamd.log:Feb 26 05:11:53.011069 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0002 admin status to up
teamd.log:Feb 26 05:11:53.038120 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0002 MTU to 9100
teamd.log:Feb 26 05:11:53.089232 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0003 admin status to up
teamd.log:Feb 26 05:11:53.102062 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0003 MTU to 9100
teamd.log:Feb 26 05:11:53.194389 vlab-01 NOTICE teamd#teammgrd: :- setLagAdminStatus:368: Set port channel PortChannel0004 admin status to up
teamd.log:Feb 26 05:11:53.213747 vlab-01 NOTICE teamd#teammgrd: :- setLagMtu:418: Set port channel PortChannel0004 MTU to 9100

swss.sh called at Feb 26 05:12:06.759357

Feb 26 05:12:06.759357 vlab-01 INFO swss.sh[2807]: OK
 2721 Feb 26 05:12:07.451206 vlab-01 INFO swss.sh[2807]: Warning: ignore interface 'fortyGigE0/24' in DEVICE_NEIGHBOR as it is not in the port_config.ini
 2722 Feb 26 05:12:07.451679 vlab-01 INFO swss.sh[2807]: Warning: ignore interface 'fortyGigE0/40' in DEVICE_NEIGHBOR as it is not in the port_config.ini

When creating router interface on the portchannel, it has default mtu 1492.

root@vlab-01:/var/log# grep "Create router interface" *
grep: lost+found: Is a directory
grep: swss: Is a directory
syslog:Feb 26 05:12:21.100838 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface Vlan1000 MTU 9100
syslog:Feb 26 05:12:21.683519 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0001 MTU 1492
syslog:Feb 26 05:12:21.684545 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0002 MTU 1492
syslog:Feb 26 05:12:21.692284 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs: Create router interface PortChannel0003 MTU 1492
syslog:Feb 26 05:12:21.701900 vlab-01 NOTICE swss#orchagent: :- addRouterIntfs:477: Create router interface PortChannel0004 MTU 1492

configDB

127.0.0.1:6379[4]> hgetall "PORTCHANNEL|PortChannel0001"
 1) "mtu"
 2) "9100"
 3) "admin_status"
 4) "up"
 5) "members@"
 6) "Ethernet112"
 7) "port_offset"
 8) "0"
 9) "min_links"
10) "1"

appDB

127.0.0.1:6379> hgetall "LAG_TABLE:PortChannel0001"
1) "admin_status"
2) "up"
3) "oper_status"
4) "up"

At least "After=updategraph.service swss.service" is needed for teamd service.

@dawnbeauty
Copy link

another scenario, sudo systemctl restart swss without enable warm-boot, all the lag configuration in app db done by teammgrd also be flushed.

@jipanyang
Copy link
Collaborator Author

Actually it affects warm reboot too. After warm reboot, the correct mtu will be applied on portchannel interface and cause traffic disruption.

@stcheng
Copy link
Contributor

stcheng commented Mar 5, 2019

I think the better fix is not pushing the configurations into application database; but reading configuration database instead of application database.

@yxieca
Copy link
Contributor

yxieca commented Sep 12, 2019

We added service dependency to let teamd start after swss and that change addressed this issue.

@yxieca yxieca closed this as completed Sep 12, 2019
yxieca added a commit to yxieca/sonic-buildimage that referenced this issue Jan 12, 2023
…le head

utilties:
* 7fc3fb7 2023-01-06 | [storyteller] add link prober state change to story teller (sonic-net#2585) (HEAD -> 202205, github/202205) [Jing Zhang]
* d8202e8 2023-01-12 | [202205] Added a new option in show queue counters command to display voq statistics (sonic-net#2591) [Sambath Kumar Balasubramanian]
* 98dde31 2023-01-11 | resolved conflicts (sonic-net#2589) [kannankvs]

swss:
* 268c3f3 2023-01-11 | Added new attributes for Vnet and Vxlan ecmp configurations. (sonic-net#2584) (HEAD -> 202205, github/202205) [siqbal1986]
* 50235b6 2023-01-12 | [routesync] Fix for stale dynamic neighbor (sonic-net#2553) [vganesan-nokia]
* ad9d826 2023-01-12 | [202205][voq] Add voq counters. (sonic-net#2606) [Sambath Kumar Balasubramanian]
* 34995f1 2023-01-10 | Revert "[voq][chassis]Add show fabric counters port/queue commands (sonic-net#2522)" (sonic-net#2612) [Arvindsrinivasan Lakshmi Narasimhan]

sairedis:
* 92e6442 2023-01-05 | enable cisco8000 SAI bulk API feature (sonic-net#1153) (sonic-net#1164) (github/202205) [Keith Lu]

platform-daemons:
* 10eb2e6 2023-01-06 | Fix bug where transceiver info is missing after port breakout change (sonic-net#329) (HEAD -> 202205) [Tal Berlowitz]

Signed-off-by: Ying Xie <[email protected]>
yxieca added a commit that referenced this issue Jan 13, 2023
…le head (#13353)

utilties:
* 7fc3fb7 2023-01-06 | [storyteller] add link prober state change to story teller (#2585) (HEAD -> 202205, github/202205) [Jing Zhang]
* d8202e8 2023-01-12 | [202205] Added a new option in show queue counters command to display voq statistics (#2591) [Sambath Kumar Balasubramanian]
* 98dde31 2023-01-11 | resolved conflicts (#2589) [kannankvs]

swss:
* 268c3f3 2023-01-11 | Added new attributes for Vnet and Vxlan ecmp configurations. (#2584) (HEAD -> 202205, github/202205) [siqbal1986]
* 50235b6 2023-01-12 | [routesync] Fix for stale dynamic neighbor (#2553) [vganesan-nokia]
* ad9d826 2023-01-12 | [202205][voq] Add voq counters. (#2606) [Sambath Kumar Balasubramanian]
* 34995f1 2023-01-10 | Revert "[voq][chassis]Add show fabric counters port/queue commands (#2522)" (#2612) [Arvindsrinivasan Lakshmi Narasimhan]

sairedis:
* 92e6442 2023-01-05 | enable cisco8000 SAI bulk API feature (#1153) (#1164) (github/202205) [Keith Lu]

platform-daemons:
* 10eb2e6 2023-01-06 | Fix bug where transceiver info is missing after port breakout change (#329) (HEAD -> 202205) [Tal Berlowitz]

Signed-off-by: Ying Xie <[email protected]>

Signed-off-by: Ying Xie <[email protected]>
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this issue Apr 11, 2023
Update sonic-utilities submodule pointer to include the following:
* a1057b27 [config reload]Config Reload Enhancement ([sonic-net#2693](sonic-net/sonic-utilities#2693))
* 04d0b34a [voq][chassis][generate_dump] [BCM] Dump only the relevant BCM commands for fabric cards ([sonic-net#2606](sonic-net/sonic-utilities#2606))

Signed-off-by: dgsudharsan <[email protected]>
dgsudharsan added a commit to dgsudharsan/sonic-buildimage that referenced this issue Apr 11, 2023
Update sonic-utilities submodule pointer to include the following:
* a1057b27 [config reload]Config Reload Enhancement ([sonic-net#2693](sonic-net/sonic-utilities#2693))
* 04d0b34a [voq][chassis][generate_dump] [BCM] Dump only the relevant BCM commands for fabric cards ([sonic-net#2606](sonic-net/sonic-utilities#2606))

Signed-off-by: dgsudharsan <[email protected]>
yxieca pushed a commit that referenced this issue Apr 20, 2023
…atically (#14752)

src/sonic-utilities

* ece22b7d - (HEAD -> 202205, origin/202205) Revert "[GCU] Add PFC_WD RDMA validator  (#2781)" (4 minutes ago) [Ying Xie]
* 7d16b184 - Remove the no use new line in show version (#2792) (21 hours ago) [xumia]
* 3a880a2b - Support to display the SONiC OS Version in the command show version (#2787) (21 hours ago) [xumia]
* a5199f75 - [voq][chassis][generate_dump] [BCM] Dump only the relevant BCM commands for fabric cards (#2606) (21 hours ago) [saksarav-nokia]
* 2410d364 - Fixed a bug in "show vnet routes all" causing screen overrun. (#2644) (#2801) (
dprital added a commit to dprital/sonic-buildimage that referenced this issue May 1, 2023
Update sonic-utilities submodule pointer to include the following:
* 600377f7 [DPB]Fixing typo in config breakout output ([sonic-net#2802](sonic-net/sonic-utilities#2802))
* 8ae2424a [config]Support multi-asic  Golden Config override ([sonic-net#2738](sonic-net/sonic-utilities#2738))
* 79003ab2 [chassis]: remote cli commands infra for sonic chassis ([sonic-net#2701](sonic-net/sonic-utilities#2701))
* cbc55eeb [voq][chassis][generate_dump] [BCM] Dump only the relevant BCM commands for fabric cards ([sonic-net#2606](sonic-net/sonic-utilities#2606))
* 39c94b7e [GCU] Prohibit removal of PFC_WD POLL_INTERVAL field ([sonic-net#2545](sonic-net/sonic-utilities#2545))

Signed-off-by: dprital <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants