Increase ip2me CIR/CBS for faster in-band file transfers #1000

rajendra-dendukuri · 2019-07-29T20:18:38Z

Increase incoming packet rate on in-band interfaces to support faster
download of large files. SONiC firmware image download over in-band can
take a lot of time if the incoming packet rate is limited to 600pps. This,
change increases it to 6000pps. Especially when used by Zero Touch Provisioning
or by sonic_installer for firmware upgrade over in-band interfaces.

Signed-off-by: Rajendra Dendukuri [email protected]

What I did
Increased packet rate for traffic destined to CPU.

Why I did it
This is required for ZTP or sonic_installer to download large files (firmware image) over in-band interface.

How I verified it
sonic_installer install -y "url" over in-band network

Details if related

Increase incoming packet rate on in-band interfaces to support faster download of large files. SONiC firmware image download over in-band can take a lot of time if the incoming packet rate is limited to 600pps. This, change increases it to 6000pps. Especially when used by Zero Touch Provisioning or by sonic_installer for firmware upgrade over in-band interfaces. Signed-off-by: Rajendra Dendukuri <[email protected]>

lguohan · 2019-07-30T05:08:41Z

retest this please

prsunny · 2019-07-30T19:14:16Z

swssconfig/sample/00-copp.config.json

@@ -50,8 +50,8 @@
            "queue": "1",
            "meter_type":"packets",
            "mode":"sr_tcm",
-            "cir":"600",
-            "cbs":"600",
+            "cir":"6000",


I think this can make the cpu/kernel busy if there is any packets like ping destined to switch. I would assume it was set to 600pps to control such packets. How about setting it to 1000pps instead of increasing it 10x.

Code downloads are the pain point. With 1000pps we will see speeds ~ 1.4MB/s and it takes 7 minutes to download a full sonic image compared to 8.4MB/s with 6000pps. I think 6000pps is not that bad. Most of the switches are having an intel CPU and 8.4MB/s b/w for in-band access looks like a number on the conservative side.

@rajendra-dendukuri what about keep the current value as 600 and leave the user to change the setting?

@stcheng Are you suggesting that end user changes the setting in source code and re-compile's an image?

jipanyang

Due to some issues related to SAI attribute property and the way copporch was implemented, we skip the reload of new copp json during warm reboot: https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-orchagent/swssconfig.sh#L44 while assuming that copp config won't change.

#582
#583

Probably it is time to revisit the assumption and find a proper fix.

lguohan · 2019-07-31T00:18:40Z

agree with jipan.

rajendra-dendukuri · 2019-07-31T19:13:38Z

So. What are the next steps for this?

lguohan · 2019-07-31T22:22:52Z

we are changing the default value, and there is concern on the cpu impact. can you provide cpu utilization data for various pps limit.

what is the limit for knet driver rx per second? Can we get cpu utilization measurement for 3000/6000/10000 pps?

warm reboot is another concern, but we can address it in another pr.

rajendra-dendukuri · 2019-08-06T04:10:39Z

When no policer installed, the stats that were observed before we start seeing drops
AS7712 24,821pps CPU: 27.72%
AS7816 37,792pps CPU: 14.45%

curl command CPU utilization when performing large file transfer over in-band
Intel Atom E3845 @ 1.91GHz, Quadcore
CIR/CBS 6000pps ; CPU 25%
CIR/CBS 3000pps ; CPU 14%
CIR/CBS 1000pps ; CPU 6%

we are changing the default value, and there is concern on the cpu impact. can you provide cpu utilization data for various pps limit.

what is the limit for knet driver rx per second? Can we get cpu utilization measurement for 3000/6000/10000 pps?

warm reboot is another concern, but we can address it in another pr.

rajendra-dendukuri · 2019-08-12T17:03:49Z

Can we try to conclude on this? Without these changes, in-band code download doesn't make sense.

rajendra-dendukuri · 2019-08-19T14:55:13Z

retest vs please

rajendra-dendukuri · 2019-09-10T19:56:23Z

retest vs please

rajendra-dendukuri · 2019-09-10T19:56:37Z

test this please

skg-net

@rajendra-dendukuri instead of changing the default config, can this dynamically changed only during ztp process?

rajendra-dendukuri · 2019-10-24T15:01:24Z

@rajendra-dendukuri instead of changing the default config, can this dynamically changed only during ztp process?

Increasing in-band bandwidth to CPU goes beyond ZTP. It is applicable even to a simple case of download code via in-band network using "sonic_installer install" command. Also there may be more applications which may require more bandwidth that what is set to by default.
We can have ZTP change the default value at runtime but it will also have to roll back the change. So if user changes the value to something else during ZTP process, the ZTP service may not be aware of it. It would have been easy if the CIR/CBS config was available in config db. However, that is not the case as we need to use a different copp.json file and a command to apply the changes.

Given above two reasons, I feel we should increase the default value to a reasonable agreed upon value.

skg-net · 2019-11-27T05:40:41Z

@rajendra-dendukuri

@rajendra-dendukuri instead of changing the default config, can this dynamically changed only during ztp process?

Increasing in-band bandwidth to CPU goes beyond ZTP. It is applicable even to a simple case of download code via in-band network using "sonic_installer install" command. Also there may be more applications which may require more bandwidth that what is set to by default.

We can have ZTP change the default value at runtime but it will also have to roll back the change. So if user changes the value to something else during ZTP process, the ZTP service may not be aware of it. It would have been easy if the CIR/CBS config was available in config db. However, that is not the case as we need to use a different copp.json file and a command to apply the changes.

Given above two reasons, I feel we should increase the default value to a reasonable agreed upon value.

@rajendra-dendukuri #2 we should block the user configuration during ZTP, since this will have other implication like loosing user configured data when doing ZTP disable.
#1 The value can be fine tuned based on the SONiC capabilities to handle the incoming rate. That value can be same or improved for ZTP scenario, since during that time no other protocols are running yet.

If value need to go beyond the SONiC defined default value, I prefer dynamically change based on the ZTP enable and disable.

Increase incoming packet rate on in-band interfaces to support faster download of large files. SONiC firmware image download over in-band can take a lot of time if the incoming packet rate is limited to 600pps. This, change increases it to 6000pps. Especially when used by Zero Touch Provisioning or by sonic_installer for firmware upgrade over in-band interfaces. Signed-off-by: Rajendra Dendukuri <[email protected]>

Signed-off-by: Stepan Blyschak [email protected] The motivation for this change is described in the proposal sonic-net/SONiC#935 and proposal in SAI opencomputeproject/SAI#1404 NOTE: Requires to update SAI once opencomputeproject/SAI#1404 is in.

rajendra-dendukuri changed the title ~~Increase ip2me CIR/CBR for faster in-band file transfers~~ Increase ip2me CIR/CBS for faster in-band file transfers Jul 29, 2019

rajendra-dendukuri mentioned this pull request Jul 29, 2019

Trap DHCPv6 packets for supporting ZTP over in-band interfaces using … #997

Merged

stcheng added the Enhancement ➕ label Jul 30, 2019

prsunny reviewed Jul 30, 2019

View reviewed changes

jipanyang reviewed Jul 30, 2019

View reviewed changes

rajendra-dendukuri closed this Aug 6, 2019

rajendra-dendukuri reopened this Aug 6, 2019

skg-net reviewed Oct 24, 2019

View reviewed changes

lguohan merged commit 03be983 into sonic-net:master Dec 10, 2019

msosyak mentioned this pull request Jan 6, 2020

[COPP] Fix COPP test sonic-net/sonic-mgmt#1317

Merged

1 task

rlhui added the request_for_201911_branch label Feb 25, 2020

abdosi added the Included in 201911 Branch label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Increase ip2me CIR/CBS for faster in-band file transfers #1000

Increase ip2me CIR/CBS for faster in-band file transfers #1000

rajendra-dendukuri commented Jul 29, 2019

lguohan commented Jul 30, 2019

prsunny Jul 30, 2019

rajendra-dendukuri Jul 30, 2019

stcheng Aug 6, 2019

rajendra-dendukuri Aug 6, 2019

jipanyang left a comment

lguohan commented Jul 31, 2019

rajendra-dendukuri commented Jul 31, 2019

lguohan commented Jul 31, 2019

rajendra-dendukuri commented Aug 6, 2019 •

edited

Loading

rajendra-dendukuri commented Aug 12, 2019

rajendra-dendukuri commented Aug 19, 2019

rajendra-dendukuri commented Sep 10, 2019

rajendra-dendukuri commented Sep 10, 2019

skg-net left a comment

rajendra-dendukuri commented Oct 24, 2019

skg-net commented Nov 27, 2019

Increase ip2me CIR/CBS for faster in-band file transfers #1000

Increase ip2me CIR/CBS for faster in-band file transfers #1000

Conversation

rajendra-dendukuri commented Jul 29, 2019

lguohan commented Jul 30, 2019

prsunny Jul 30, 2019

Choose a reason for hiding this comment

rajendra-dendukuri Jul 30, 2019

Choose a reason for hiding this comment

stcheng Aug 6, 2019

Choose a reason for hiding this comment

rajendra-dendukuri Aug 6, 2019

Choose a reason for hiding this comment

jipanyang left a comment

Choose a reason for hiding this comment

lguohan commented Jul 31, 2019

rajendra-dendukuri commented Jul 31, 2019

lguohan commented Jul 31, 2019

rajendra-dendukuri commented Aug 6, 2019 • edited Loading

rajendra-dendukuri commented Aug 12, 2019

rajendra-dendukuri commented Aug 19, 2019

rajendra-dendukuri commented Sep 10, 2019

rajendra-dendukuri commented Sep 10, 2019

skg-net left a comment

Choose a reason for hiding this comment

rajendra-dendukuri commented Oct 24, 2019

skg-net commented Nov 27, 2019

rajendra-dendukuri commented Aug 6, 2019 •

edited

Loading