From 7002b69dcdb9a34f15f9bab0ca554198232e1deb Mon Sep 17 00:00:00 2001 From: stepanblyschak Date: Thu, 11 Aug 2022 15:05:50 +0300 Subject: [PATCH 01/43] [BGP] Add BGP suppress FIB pending Signed-off-by: stepanblyschak --- doc/BGP/BGP-supress-fib-pending.md | 353 +++++++++++++++++++++++++++++ doc/BGP/img/pic.png | Bin 0 -> 23827 bytes 2 files changed, 353 insertions(+) create mode 100644 doc/BGP/BGP-supress-fib-pending.md create mode 100644 doc/BGP/img/pic.png diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md new file mode 100644 index 0000000000..fe899dae2a --- /dev/null +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -0,0 +1,353 @@ + +# BGP Supress FIB Pending # + + +## Table of Content + +- [1. Revision](#1-revision) +- [2. Scope](#2-scope) +- [3. Definitions/Abbreviations](#3-definitionsabbreviations) +- [4. Overview](#4-overview) +- [5. Requirements](#5-requirements) +- [6. Architecture Design](#6-architecture-design) +- [7. High-Level Design](#7-high-level-design) +- [RouteOrch](#routeorch) + - [High-Level Flow Diagram](#high-level-flow-diagram) +- [8. SAI API](#8-sai-api) +- [9. Configuration and management](#9-configuration-and-management) + - [9.1. Manifest (if the feature is an Application Extension)](#91-manifest-if-the-feature-is-an-application-extension) + - [9.2. CLI/YANG model Enhancements](#92-cliyang-model-enhancements) + - [9.3. Config DB Enhancements](#93-config-db-enhancements) +- [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) +- [11. Restrictions/Limitations](#11-restrictionslimitations) +- [12. Testing Requirements/Design](#12-testing-requirementsdesign) + - [12.1. Unit Test cases](#121-unit-test-cases) + - [12.2. System Test cases](#122-system-test-cases) +- [13. Open/Action items - if any](#13-openaction-items---if-any) + +### 1. Revision + +| Revision | Date | Author | Change Description | +| -------- | ----------- | ---------------- | ------------------ | +| 1.0 | Sep 15 2022 | Stepan Blyshchak | Initial proposal | + +### 2. Scope + +This document describes a feedback mechanism that allows BGP not to adveritise routes that haven't been programmed yet or failed to be programmed to ASIC. + +### 3. Definitions/Abbreviations + +| Definitions/Abbreviation | Description | +| ------------------------ | ---------------------------- | +| BGP | Border Gateway Protocol | +| FRR | Free Range Routing | +| SWSS | Switch state service | +| SYNCD | ASIC syncrhonization service | +| FPM | Forwarding Plane Manager | +| SAI | Switch Abstraction Interface | + +### 4. Overview + +The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. + +The solution is to provide a configurable option to check for the FIB install status of the prefixes and advertise to peers if the prefixes are successfully installed in the FIB. The advertisement of the prefixes are suppressed if it is not installed in FIB. + +The following conditions apply will apply when checking for route installation status in FIB: + +- The advertisement or suppression of routes based on FIB install status applies only for newly learnt routes from peer (routes which are not in BGP local RIB). +- If the route received from peer already exists in BGP local RIB and route attributes have changed (best path changed), the old path is deleted and new path is installed in FIB. The FIB install status will not have any effect. Therefore only when the route is received first time the checks apply. +- The feature will not apply for routes learnt through other means like redistribution to bgp from other protocols. This is applicable only to peer learnt routes. +- If a route is installed in FIB and then gets deleted from the dataplane, then routes will not be withdrawn from peers. This will be considered as dataplane issue. +- The feature will slightly increase the time required to advertise the routes to peers since the route install status needs to be received from the FIB +- If routes are received by the peer before the configuration is applied, then the bgp sessions need to be reset for the configuration to take effect. +- If the route which is already installed in dataplane is removed for some reason, sending withdraw message to peers is not currently supported. + +[FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst) + +Consider the following scenario: + + +##### Figure 1. Use case scenario + +

+Figure 1. Use case scenario +

+ +The problem with BGP programming occurs after the T1 switch is rebooted: +1. First, the T1 FRR learns a default route from at least 1 T2 +2. The T0 advertises it’s prefixes to T1 +3. FRR advertises the prefixes to T2 without waiting for them to be programmed in the ASIC +4. T2 starts forwarding traffic for prefixes not yet programmed, according to T1’s routing table, T1 sends it back to a default route – same T2 + + +When the traffic is bounced back on lossless queue, buffers on both sides are overflown, credit loop happens, with PFC storm and watchdog triggered shutting down the port. +To avoid that, the route programming has to be synchronous down to the ASIC to avoid credit loops. + +### 5. Requirements + +This section list out all the requirements for the HLD coverage and exemptions (not supported) if any for this design. + +### 6. Architecture Design + +Described functionality does not require changes to the current SONiC architecture. This design follows existing SONiC architecture approaches and uses existing SONiC infrastrcuture. + +### 7. High-Level Design + +In order to support this feature + +This section covers the high level design of the feature/enhancement. This section covers the following points in detail. + + - Is it a built-in SONiC feature or a SONiC Application Extension? + - What are the modules and sub-modules that are modified for this design? + - What are the repositories that would be changed? + - Module/sub-module interfaces and dependencies. + - SWSS and Syncd changes in detail + - DB and Schema changes (APP_DB, ASIC_DB, COUNTERS_DB, LOGLEVEL_DB, CONFIG_DB, STATE_DB) + - Sequence diagram if required. + - Linux dependencies and interface + - Warm reboot requirements/dependencies + - Fastboot requirements/dependencies + - Scalability and performance requirements/impact + - Memory requirements + - Docker dependency + - Build dependency if any + - Management interfaces - SNMP, CLI, RestAPI, etc., + - Serviceability and Debug (logging, counters, trace etc) related design + - Is this change specific to any platform? Are there dependencies for platforms to implement anything to make this feature work? If yes, explain in detail and inform community in advance. + - SAI API requirements, CLI requirements, ConfigDB requirements. Design is covered in following sections. + +### RouteOrch + +```c++ +auto status = ReturnCode(saiStatus) << "Failed to create route " << ipPrefix.to_string().c_str() << " with next hop(s) " << nextHops.to_string().c_str(); +SWSS_LOG_ERROR("%s", status.message().c_str()); + +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), status); +``` + +#### High-Level Flow Diagram + + +##### Figure 2. BGP Configuration Flow Diagram + +```mermaid +sequenceDiagram + participant CONFIG_DB + participant A as BGP /usr/bin/docker_init.sh + participant S as sonic-cfggen + participant T as bgpd.main.j2 + participant E as /etc/frr/bgpd.conf + + activate A + + A -->> S: /usr/share/sonic/templates/bgpd/gen_bgpd.conf.j2 + + activate S + + S -->> T:
+ + activate T + + CONFIG_DB -->> T: DEVICE_METADATA|localhost + + alt "bgp-suppress-fib-pending" == "enable" + T --> E: configure "bgp suppress-fib-pending" for router + end + + + deactivate T + + deactivate S + + A -->> A: start supervisord + + deactivate A +``` + + +##### Figure 3. BGP-SWSS Flow Diagram + +```mermaid +sequenceDiagram + participant orchagent + participant APPL_DB + participant ASIC_DB + participant syncd + participant APPL_STATE_DB + participant fpmsyncd + participant Kernel + participant zebra + participant bgpd + + Note right of bgpd: A new prefix is received + + activate bgpd + bgpd -->> zebra:
+ activate zebra + + zebra -->> Kernel:
+ activate Kernel + + Note right of Kernel: Route is programmed to kernel
without RTM_F_OFFLOAD flag + + Kernel -->> zebra:
+ deactivate Kernel + zebra -->> fpmsyncd: Update via FPM channel + activate fpmsyncd + + fpmsyncd -->> APPL_DB: Set ROUTE_TABLE entry + activate APPL_DB + APPL_DB -->> orchagent:
+ activate orchagent + deactivate APPL_DB + + deactivate fpmsyncd + + deactivate zebra + deactivate bgpd + + loop for each route + orchagent -->> orchagent: Prepare bulk create request + end + + orchagent -->> ASIC_DB: sai_route_api->create_route_entries + activate ASIC_DB + ASIC_DB -->> syncd:
+ activate syncd + syncd -->> ASIC_DB: SAI bulk status + deactivate syncd + ASIC_DB -->> orchagent: SAI bulk status + deactivate ASIC_DB + loop for each route entry creation SAI status: + orchagent -->> APPL_STATE_DB: Set ROUTE_TABLE entry status + activate APPL_STATE_DB + APPL_STATE_DB -->> fpmsyncd:
+ + activate fpmsyncd + + alt Route creation successful: + fpmsyncd -->> Kernel: Set RTM_F_OFFLOAD + activate Kernel + + Kernel -->> zebra:
+ + activate zebra + + zebra -->> bgpd:
+ activate bgpd + + Note right of bgpd: The prefix advertisement was suppressed
Now RTM_F_OFFLOAD appears in the flags
and it is going to be advertised to peers. + + deactivate bgpd + + deactivate zebra + + deactivate Kernel + end + + deactivate fpmsyncd + + APPL_STATE_DB -->> orchagent:
+ deactivate APPL_STATE_DB + end + + + deactivate orchagent +``` + +### 8. SAI API + +No new SAI API or changes to SAI design and behaviour needed for this functionality. + +### 9. Configuration and management +This section should have sub-sections for all types of configuration and management related design. Example sub-sections for "CLI" and "Config DB" are given below. Sub-sections related to data models (YANG, REST, gNMI, etc.,) should be added as required. + +#### 9.1. Manifest (if the feature is an Application Extension) + +This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. + +#### 9.2. CLI/YANG model Enhancements + +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which be set to ```"enable"``` or ```"disable"```. + +Snippet of ```sonic-device_metatadata.yang```: + +```yang +module sonic-device_metadata { + revision 2022-09-15 { + description "Add BGP suppress FIB pending configuration knob"; + } + + container sonic-device_metadata { + + container DEVICE_METADATA { + + description "DEVICE_METADATA part of config_db.json"; + + container localhost{ + + leaf bgp-suppress-fib-pending { + description "Enable BGP suppress FIB pending feature. BGP will wait for route + FIB intallation before announcing routes. This configuration requires + restarting BGP sessions."; + type enumeration { + enum enable; + enum disable; + } + default disable; + + must "((current() = 'disable') or (current() = 'enable' and../synchronous_mode = 'enable'))"; + } + + } + } + } +} +``` + +This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. + +No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. + +#### 9.3. Config DB Enhancements + +Configuration schema in ABNF format: + +```abnf +; DEVICE_METADATA table +key = DEVICE_METADATA:localhost ; Device metadata configuration table +bgp-suppress-fib-pending = "enable"/"disable" ; Globally enable/disable BGP suppress-fib-pending feature, by default this flag is disabled +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "DEVICE_METADATA": { + "localhost": { + "bgp-suppress-fib-pending": "enable" + } + } +} +``` + +This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. + +### 10. Warmboot and Fastboot Design Impact +Mention whether this feature/enhancement has got any requirements/dependencies/impact w.r.t. warmboot and fastboot. Ensure that existing warmboot/fastboot feature is not affected due to this design and explain the same. + +### 11. Restrictions/Limitations + +### 12. Testing Requirements/Design +Explain what kind of unit testing, system testing, regression testing, warmboot/fastboot testing, etc., +Ensure that the existing warmboot/fastboot requirements are met. For example, if the current warmboot feature expects maximum of 1 second or zero second data disruption, the same should be met even after the new feature/enhancement is implemented. Explain the same here. +Example sub-sections for unit test cases and system test cases are given below. + +#### 12.1. Unit Test cases + +#### 12.2. System Test cases + +### 13. Open/Action items - if any + + +NOTE: All the sections and sub-sections given above are mandatory in the design document. Users can add additional sections/sub-sections if required. \ No newline at end of file diff --git a/doc/BGP/img/pic.png b/doc/BGP/img/pic.png new file mode 100644 index 0000000000000000000000000000000000000000..33934b271440c6d1081574b3fb07b182b027e9e1 GIT binary patch literal 23827 zcmd42WmHsc8#b(hAV`U%l+q3Y64E80bjb`|(nv{zlynJ53`lnkCEZ;zNOwyN$RII8 z_qXxBpXd3$wZ6aaTJH}QYwxx9wa@Fkj^peJQ&pD1$9;l(=gu8`Ia%psUt z&x3ow9XU~3Tj1Yar*|@vcglvSw}1;wi`Po8@7$?~et7x*K5&imQC8dO&K-iT+rPVg z_JyW*?&uZDNxufW8|<`PsFS$0-rQVyDyN-m*xGvg2)KOM_9`ASUrWDYOj)cN{nKyD z&sEl7c}&6kWbcXnU8#5qluuYTia3CkT#bZ=BRljd{m+jyrgs_wOeLA&CxXtxeE7Vg ziMu{+p6St_$ljdbb^UZc$yvTao-|$DP_hIP(F=kFp9RViLX;VqDe#u;Ew9De=$T5N z#y%D_bF=pDVqieVDSn>r$S_T>FY~5RG-&jho?h3?kPdBEMM%pF-P5OXGrTVE*%UiN zBh#x+;4tBV4-?cvhIL*RsEl_*YdAzL=h!@|j>WF11d(Qf>-C>z%Y$=evM>F&$3w{Bdk=f3tSrzx-3Y%Gt9chz&aTq1TNZev(?2T*Fls=e}QHNPo5h zbDJFMJmDw(X8A>h!NFFUe9G4)_q0Eb1)0LoZB^)KENE%DC!ix@e->12Z?E4nxx`OI z?QzM2c6UDia((Kg?P9dq-{gC_eo1~2vA@}4cEY#I7Fo%4t+yIVzfA?9X~F0SZDJUe z^@{Ct!J1}nnvqF)?2ri$3(C}vG(YVe=1gPz@Q3PDjjP_T`;q&lb|?OR1i#L8MbT5l zKyg5q_*Qr6plKMxr?BDpBu2YGs1eSU)aND3X)bAeEc6o4$HhNdowo-eRuQEgpvj4&!s5xy{n@IGkC;Ne2^fy7mNpa3>R*_@FCchTNbEJi&cHsL1a%)+_j zXP1S79b{5v=cDN88%AFX4KvN*wa{*S5wW=LNKx9V?wC_1v6zBIZ*h)=ul&5cB|eUl z)czor@l3q-vz?1&&RHLqn}#_mH=*EQ-{c?Pl4H13AisVGVHdYGaJikm$;&Riv^~jL zI+;QwEZ}RgI50jU&RBOyt8qyJp=8UJd_woa&B#_?fCZBR=}~S|8wgJw>=55~+;Xdr zX=Q?6^*^^WI6cej(RFmfwQp-H%Ip^xDDIAtmZj#}M%^mK0V9l|O$5iHd;Gj@4M<}Tr9^s%mOdo~Jsht^af(alg3X0V=F;vM z3dSf3cRdSReO}_|&NVeTk=S%UI?hiz@1mEI>sIMgm9rkFWWnRoJY6CqY7KUO({fpU z*~W{qWV~Kj)3NT_@Elr*P-NCy?WL_#U;PnRF!#mquv+qW0Y`kXA=k4hDJaV;8o#oj z3~~`9Riyf=P_%VUj5`T!wDzhj9=d*vErF7(-WuvR8)%#vK+&q~Li-vPu}peYh`RQTB`Tr0qJry)ilv*o5Y5O$ zf#*{#SYBJf|0zuTq<&~|j#2XVOTgZyfW5^cqaeZ$XB&dF@FD5BUM}l%Ns@a`Vd4k% zL%(FH{{;fq*ZPV+lQ;q5fbK^a(c6r){eDi9B`J@K0X@WM`1S!j>0x%jh#qEiFp2Ja zaf=|CRcO6mUrdJ2HNw8w$@5~SAV}*0B;6S}_BSa9Fj|ptSueh5U8et`4fu#(pGkr{ zV~?AVKyauQF7WiaVN;yzdcrGGD1d}-N2iL!{Kxg`#L?@}cs;{$DHlqjSj$56`aLD| zbb_Jf4h0`4AcZxfSpS-k^|moUjeicaz>4SWS44`hzLU4ji(C#INn&7ey@k>v@5Z&{ zph$CpM+Iq)Qju_5Wwc+p>e>iT5W-SlnUqC?2RI_^OjgSqHG%uVTj z6alDm-rDKr3=v66$s}?mM@#^`rpHz~>$soq2XMUX{|_3BOKU%3cQnHXc3UbJoNB5y z^ce#gAHYPyPK|xllpPLCqgjH0ckQCPp%Msuu^Yl5H$!~{<&!6BiMY?gK7&-~uPdIN z*mW@+5X%^U6)^sFOZHJ&5!~nqP6wuNc-PhWxQLYue0DIEk?3|=z>1vUUltzKD=*A3 z{)_C;L1Inx=u~II)W>N@L#K;W5*3JTT&3QFFglos_HUnwi(YudBY>fL3@BESTQl{~ zS+LmIGAmG5^soK6x!Pk`k~eG5?n+*&SPUK z@13vaaN)mbZ@z`i7h+Avid>h?QRAt;us5m*j@Ta z1VG+srT_4;sx!#iJr??dK@pdL$ z|IJDy66gPe>lpo2`Hk{6J@@gO=tvKB&H_^0ppDWMOCMrRaD!*P?zV2_6|#g_hWMF7 zcS)Z`>_1eakME8Nm_6YcUh3i$t^^&FL0}3BQt&9^s0|heNoZ3@*dqziidVfZfVk#i z3}IgdR=;=q0w}Du-+LOxB?>E0q(r*zQQ-2kjOQD0a;%OmcFr0g?)X8x&5*SvKc|LtM_I z7_q__uxJ7MOdi?I;&E##7Z@Fwx^l#~H^5WkI;|~6AJ~~la9iqva?a%b0Q$;g7&+Q) z6iI7^{zXrQ+&}sa{Z7pP!%)*8^*CYBWob(Ad3rb~Lr6)M7kK5@*iqAbTZ?E_ch0{1 z&;R1Iz9#8~gET=jLvd@XPl5R-!#!d3vtA(;(~siMHnQN0?SP->FtB%s1MtsL-VHTP zFB0@Ui)vhF<`F@kmg1*(ZQ51*l*|qSxnU^Ok;iLb=esYI{IzN&r4ese)!Y{XyXX0T za&o#Rc0);)4hv{mljWw&p!nr4PDa)lbqXD~P_OFR>eqW~;CBDnKHjbCpJBXqvfTgO z$kUcozOg4LrOsF;UGA9HUA`xLkm5~!J8uB70CImMuh#@Zpn0XPELS&HaD5Q~NcR6!B2|4k_(3S$aWzD^C_7wipI zW$FL5cG*RFtRPcsGof8y zFZs8TI-TsO<49OXh8WI73x*PQaP$fxOE`$gI0c3dpBms{T7=00yW${3}XP(Owh0`DqyEbY+cC@kIkI5k-lT+FKKadu=u3f;aKvT4SE zir3Y`9C`bvCu@{k^L}xO?G)uPj9e8KsH8pZCIZ&xuC!}y{u!X8pM+72FFbQ!L}8sWLW}P^@E3W>Or>2~L*V{A%x(3qOUds5N||TE z5o_+sh`j`NWOFI<5c6z$*XwM@Hmw8tU*p50Ad+gBj^|yhekQGECb_KAb@gsynNFER zE4T}!iTfgLyr@~-<97pV&v1cZQVVtvcQRd*tt$^pl_XRFE2@44RSCy(Ij0=w`^lp4 zpWp~J%UEU_4$n}e&?_qY!?sMJA*i#e+HhRGxj&BD0*j#{&g)_Y^gXLKwAJ1o94uX$ zY7{!2WP*RL#sBM$txHP>s;+}#3Q0+&EhH?e4or`>i*dP!m$qr}khI|-vlF3dbCS1{ zKbBn3Q&ig`>YR}Zp=7WH2(^fQ<%uFx;MS)4b-y{NkDVG?v3J@w)K3q|Xbkze*=5W0 zhG|qkF*WBbjIz+N4z(DBKD4(djJ(VR!h^X+OGetDzBuXCxy!=?PRkU!pRMI7_RN-} zZPkZ!7rHsEMjF~?U4UOuunrMY`ax3Yi1R=xQ~BGk)k&BK8;2=5KVjsTL8OB)Q`8SO z2ir_Eg-t6%G1_LS)Q#U`$}Hin?#tOvg*I}Za0XtOG8a0Iss-P>Qgzo%;d)(t+I{y=xSANpNncGl<=sdZ^pz}(J8gcY zcA@|{jnsN8y;49j(?Z9r*Cy2L0@Z8rSrW`0hU{~(*k**{v`z0uTT$($(Y5;Ol{>Ob zfp)iUb(cGgjgt22OeI6_6t!vSd|&~OonXTkYpeVWKxZGlbqOo9iE0}=l3-%I2jSoa z94X*^!(4wX2QMZ*9GIt9vi~Qvj6a?DBcgq!pQXv~H1|i$Wi!NIZj&8~KA9X7a!LMr znhEbg>^JCgavoHlV<4*qTd%SXRb3;Mm*tKB8;U8iSm_9d7#c$M2jb5fQ&|gz-5X6m zU|fcDHnr)l7L{5^r7+%s>8Zx=h|FOiS7KtOHS{QaMDpYQntqi0!Yf)$bjL=vr;>ML z(j2vb=GA1(Y7*}Aiy^SF{o>@{!_u$rS@1WQGRxxxTdB}XYqzoJ#dPiempgVx%j)%S zNY!yBR3ptnTQqprJV)@NhX`b4Qj{tyW=hpP<<$@nIf2d0zI(O41__zn&$O%nI3th& zr4sXJSa~u@t^yA3rg2p=V3=42JD}r@Rr=gHnM*RjaEXN6L~1cNg|yIMAk4f-!Ci~_ zPY}wxS!oHO{Y~@cq4z;Y%2xq`=p^@n@YRp0I1(cD1*{rpDX&FqongX z#J_(uIYn#Xu#n)EEOtp84j|&5`iw6V)iHSiM_lw8F z?)1(1o}PyEiC^2iTg9R+=#J#^^=@rrxciFBMCZy+GvPqvV>b!70mB)yaRnt>U`#n? z^%DpJaOLM&9Tal(wp6cr+4CFA!dyNYW5E{eKK{DwZQS>(A~m@ud{2L`Y&c`pLH%#q zHLg9DpWA3Mscfyl7TA3UV#g1hF&nph3Dta>>+kcaoXOEoKx(kNf9e)w5LXxWQr3EN zKb!sRZ+^y3TY4>#J(T^pArroc;YwI=Q0!CJeUM1{hOkM7)xj38?5#C7{^%LkspLK)ZX3NiC8`M>!tz1(Yu2cgl+rHjUX@R!L?t<$3 z(3d*@xL4Vdor+x3dVSyMDgjoc5RI{0dS#BOzWH-DiGC~yhiFXi=>~>b7eh2Osg+sW z#$auGvOzeupIAnZ&0)PD!}siJ)#h?ykUuGheAj-#TRX53msix<;xz0?eK@z zJPNICb2Fm{T3qhly1wr{=JNyiZw}>Lk$%~EThsm~T(w$xKQ52X23lWyt6N?=!UFRB=UFso*v?3l?_)&kR} zY@1tQaN*|ha5Dtbw%L8uR`x^9E*5}32P0{NTxUyTuW!DAUz_t=1@LVq_bq}7RdA}@ z*SnaA;u_5z6fz8^nFsJL{>_{2fa($h?)2*felGK)uHlHzhU2np)#=m+C=xziHavaU zZE%1vqouwRHX@F>_g*=CvlIFRBW2n{7e&1^B<_(3XP2fNqny_lvdy1dMy+i4OYB*8 zf5uUqZ5eX$v`3oHJ^mwUpzHA7(o*&+^|3*#?)C+c3;FbMD9zSg2_i0miolPsRBkU) zq&iE06kT;tg-I_hiIRy+nba?$_fu>sdqN&@{E-T@g^tC)A2_!)!egJMk&Rvzx^)CM zKjcq#;Zu}d()JY`!>r|z;am_;*N6FZO79mI*dO?DKM-Zu&ab+Es9kOoM=@;nv|C+ElQvI10@s+8~v{WU^|ic zE=Kp7BJWUr_6pJT?OKb>os#w*TSEYR+2JL1~k0BYU{;2+p^c{?xLCziRNe-khm zL}x$P1c5}8cTZO`mX2Vh?A?0od%DB#jASn0=#S*&wJ~7~stEsB&zcQ{nJa&+&&GbMat8|AGZ9b#P@bKnIt@Qg zZ8STuqqrF|BImXzl{sjlgy{(w{j6p{oE9ky<6}C^s$Z;-)BfICR9dk^qY|Ty4t#BX z3ahCWI5~)Pw5|jUCfbxm4@(0<9uQW-3;hFAUvUL+J6q-tZOBxy86sNF79T>=?_@3p zlJo`9li+s9$oJgA74X{CFjm=KIU-Ewkdy6`;q6s1_@GtIB+f0Sj=6u=K`*GBK)G%Q zR>i`h(Nfv$JU6uAE^cqDx;t1t^vup56irU8I>4RF1N!A4ORamo^C9P1kb`4emIr|E zMGKgTFgZ;2vWAqLAYD!|F)J{@?S9t0ww0Ptib~UgYVr>C;GTRTx?VNdIeQDHEJLVs zXt^J*u3V7;1V^cz^S40(U}4R!7}82Cv~#sv2Gb4ZA1TpYT&YbVc7+Uy^&2p+L{!BO6|^aJ(q%_0nDSljYzaI!&RyqxA| ztD*b2Z+Q{n)Z1~Y@Ds3GI;+i@jr~z^u_z}8s`<1(@}!Q;kfYRPzc>QFggsGos@3+z z_(^Txba4^9_a?2b*GZ%xHs?$q3qs2?||(cA&K|C$MsF!e&<6v!}GO*>3y*n@p4v!t4Nkj_#z!{@&0`f zPhxIj>QH&CSs;PNw>MS^8IO9#0xQR(;#FQ9M~0sRx%n}HYoaqL*yBa!`x=fa5+(G^ zkSi$C;SUz|mNy=PM$Bz{67n7=o6A6Bmgqn&C{IEvyeP}2n%-gLWew9qQsa=os}ZX0 z&f)Hm&LO#QSD_~x?!ym!)GU~u0M#R)O0kSk9=~JQV13;5=n>#WP3|W6y)q`M7^4Vr z9tp2D%yjRXx%$;+!fAZO)#|Y?Dj}qfEw{S!zV*x=CxhTZ_Kz`26ZxW|!{?F3W$3+qUk6oA$!GVI-?C9H zo94>;m%s6TG5X7Qg*CUnS@|bHj=Lui3a?VND;MTLHMtnMB%v}&+GbKVJ7SllbDg!X zdY{?7o?FB{{dI~2_;lZz?Yx&uj4>i1Coi_=2J)$oO&ITwIA*gVF2QrYKltSFTAj=l zuc1x&yp8YK@?7y;U{3e%)OPAh&OesH`dZ{-ahZQ#vd`hpF|~XcPKXSUJB)(KKiHE? z^+yri2kJKxa+YW|e1jR>M(5o0JFlh2fj}eE=6zGVk$uRXycKNi$5W3+#7h0hhA3+V zK5PL5L~z|a&OaFCCnYP!iplWG`7w?u-3d>p;byr+f7Ae(napi<9=5g=*%LcqdTfsD z2DjwzcH73r(=g5SAce||`pmaJ8yFLU&;GE&-mxft#;RP^pn7|d;}Yg;dbEF@6bg1E z3g8HEtZ>wHpAA!m^rdfDx}JVvJ&wVTv_&ome6HCn9W@NTpW&b!J}N31Mxu(S`ke5n z76}*{dr_$ zhpoRTid@j@=uDoQ_1ezj_L|`$XYXbV1o7X~ef1<|zcLBeB$vzXO@>1rNmP!Bucs2B;S{! zmW`FO;&$1mcFlL2xe4}ntiG(Zv7SdcQ#w!Stp)@|UDc&QTwcT+Xm%#Pj&>^982tm+ zOi28&So&qdqDt>wA$rU2hk=3laYB;04=f0hOX~ZC~ z>@;KYDeg$8rR|IDL_0RaXxViMg9wEfH2L~0UUyxNMd~DIn+Ae1uxuh{xOc1`sNClc zGJPzjv#zE>=PGEq#kQp8N{VyCDfjR&auN#^G6LNr3ny=83=-wnS%uU@A`eebfpXz@ z5BoOiCVHU0HXV>4!1rdpOf-V8K98z-mE1P5WBd$a8S7LS>S9FtZisY#>Pb6@mxXB| zQGAFe`|Cya#QyeoeGVFM11)bj=djnJ8U+!)ZgukHmey!OXAZKOmVN&=xCIP>MPXH_ zlo6uCzujjdoQLSKg>=`O^u33>QBhGaryOkT5AGpI%ouE#2=2(7vaX4;b5v&PqqMqwlH$VJ@@H~b zg^uc(qCq*zHh!m+QdakT-{~0?=#7x3P*zh}#60nhA`h%*(8q#yB#2l&7i)wGB|M!9 z!Q`6^k)fp&-cvsz5y9qfAxuVI<5x0zJ(M0Be2KmcaoIx>X1eU zzTP#=8wXhas9mR@pzbmUhSE_HyK%`Khuz$YYcKEY&dY|8oY^|(w85|!s)&_mX_lMH zJ8%8*aj@IeUiG5r_g_5G=E~>?H{ux&XH`#(9By#gk5vL(i!UdIxL{PL4U&7WG&rx( zTJOI5`Ue!kYca<3L>fqG&7Z8fGhb3s8bnWjL7FmOd+ts?u}yqUO_DKG-pSE@Ep?Zk zfDAGj$>!+JY9@2qD3z15@>qm=kn^EjDVi>ua!io@RM*VnGdq3?^#t9HdGO}ia0Nzt z`W?yW275dQ{yCVCe+^Tu--W=(z_bd@LRcEA8bAGDDdD51vCRvSV9Pge=_5|N>0>n! zmJUX7p$)|Ty)k2<2C)YIiOD({qkd!wV`fCYg+ug=pW^d{G;+TQN1Ny`x^R18J&3Zi z+ELw9-WaW7OR00i%=B@!CCOlf2zrG>w1smYm(mu_Uha#OY@a%<;hOh3>thYxJe2$y zu+~r-VwYSFGeInTuOr!JjACX;d_U?>9@yPBP}BIPb`5BtufQ~cInbH&yID;|qrM9+mDybZ{CKxf+X))af0A}uI z--k9w&Dj|(cV6)?!>$cKZkKD6dHbgJ9D~m15!r4vG+`!qdM)Xc?XtRb`tGww-^bF2 z%5~6x#ELEDGK=iIt}oxdxzeTT)a|Gqbow0pBb6ivE##1D3KpFEVOe^o1O;y_D|XJg zYbVl78x>px!Q0m3l$JN8XFux?zsqeg-k~{;A8kVkc>p7m}`ii`En(I+IsSVAZE&8W(a=d^tH>}xi zmAh7*YVGCGz&L(+-#CeP&fO>(!I{_}1kED$O776Nardzk>k6pE{U)z3p>iz8S=ZVT zd)?1t)eu@TE|pOu;x9v8&nKqB%NrIityCm)B{U&@C{`Vw7${vzzo%~J0bx3XGobZZ z6$yT31=pOObgU$z&j~aduff7RHM)v!B+LLl+!!Gt=0JW*1?&SNt4`Vs{{;BaD zrNPZ)g<~~**6b@0&mUo@?W3A>!rGv^%sT!+-(3-nw@D9!QXoFNV~fjI_U<)ZDsf%g zIM399GG#Yc@mK>}s!FH$Nh&OJu;*C`8ZHb$*sY&DPbW`|Z*CkrK*YRrNnm4r&Ym&1 z{5iCLtuI<(06xLQBCc7UxFZZXM;(s|X1Eh++G-Bh``)xR?&s2%pvNl1Roukpm$^F& z3hQk?eqFR$zoURVpiqZN`M&B*!V2P213c&tN(A~;(y9o@AdtnMhY${+>aDDeSIuS^y3#XAem1nijY-1DonL48E7A(~RcsfKUhQt7G3W4H>O$NRSkg~VO9^FO}MJUw`PWmBCnL8R$bq;k(M zh<4ga|m2KfA((PtQeTlCgOu+#-WhztGW-JD{?w6dh~Q znDj0flo-0M>QN7giX5N;m5W!@Y#GTj>Ze>+*&mhDZ*5sZ$V-hFBj8*IR!S{)`MJQN z^pxPJq_&BPVF#ts%Y2rv=dl{gG1cBYH~n#68!hiMJO|Sr*Xs^>&;tmoH$c3PZbGE8 zWRqy!vsH?ox@j6(`_Y9BRY@0716Hm#M9hEa)gPzT1dtcl1Zmybt+$?Alc%q7^xfHM zoot)>)NHby>Y$dg-o68xK2iO%nD6mdiq65mmrBs(t-0~^rDg_i>x}v4qwO{vUH!Wp z;D+s6Jr96)&n+PY$vvS7vEoo;R2vH2d7_e^_mPM{x;{1k}g0S zy_`u$@qu?++@F#>Ik4dPv$qANdyEIuyCnmle9y#kX=ffvnQV1IiYLGK=FqEaSw5H7 z-?Qf29LG=VL-C#dc@*FcvX)hrbdMS62G#d#s=Qcy&f_@+9;}M_o9nfr3KCNs2

n$h}spUXcVh zc>T)*b1(s>$eG-Kr7?boh(z39YL=3m)cU)&LLeoS-0S$*LZNLGAXbWEw_ys z90{=M)*O6OYp_5p0NmRGG}oOA_Y-$|uTo%`Mg;UF{$n`~KqnhCkC3r1{`ch#$dXrG zHJ7kds-qx;<6j0hLOGk$V@p%|1&9wo~>?)7ltr-gk;)TE*$Z%T?4L_>dHNwxbF)d{BP61GEH;3 z$tN6S>%kfSEMu$zZA*{UZ$sC`c+T()F@vB$I7pSBDYo?iyIvzW=%d@<&)G#JM|36{qa&di6%QJ4^^oyj69UU4I z)DdVv%usDetJ|5pG+-q`qcFYHZH9OMx$pDgEZa824zQJ}fWX+^);G}-AQe%)Mk9r= z0ral<6}8(s(gmjXB64hazvPgG%xYV;}y|{7O!1M=`8EGR^q0;#D z-lCYB1i$*ak##5E(I;e7^s(KjsOi_Wpo^-Ap)K?9q>YV@!t<~?Y6Du!GPh=H4=se9 znZCJ|tv8TFE{1=J1^Oe(o0_7?Ep<8uX}n|t&WF|RlQ@1IwNA(Hjy!ecD_-ey6d8DI z?C7&Z3~m>fY`|ly+HNi{Fa6O@i_wh}4P~D}b<~9m(+g#nOX4rT0foITPLUsGD(uYA zvp@j_>ka>O0zbQjG7MeZ+Qc_jc{3k5^cv>p$y==mnw!~JOooeeH;!JtoVghW*4>;X zhuREZt@oxWNJ&W{ySUdx6Gyc?7Ex-F{n_XTmU`5~MbZp3(b*kQYlcU6B~vp+y?0d& z%Y~ld&2W5oF{0{7@ZZTRZj$jf@)9Az_4e`UfL6i^!34FN9Fc`0v5WV&?H1V{t-lok z3JkhK)u8ClAp{vsM@Z0ZuV{W!Od}(V;n3#tWMt->&Jo|7Jz>)H7}d5Pwd#|>VJe3+fTv%XnJ#*nq+u7Er`18y6hhe# zORs7kXzsTsM<3DY-mSfF^9Z@{oK*=O6O%;&ru~f^nbP{2w|T7vF??3j2UL*-nIUDL z8RSdSQV8SU+rIV8Wf&XpJNsbRYKh192-6JDsFZZF8EHbi*#&wsyUGj&yn!_@Y}UUB~3xLgm)3; zKD_f?*;Gs6v5UOnJR>$&vDXI$S=W?@j&ImuA|K)jbY~Imgs53;eS6sp#H?a{d6O2*hn+s>}riaR?M(LsqN05T) z>iv&%8FijKj5?J+m5`L{?u2p)+;4)9d}TEh>~W8R z*28O7y;zU*>B$HDa$E_;v2SUk&V4z`-QmaCBtf)Qi)z{QqHaaC?fArHj^EopT6JQ{f+lxBFe%RC&82OMD4_+3^$J*jnV_nqT?Wap=#nupsIQ@lra z2WFskl}GO*giv^FV1oTEZnOq6>P~kEsE!O9LYviChGX6A=s#g3!YikIuZT19TdPne z!}hz(Vg^ySt%KvF zLkyXptujyquFKo~1uexou2x6 zQVyycBn+lq8SPFpTw3hbeJ?9z%Ks%V*sMS}iLy&o#gl@P(jeoi7S;du0>XioP<>?A zr~)0pwhwwHz#L(suWRriDtyL%?&tmP9ZkV@a<&Z#RoH^g0u242nKan)o46T1+iS#- z-Q~D{@80GnIi3}Vk{Ekb;q4K3ZDixqWW!5)jzF62wd7kP!F)#CUZ$h%Zc+APu&CVe zC_-XCtMUK=(gHBSB~7U(F5iP!&Y_^X1V%LvOx{Ca@NvJWhtho2uDYvfe^3Up_33xh zvb%HMXHWNegsgOcPYdJos4AF=Ex{LM`%9}Uj$)NCN!KCb? zGM@V1;RF^Cf_N$_9AhUy{#XrwQZ^F?vCm=V45L}phm6Q~mXVsu=Vcbfk2{U7qxz$Y zsr6o@p@_Noq@cPFA+)a>RFTESo#gVbw0|<;!iyew*~5!MrS&ss-1tgs@+H%Vu+|=> z>qx5#vRbp2*cN_B@*gY(=7@9((M5o3c+MhQL4($y=#{oR`fUGVWf^jC-J@@7?0s_` zHKp&trW6%TE{)*fGCdEb#p%#3q&87QOi4h8F?Ev7nn`={W}J#c3U}#D39%BZ({LeW zJ-W_Qy3WaGbz?cn&x-sm+S;LZ??fvq6ta6>yHW@*#QQqA~wPf;aaf< zjshRM#yM{EilZ9_z zhDVRlFgdPH7fp7#iO%Z^tm)JI!4|HqnYEXBZUQ2;fqYGN!rr`21BtR=|m(a z&!dobqwJ%^fn4@5yf_?ER&03KyaayuZ zdvV9@Qb%`nGvTE{pt^fI9+K~Y?YfaROa?r#aJa8|X0d4yajX+yTqLr<9+o!4%u}q35}!hU|cGFxlg;VyzjGLmcNATcrC5r zQMFuG@KFjpN%}MvKJPeu*{8$XkJ{#eYED7v+RkgL22FusdAVk1#a7bbD;8N49m@Aj zDAsNrt7c{c?XT%x=I;6EVq3bdwHx{W{#P;SpLaZI8I{gfd&F zrh9NUaue~zubWOV0Tt-ui%^($+Wy;?07V#7vz$a4k)`^(%Bj3($`~q(_ysBv#&^|p@Z2v-P!(Y{3zI-Vh9{D)zxhaj}Pl%+#orY^0RbayO zOlg()wZJng$qw`YQ=Z3Tbm$^@-rJ5+D;%4+^v?mmz_Pht*=n;&%<(N+21PCMc$jy{dJcU4MQipyx@Uet%z?BudlaraP4g22 z^)xmd_2JIv0y7|S2lmE!tgWw_h|H36d1WgN$?%O8g3uW6DkoZOV0aa?r&-=D9y6*` zvk~-GsO3kc*L|?Z2gO8#u%|Wfjup18lV*s<32)tFSSlm&Jn|WrrVaILUZ_cQ{oz8! zs#OBK3Q45__HQYRy6jp3EZ&$?R%M z?6VhD5sa?!tpRM@fxm?*wiZKA9&atO610wq*e^YM9rc^ddO(?UPtFmF;fZxPj{#B6 z#tPX4rjvn2;qtfoNvUG-OtGg}?&9v6?)L5>?ipdA@jkinex++{2r_V7?}hkQgQp;E zP=gdgi~pUIc+?rFKBmc@u-fwx!tK|ORzDDnS|*JJ=k?5|W~pYC*JYR^HHe^Jmf13q zeejOX(j%oBq_kqa6rz23w2{r;EESEgO$IEC3W>ZhxpF*0gHUJR4wcR@z3I44kzG)g z^UBFDKy=dCMyxH9vz3aw{w?CyJCEkNsGM876dflc>J2n7=_f7ZLN=w_FQUQ6sD?i! zK$Vdd`!=8!Xjx;1npn_n65YDwLiEh<=>_DolMj*Kyd_ufW<1DHIQx8>%jYGLg^PeI z{aUpOyvJWe9mr{-`u)AvWswjKkpBYCK`o0{2bGsK3c4tzRrLRB?;})f7vEz)@xFEc zk=m{*m8@Y0NH$y!b-~aA{)Z4dz^hO?=dG___U;#z3R4nUqWEYO9~Gz?==)v(_Blf& z56<#vIP=-9SdJ#Z+~>>_XmN7=9}PmT9OuyvH-CGqIgYt=0kVOwdG7Asd@TIHHjC?srmjDE6wIwGe`nRE@fzZTqN)K zu!`y!s4GHM2CjhvV6PeV5nOVjjD$`XITmNaYoxKHI7SzceiS))7CrpId+fCpUN(v22vYk!iF-fY8j;q3b{Ly=cb*-X@i}oUo?+PUaWry|WMk)7PSL~nx`uB@3 z33VsJW^X@22` zkUH#Lz$U}AgDiO$&=2 zkA8M}>d1yyLPZuyTv*tGYT0}Vu}jBRI*BrnyP8HXSIkEFqs2Sqi00>8NKZGd{>8 zOKjiPLq}bVa&vPxpLKRtP}+^m{ae)Re0Dq87JY;}Ep0#>jR({XYIdW3$`>S_mWxGe zc%1x1+~Z+Pf&0BKmjLvTBq|yhO=444d_;*7FGQQ!I>V@>z$}v(`ZQY&3TuJ~Nr512 zpwO_`sLAc}2*`Dei{L1@CBF8A{+A!2NsrPsa(0m6N8?6UhM#s-vQ+=Px>^UHRlAXQ z#;8swkQX^VtdK;^y`~0+QkkPI!nU`z{#Xv`{rXdbsx}(zJ`bw_Q+}ShlJ)gN3vj}q zsi3}eN64{tqt6Q|j4PV0ay0~5tORskq&itPgn#dAIj>3@5l%_5mWjALzT2l!12Xy{ zpbuQ+_%*$a_vP7#Jz#FLyWBC)s!Lat$mDxEz~0I-LSxDHx#AWrT^0M`^FMVayZY(X zpyB6f_5P6?p4%U+ZG`Io;6UtXxQg|j3*^&0*Tju#p2Xn7+Ht6cPvWeoab$gQ@th5N zvsrGCWHl4PC0C3tnl{OTv$J49W$XyP#<;`*tYWrO$q=3(-&^F~+59=_R2=OaRYSAw zRc-A#2Ju z>IY|YgFd$iZfx|7-&9oKF4l9~sMMrDW7I`JCew(Db&p5uHDvD%2nhzj?Jp-&OY>IkxIb&vhoQreGCJ%AxDlWn=n z8P`$LZ2^a)6y*6lvfK1~WNN)u(9V5brT$XvIW?00634}FD)vK5`Qh|^Y&N@^4X&3| z8e(yiPLe-~ZuFU{Sw z(eZ&F-BBB>;^w|lCYh4GuDGk@Qg+G9+bt#=&G=O!qHfCwUNXq_g43lx3?4JEWl=`+ zjs>s7HLw(Ceqyyc_EnR@M8mG6=ROy^5H#ZB!)HVq9#+~tD|-XA)T}PrOY^H|tTm=< z23KCQPF?CzE#;M$p#k3-1~3yy5)$oj!V?CuBi<>uHEbFA*XSQryb8$q_HsmLID|)rh_eLOJ>~~0Y!qTL^Z?%oAgv$Cljd2==Lr>;}2Dpe^%4ZZltNo z1<5SIwGPfTQ(SoT=}e$(bp~U#9p2h0mnUpWWq1jH`$`MIqbDiNAel_GX;p(CT;^@? z_<=YLEv;$+3z4Z85zwW02-$tH@em>L0m{&C?t}r|?%fVhWu2%(@zWnlWByf5TAua5 z!;Vok7h^tCO~HH6HfV+U5X&@htVv1Wr5O$1?;p3v3x@YFu+Kdk=UtoSpgdf`AUC1| zQ8=kyrsH}|_0GfS)-*hm*98!dmlTS_H_k??zQb)H_>--1Dbl}N|5c`}wNlXDAvrP- zXCcdRJr62=xwbECzq{~jcjF8U?OJPk$Y*I!Me;f|J@oV6CP;v^%g|#W(v-ipxZrc+ zA9exJQBtP&uStAmgR=d_huc@nz4a|_D8sH@x-~@XB>}a@Wu!MbfAAKOd`a7i_&`zJ1)Ni@U*gDQ3C*!>{k@)XQ|mc9P&^F0$W~r*WjE`#=bKjd!RX_$uAKWp zRN!!Os2VHwKc`~6`tAHa%xmuFVx#rSXdOAV&OU(Juf<=YT1J~WUA;d3Ir63UvABvc zQ5R-Vj(7@$aR1=vc`(-b{P^ZO1eS-{BFgQA-QaMM4)xgQ5|G|mg$U%kUmMLA)bhH^E`R6IZ{GsE% zD8!_Nfx>**AN!^Gn~vfXo|--)W(dcS85|gz1~}`63lj;JNV}=>o);%*3+UyXI0I1p zUPZVVDZt%cac?Og+y0LZpvC~v!D?@NMc9HTM@|L0Q6R|iU-^Mx0Q;R`Hr!;u*IQ?Q zGZKKe+p*WZjO&9x5&IX*uABj;Svri@K9QCeYcBLRW*u4gPdS0LfxmBZ%X2_ql$8WG z_nParVHff5Iffy1blRCfDo1rza*L}UDBEvEj8%6y{=Sc z_h;t$83Cx*#f-oi@A)uEH40kb?X^J&r!!vu+%G#%r!#%kr)}~Up>G?hgq;}nQCtPc zGJr-a4I_+g;a*}5oSZDGmQ3SKNz2aeDd(%BqE~B7ea32MKmdp!lozHEnrhS~uWp4F zFn#a(LJ5s_KunE|Ac=~wS{A{=Xn-B@Ohb0Bm!gp>{7L!e9TfC*iLmRWcTsTm0 zd1r~kNnYcRx)-z5Aqk{NRvn1$tLeWVpZMGowr9Xn=T@^$eHUS{zNf| z=qS5Bsa)0!=qwxWyF?T?{{vFyVwaFq`A;+#7%F0Qkk&$k(W$tj*69(Re4bO1r`CHh z)RFE8>5#)PvASCZ(mmHp_PzfZrqZD-y;i7BO%U5GEV*hGI&(A_>udXxBarr1Us564 zxS4*WK;~)0HY(X!g}kl;u3b*mV|2S3cc>X&xh=LUg@q=JB3T>A&3Nl?761-INBOqd zEX*@Nb7&$o$+64FKC)WEHpU6a0!Z-mgtyW@m@kM+c{YG^J({Zl(CjGM zF(^KnjfEO-=9LB^PE65N5*SJOWoFFv%- zFh{;Sd<#v4Wgy!5l&I^kWO3d!d;ibWsow6?`|Qcy{Vu)>?_7ezzAU4rwA>X2ZmT8uxDoybo%@>5Y59{){>jTm_`JoWF)TDjj7@xZxR7n!)H>STotj=@#8#d9)J z6Ou~Dzpp2ziUXVi)&m07(jnhcM*jPBgjfR_^lyVY{U5PbU4`qjOP0or@zvtI*+lX7 zw;ir$96j6eosFIYthr0#47;GE^~9!QA3WC_kJoE2oS&X{to2zc{io1+Bj-SI@sDq7 zMdexR(Wedkr-$o{#bx9ct@1yJ5AAS2i)$%!2rxLJCX^LDezc2>E%`H>4j|^9a48JsV=m#i5K0Wg5#J3x5 zYxlp-UK$CDql5%|M(i*Dr5bFx8+;@i5H z&n^X@s9D{{Hy=Fi(yJB41`2(39UCOEeSoGSdZ^BSF;G+++SXFrvJQQ_l`4uk*SH>@ z;qipPhNSuvp)@!cV{mnp@HBXQK1j8S3*npTz;u3$#?@YIn>u`dhByW9RW%HJE+~qQ_qVLf_n;%<;`;dpR_l-1T3%d z(Y*cvCsx?1H?eh@GR7Eopg1N%h*yn?uiN-*@vl85-z8qJv1@&Qz4w4_m5nx;E||b> zreN63Z};(`9FDU%dBLRP1do`-UJC7I>w__5 zw}1n8=p&K1j1p}w@70Afi$yCDTp=g|kv8Es_N)wTtB$s%t;EsxYI0jGIBZK#= z4p#gCL=F|{LcpMHGO87mXkf^ii6ygU9Y zvZW>8xi!75RWPXG%5|@lDU?({mR{azFB-@2wSNJ7_*8-BD=;lM;WjTG12o=HRXX^j z%!7&OmZ!~FWP&xV8n_AUXxuCFasPG7XqwpWAnn2fGXgh$E>ty>ohj>ZFE@@2c^qei zn3FJSL6s))+LG8i%$zZg`~h;8lE6r9$R30=R1+T(6ZY#X+w;bkGL*Qc5!F!wfh{s& zaej+H!+Rait9&nTuoroEd{0M^xWad@Ttr&=(3X29F-jc>I8VF)S&xT8dAdFzB)C2LVUK&`Ga%Wre1po#t2QJOP`R?3nR-NTF9T>IKwBzCx=%}J~m@A*wdS( zslJ;c#L|5~71Y?B5n<_Ho#A^*hX6!G4!+To#C{?I`ojO$>M!2`r?6Ddn zg@3inH(jFGD$)4U@R$1_JNw1*D~s|gQtvCqUK}winA%id>_Ya&T~%bc(Ze z(weQ7n<_a74g)e(^jzZb+Kyi8XP$3wWYJeD4>`=av3n&Rv*Ycxz21)rpCo{dZp%tp z{j35Z67WSWfxS?VCm6ALkrlKnoL;n(-(*x-&#NymG-mUOcV~Tw%QC%{pb{BT>P{an z%GYsfFO5~h(MI%AteE=#pK}psl#^LCokVQz=Ra{w1qGC{GiQLL8-=%9b_h}bQOknI zyO}n{ILGJmxY7VM8wXmt?xjiH>(rtmQP9eMRA>~lq>y*59V#p9aVgPsh{Wdw;hLBf zK|;JuCuo#JzDvs4;&0*GFBmVzHTd))z}~*H9TGz=0*#XoLKQ#3AqXb=xzNyeMx_)T zNWBb*vbt~izh+S#cNSnjQHsQQHdKbp&<`gF5bkZHIywzyq(;gdzsKk_(T{e#GgcWE z#e`rn2u6Ss+di=I_Q{08Tx8|fW4?Q;Aos}S8!>C>D=RU>oVp_%_B3I`$c-u;v#x>T*1^W$2sL&4V7Ut%)?o3 z1op2Sl8(b0?8h>RR>yC$8IpKR7asCn$UNtx0{$O-DFX_~FOI2A7b-em`?b7iju4)`hMu+sA~-?$?>r|T&KI5SXxW{=a=|4kuEmftf0BU&w%nV!v-1#+ zmGdh)3HK1j%IV~ut!N8}E!ojfl_kUmCFSXF?yfS{Ni4(jCS#D>cPv54rKNQ|ZL7y@ z2R82Y%J8`LGZ!Q$fKD@@aTbUcbB>L&{P^O2D>@qEVnv{QMZQ5`Q-evCG!e1a*t9Db z829n^zLW#{1ZlZW7z+2hah{sW#YMu@{MgJKKLqt)`3I4(5q8M=RU1%R`E{W?S4(=o zRt!7b`v{uIjzZ%ut1KT0k-^Vg>z=vw1eA~Q_~ShkRlW!k@H>VdoNDj4@eMs1csA-= z2<6zgYN&c$$i{WcE!9j?EFw|md9)<jJV8c>2OIekU65?5cOc5=?IbqjT2sT?( z3a{`9%T4Ns=xi0YrOBom`Rk>a@hvISqePno=TEfeDjR#+&d}z$Si?j{L0A%ayYxHCTA9v<$GOKR ziX(Ccv(}mq45V68O%LZ}GRdrHKv;%w~IQ#237liH0B0JK<7_jtVC32ic#Mc>QC zrTxhP%jx!k1F}+n-U8&pJHgiHU76IlwQQ9|*l>4wOXI@o)}qcCsPuTyTqr1ih^^e2 z%~)~BUT4$eLlnW-(xZbmpxOK!NI|8 z^8)Ul#9sdxhxq-gKp_G-+c-#)wnE(Av>Mt>$65CD^h}cC==I2H)8Z>1_c946M?~a1 zTif%P?GQ=6;tA$?DCcWN$z7U+sRBf_mr>k=u%*g4haF%a5jVYNDW3Pv%ze*}8 z|Ec-WDbMZTj+k$D`r~lQC_}tD^O+(TW?Lmfnw^zD`#KpvRy8DxQ|c1LqSZrew0 zsy{10#Q|sk_XD`}5_SQktO%ln|J%O+$^_WGHhG$nQl~R0`hI`sBGu=Wq0yUOrMk{3 zK@b?C1~?!*StCy^@5u#N7B7{$jyAc7X^TRhb6*&x)M*;-m4Vt2#Ao%OAMz&srmKtq zzuz!|$o|K0+ea+}(Z&BFC{j|~2MAJWCV)}9RZ)~{Q#}S@vy6Xv240`dFDrlD6ey|K zsz>sO1^>ebQm%Ax{%%!XO&_GfK-@?WfZy`#WmHAju0gE%$gwQV4lvKRWc{YAXQ&G{ z5PMjXCqn-8-a^ex9y5pH6PV5If;}_5yaA_09|-Hn)ywoANF0-T*tJOvfh{C`B0ATh zW$K^m({ei`FM1W!FREFT;f}0+bni$ue^xJ>6cc3^JzXrwpeLz~Mn_+ZH!;nm<*pAi z+5F~_2!%zYw~7k=*^z=ZO&vl*0{a4vA5~9*Rl+UFx ze}DDm*@0_=Z@~#5e9D-S4?&7>MqEv8Qyneplk}nwd1?GgW?Rf~@Mt-H_ug^cD#gwM zu6l^xx6XHkV%S+veN6JmgFx8uq$%HP9|h8k#+pK1fM=8=UE6_2ZF4pHYFd!V=S zy5OL_`F%&|Sqs>Jlqxug$ZV~s|GzS` z>xpdv>(WmbrzT_E@$|cbCm+(~q$qU95g_EawE%%T5oGmW&T;lQ`$vEOqsse0A(S17 z2D*ciBAB&|8@B~g@65Nv@QSs^&@$fDwKb|jVyWZvPagX$P3NPuFeD5U0sCEvXLWTo zo^UQJy{5+4#93+A$XT~mHDyC`2s1V-6a%+}x9;cN-25v+MX_X9s<;p?GT)}Atf^Fl Hy%O|ac~xj( literal 0 HcmV?d00001 From ee90f16dcd50d0caffb05ed3877280079b21a530 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 5 Sep 2022 17:45:13 +0300 Subject: [PATCH 02/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 246 ++++++++++++++++++++--------- 1 file changed, 167 insertions(+), 79 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index fe899dae2a..b98ffca233 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -4,20 +4,25 @@ ## Table of Content -- [1. Revision](#1-revision) -- [2. Scope](#2-scope) -- [3. Definitions/Abbreviations](#3-definitionsabbreviations) -- [4. Overview](#4-overview) -- [5. Requirements](#5-requirements) -- [6. Architecture Design](#6-architecture-design) -- [7. High-Level Design](#7-high-level-design) -- [RouteOrch](#routeorch) - - [High-Level Flow Diagram](#high-level-flow-diagram) +- [1. Scope](#1-scope) +- [2. Definitions/Abbreviations](#2-definitionsabbreviations) +- [3. Overview](#3-overview) +- [4. Requirements](#4-requirements) +- [5. Architecture Design](#5-architecture-design) +- [6. High-Level Design](#6-high-level-design) +- [7. RouteOrch](#7-routeorch) + - [7.1. High-Level Flow Diagram](#71-high-level-flow-diagram) + - [7.2. Temporary route](#72-temporary-route) + - [7.3. Response Channel Performance considerations](#73-response-channel-performance-considerations) - [8. SAI API](#8-sai-api) - [9. Configuration and management](#9-configuration-and-management) - - [9.1. Manifest (if the feature is an Application Extension)](#91-manifest-if-the-feature-is-an-application-extension) - - [9.2. CLI/YANG model Enhancements](#92-cliyang-model-enhancements) - - [9.3. Config DB Enhancements](#93-config-db-enhancements) + - [9.1. Config DB Enhancements](#91-config-db-enhancements) + - [9.1.1. APP_STATE_LOGGING](#911-app_state_logging) + - [9.1.2. DEVICE_METADATA](#912-device_metadata) + - [9.2. Manifest (if the feature is an Application Extension)](#92-manifest-if-the-feature-is-an-application-extension) + - [9.3. CLI/YANG model Enhancements](#93-cliyang-model-enhancements) + - [9.3.1. APP_STATE_LOGGING](#931-app_state_logging) + - [9.3.2. DEVICE_METADATA](#932--device_metadata) - [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) - [11. Restrictions/Limitations](#11-restrictionslimitations) - [12. Testing Requirements/Design](#12-testing-requirementsdesign) @@ -25,17 +30,18 @@ - [12.2. System Test cases](#122-system-test-cases) - [13. Open/Action items - if any](#13-openaction-items---if-any) + ### 1. Revision | Revision | Date | Author | Change Description | | -------- | ----------- | ---------------- | ------------------ | | 1.0 | Sep 15 2022 | Stepan Blyshchak | Initial proposal | -### 2. Scope +### 1. Scope This document describes a feedback mechanism that allows BGP not to adveritise routes that haven't been programmed yet or failed to be programmed to ASIC. -### 3. Definitions/Abbreviations +### 2. Definitions/Abbreviations | Definitions/Abbreviation | Description | | ------------------------ | ---------------------------- | @@ -46,7 +52,7 @@ This document describes a feedback mechanism that allows BGP not to adveritise r | FPM | Forwarding Plane Manager | | SAI | Switch Abstraction Interface | -### 4. Overview +### 3. Overview The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. @@ -83,15 +89,15 @@ The problem with BGP programming occurs after the T1 switch is rebooted: When the traffic is bounced back on lossless queue, buffers on both sides are overflown, credit loop happens, with PFC storm and watchdog triggered shutting down the port. To avoid that, the route programming has to be synchronous down to the ASIC to avoid credit loops. -### 5. Requirements +### 4. Requirements This section list out all the requirements for the HLD coverage and exemptions (not supported) if any for this design. -### 6. Architecture Design +### 5. Architecture Design Described functionality does not require changes to the current SONiC architecture. This design follows existing SONiC architecture approaches and uses existing SONiC infrastrcuture. -### 7. High-Level Design +### 6. High-Level Design In order to support this feature @@ -116,7 +122,8 @@ This section covers the high level design of the feature/enhancement. This secti - Is this change specific to any platform? Are there dependencies for platforms to implement anything to make this feature work? If yes, explain in detail and inform community in advance. - SAI API requirements, CLI requirements, ConfigDB requirements. Design is covered in following sections. -### RouteOrch + +### 7. RouteOrch ```c++ auto status = ReturnCode(saiStatus) << "Failed to create route " << ipPrefix.to_string().c_str() << " with next hop(s) " << nextHops.to_string().c_str(); @@ -125,12 +132,17 @@ SWSS_LOG_ERROR("%s", status.message().c_str()); m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), status); ``` -#### High-Level Flow Diagram +#### 7.1. High-Level Flow Diagram ##### Figure 2. BGP Configuration Flow Diagram ```mermaid +%%{ + init: { + "theme": "forest" + } +}%% sequenceDiagram participant CONFIG_DB participant A as BGP /usr/bin/docker_init.sh @@ -168,6 +180,11 @@ sequenceDiagram ##### Figure 3. BGP-SWSS Flow Diagram ```mermaid +%%{ + init: { + "theme": "forest" + } +}%% sequenceDiagram participant orchagent participant APPL_DB @@ -178,38 +195,28 @@ sequenceDiagram participant Kernel participant zebra participant bgpd - Note right of bgpd: A new prefix is received - activate bgpd bgpd -->> zebra:
activate zebra - zebra -->> Kernel:
activate Kernel - Note right of Kernel: Route is programmed to kernel
without RTM_F_OFFLOAD flag - Kernel -->> zebra:
deactivate Kernel zebra -->> fpmsyncd: Update via FPM channel activate fpmsyncd - fpmsyncd -->> APPL_DB: Set ROUTE_TABLE entry activate APPL_DB APPL_DB -->> orchagent:
activate orchagent deactivate APPL_DB - deactivate fpmsyncd - deactivate zebra deactivate bgpd - loop for each route orchagent -->> orchagent: Prepare bulk create request end - orchagent -->> ASIC_DB: sai_route_api->create_route_entries activate ASIC_DB ASIC_DB -->> syncd:
@@ -222,37 +229,51 @@ sequenceDiagram orchagent -->> APPL_STATE_DB: Set ROUTE_TABLE entry status activate APPL_STATE_DB APPL_STATE_DB -->> fpmsyncd:
- activate fpmsyncd - alt Route creation successful: fpmsyncd -->> Kernel: Set RTM_F_OFFLOAD activate Kernel - Kernel -->> zebra:
- activate zebra - zebra -->> bgpd:
activate bgpd - Note right of bgpd: The prefix advertisement was suppressed
Now RTM_F_OFFLOAD appears in the flags
and it is going to be advertised to peers. - deactivate bgpd - deactivate zebra - deactivate Kernel end - deactivate fpmsyncd - APPL_STATE_DB -->> orchagent:
deactivate APPL_STATE_DB end + deactivate orchagent +``` +#### 7.2. Temporary route - deactivate orchagent +#### 7.3. Response Channel Performance considerations + +Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. + +Adding a feedback mechanism to the system introduces a delay as each of the component needs to wait for the reply from the lower layer counterpart in order to proceed. SONiC has already moved to synchronous SAI Redis pipeline a route programming performance degradation caused by it is leveled by the use of SAIRedis Bulk API. + +By introducing a response channel it is required to leverage Redis Pipeline, so that the route configuration producer using Redis Pipeline with ```ProducerStateTable``` also receives route programming status responses produced by pipelined ```NotificationProducer``` which is part of ```ResponsePublisher```. + +On the other side, ```fpmsyncd``` does not wait for each individual route status but rather performs an asynchronous processing. + +```mermaid +%%{ + init: { + "theme": "forest" + } +}%% +flowchart LR + zebra("
zebra

") -- FPM channel --> fpmsyncd("
fpmsyncd

") + fpmsyncd -- "ProducerStateTable (Pipelined)" --> RouteOrch("
RouteOrch

") + RouteOrch -. "ResponsePublisher (Pipelined)" -.-> fpmsyncd + fpmsyncd -. FPM channel -.-> zebra + RouteOrch -- "SAIRedis Bulk API" --> syncd("
syncd

") + syncd -. "SAIRedis Bulk Reply" -.-> RouteOrch ``` ### 8. SAI API @@ -260,45 +281,136 @@ sequenceDiagram No new SAI API or changes to SAI design and behaviour needed for this functionality. ### 9. Configuration and management -This section should have sub-sections for all types of configuration and management related design. Example sub-sections for "CLI" and "Config DB" are given below. Sub-sections related to data models (YANG, REST, gNMI, etc.,) should be added as required. -#### 9.1. Manifest (if the feature is an Application Extension) +#### 9.1. Config DB Enhancements + +##### 9.1.1. APP_STATE_LOGGING + +Configuration schema in ABNF format: + +```abnf +; APP_STATE_LOGGING table +key = APP_STATE_LOGGING|ROUTE_TABLE ; Configuration for ROUTE_TABLE +state = "enabled"/"disabled" ; Enable/disable response logging for ROUTE_TABLE +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "APP_STATE_LOGGING": { + "ROUTE_TABLE": { + "state": "enabled" + } + } +} +``` + +##### 9.1.2. DEVICE_METADATA + +Configuration schema in ABNF format: + +```abnf +; DEVICE_METADATA table +key = DEVICE_METADATA|localhost ; Device metadata configuration table +bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, by default this flag is disabled +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "DEVICE_METADATA": { + "localhost": { + "bgp-suppress-fib-pending": "enabled" + } + } +} +``` + +This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. + +#### 9.2. Manifest (if the feature is an Application Extension) This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. -#### 9.2. CLI/YANG model Enhancements +#### 9.3. CLI/YANG model Enhancements + +##### 9.3.1. APP_STATE_LOGGING + +A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: + +```yang +module sonic-app-state-logging { + yang-version 1.1; + + namespace "http://github.com/Azure/sonic-app-state-logging"; + prefix app-state-logging; + + description "APP_STATE_LOGGING YANG module for SONiC OS"; + + revision 2022-09-15 { + description "Initial revision"; + } + + container sonic-app-state-logging { + container APP_STATE_LOGGING { + description "Controls the enablement of a response channel per APPL_DB table"; + + container ROUTE_TABLE { + description "Configure response channel for ASIC route configuration"; + + leaf state { + description "Enablement state of response channel for the given table"; + type enumeration { + enum enabled; + enum disabled; + } + default disabled; + } + } + } + } +} +``` + +Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. + +##### 9.3.2. DEVICE_METADATA -A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which be set to ```"enable"``` or ```"disable"```. +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. Snippet of ```sonic-device_metatadata.yang```: ```yang module sonic-device_metadata { + import sonic-app-state-logging { + prefix app-state-logging; + } + revision 2022-09-15 { description "Add BGP suppress FIB pending configuration knob"; } container sonic-device_metadata { - container DEVICE_METADATA { - description "DEVICE_METADATA part of config_db.json"; container localhost{ - leaf bgp-suppress-fib-pending { description "Enable BGP suppress FIB pending feature. BGP will wait for route FIB intallation before announcing routes. This configuration requires restarting BGP sessions."; type enumeration { - enum enable; - enum disable; + enum enabled; + enum disabled; } - default disable; + default disabled; - must "((current() = 'disable') or (current() = 'enable' and../synchronous_mode = 'enable'))"; + must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' and /app-state-logging:sonic-app-state-logging/app-state-logging:APP_STATE_LOGGING/app-state-logging:ROUTE_TABLE/app-state-logging:state = 'enabled'))" { + error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must to be enabled in order to enable BGP suppress FIB pending feature"; + } } - } } } @@ -309,30 +421,6 @@ This knob can only be set to ```"enable"``` when syncrhonous SAI configuration m No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. -#### 9.3. Config DB Enhancements - -Configuration schema in ABNF format: - -```abnf -; DEVICE_METADATA table -key = DEVICE_METADATA:localhost ; Device metadata configuration table -bgp-suppress-fib-pending = "enable"/"disable" ; Globally enable/disable BGP suppress-fib-pending feature, by default this flag is disabled -``` - -Sample of CONFIG DB snippet given below: - -```json -{ - "DEVICE_METADATA": { - "localhost": { - "bgp-suppress-fib-pending": "enable" - } - } -} -``` - -This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. - ### 10. Warmboot and Fastboot Design Impact Mention whether this feature/enhancement has got any requirements/dependencies/impact w.r.t. warmboot and fastboot. Ensure that existing warmboot/fastboot feature is not affected due to this design and explain the same. @@ -350,4 +438,4 @@ Example sub-sections for unit test cases and system test cases are given below. ### 13. Open/Action items - if any -NOTE: All the sections and sub-sections given above are mandatory in the design document. Users can add additional sections/sub-sections if required. \ No newline at end of file +NOTE: All the sections and sub-sections given above are mandatory in the design document. Users can add additional sections/sub-sections if required. From 3aae79f86ec5d25dc94e87aed8801d7022a59860 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 5 Sep 2022 19:52:14 +0300 Subject: [PATCH 03/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 181 +++++++++++++++++------------ 1 file changed, 109 insertions(+), 72 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index b98ffca233..f9b42c5bfc 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -10,25 +10,28 @@ - [4. Requirements](#4-requirements) - [5. Architecture Design](#5-architecture-design) - [6. High-Level Design](#6-high-level-design) -- [7. RouteOrch](#7-routeorch) - - [7.1. High-Level Flow Diagram](#71-high-level-flow-diagram) - - [7.2. Temporary route](#72-temporary-route) - - [7.3. Response Channel Performance considerations](#73-response-channel-performance-considerations) -- [8. SAI API](#8-sai-api) -- [9. Configuration and management](#9-configuration-and-management) - - [9.1. Config DB Enhancements](#91-config-db-enhancements) - - [9.1.1. APP_STATE_LOGGING](#911-app_state_logging) - - [9.1.2. DEVICE_METADATA](#912-device_metadata) - - [9.2. Manifest (if the feature is an Application Extension)](#92-manifest-if-the-feature-is-an-application-extension) - - [9.3. CLI/YANG model Enhancements](#93-cliyang-model-enhancements) - - [9.3.1. APP_STATE_LOGGING](#931-app_state_logging) - - [9.3.2. DEVICE_METADATA](#932--device_metadata) -- [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) -- [11. Restrictions/Limitations](#11-restrictionslimitations) -- [12. Testing Requirements/Design](#12-testing-requirementsdesign) - - [12.1. Unit Test cases](#121-unit-test-cases) - - [12.2. System Test cases](#122-system-test-cases) -- [13. Open/Action items - if any](#13-openaction-items---if-any) + - [6.1. BGP Docker container startup](#61-bgp-docker-container-startup) + - [6.2. RouteOrch](#62-routeorch) + - [6.3. Temporary route](#63-temporary-route) + - [6.4. FPMsyncd](#64-fpmsyncd) + - [6.5. Response Channel Performance considerations](#65-response-channel-performance-considerations) +- [7. SAI API](#7-sai-api) +- [8. Configuration and management](#8-configuration-and-management) + - [8.1. Config DB Enhancements](#81-config-db-enhancements) + - [8.1.1. APP_STATE_LOGGING](#811-app_state_logging) + - [8.1.2. DEVICE_METADATA](#812-device_metadata) + - [8.2. Manifest (if the feature is an Application Extension)](#82-manifest-if-the-feature-is-an-application-extension) + - [8.3. CLI/YANG model Enhancements](#83-cliyang-model-enhancements) + - [8.3.1. APP_STATE_LOGGING](#831-app_state_logging) + - [8.3.2. DEVICE_METADATA](#832--device_metadata) +- [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) + - [9.1. Warm Reboot](#91-warm-reboot) + - [9.2. Fast Reboot](#92-fast-reboot) +- [10. Restrictions/Limitations](#10-restrictionslimitations) +- [11. Testing Requirements/Design](#11-testing-requirementsdesign) + - [11.1. Unit Test cases](#111-unit-test-cases) + - [11.2. System Test cases](#112-system-test-cases) +- [12. Open/Action items - if any](#12-openaction-items---if-any) ### 1. Revision @@ -91,7 +94,11 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a ### 4. Requirements -This section list out all the requirements for the HLD coverage and exemptions (not supported) if any for this design. +- ```RouteOrch``` must use the ```ResponsePublisher``` API to create a feedback channel and write each route entry programming status for the ```fpmsyncd``` to consume +- The response channel can be turned off based on user configuration in a new ```APP_STATE_LOGGING``` table in CONFIG DB. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload``` +- A configuration knob ```bgp-suppress-fib-pending``` in ```DEVICE_METADATA``` table in CONFIG DB to control the enablement of the feature is required. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload```. This knob can only be enabled if the corresponding response channel is enabled in ```APP_STATE_LOGGING``` +- ```fpmsyncd``` must consume the responses from ```RouteOrch``` when the feature is enabled and communicate the status of a route back to ```zebra``` using ```FPM``` channel +- ```FRR``` must support ```bgp suppress-fib-pending``` as well as response channel via ```FPM```. Available as part of ```FRR``` 8.4 or requires a patched 8.2 release ### 5. Architecture Design @@ -99,40 +106,7 @@ Described functionality does not require changes to the current SONiC architectu ### 6. High-Level Design -In order to support this feature - -This section covers the high level design of the feature/enhancement. This section covers the following points in detail. - - - Is it a built-in SONiC feature or a SONiC Application Extension? - - What are the modules and sub-modules that are modified for this design? - - What are the repositories that would be changed? - - Module/sub-module interfaces and dependencies. - - SWSS and Syncd changes in detail - - DB and Schema changes (APP_DB, ASIC_DB, COUNTERS_DB, LOGLEVEL_DB, CONFIG_DB, STATE_DB) - - Sequence diagram if required. - - Linux dependencies and interface - - Warm reboot requirements/dependencies - - Fastboot requirements/dependencies - - Scalability and performance requirements/impact - - Memory requirements - - Docker dependency - - Build dependency if any - - Management interfaces - SNMP, CLI, RestAPI, etc., - - Serviceability and Debug (logging, counters, trace etc) related design - - Is this change specific to any platform? Are there dependencies for platforms to implement anything to make this feature work? If yes, explain in detail and inform community in advance. - - SAI API requirements, CLI requirements, ConfigDB requirements. Design is covered in following sections. - - -### 7. RouteOrch - -```c++ -auto status = ReturnCode(saiStatus) << "Failed to create route " << ipPrefix.to_string().c_str() << " with next hop(s) " << nextHops.to_string().c_str(); -SWSS_LOG_ERROR("%s", status.message().c_str()); - -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), status); -``` - -#### 7.1. High-Level Flow Diagram +#### 6.1. BGP Docker container startup ##### Figure 2. BGP Configuration Flow Diagram @@ -176,6 +150,9 @@ sequenceDiagram deactivate A ``` + +#### 6.2. RouteOrch + ##### Figure 3. BGP-SWSS Flow Diagram @@ -249,9 +226,11 @@ sequenceDiagram deactivate orchagent ``` -#### 7.2. Temporary route +#### 6.3. Temporary route -#### 7.3. Response Channel Performance considerations +#### 6.4. FPMsyncd + +#### 6.5. Response Channel Performance considerations Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. @@ -276,15 +255,61 @@ flowchart LR syncd -. "SAIRedis Bulk Reply" -.-> RouteOrch ``` -### 8. SAI API +A snippet of ```ResponsePublisher```'s API is going to be used: + +```c++ +// Intent attributes are the attributes sent in the notification into the +// redis channel. +// State attributes are the list of attributes that need to be written in +// the DB namespace. These might be different from intent attributes. For +// example: +// 1) If only a subset of the intent attributes were successfully applied, the +// state attributes shall be different from intent attributes. +// 2) If additional state changes occur due to the intent attributes, more +// attributes need to be added in the state DB namespace. +// 3) Invalid attributes are excluded from the state attributes. +// State attributes will be written into the DB even if the status code +// consists of an error. +void ResponsePublisher::publish(const std::string &table, const std::string &key, + const std::vector &intent_attrs, + const ReturnCode &status, + const std::vector &state_attrs, + bool replace = false) override; + +void ResponsePublisher::publish(const std::string &table, const std::string &key, + const std::vector &intent_attrs, + const ReturnCode &status, + bool replace = false) override; +``` + +Example usage in ```RouteOrch```: + +```c++ +auto status = ReturnCode(saiStatus) << "Failed to create route " + << ipPrefix.to_string().c_str() + << " with next hop(s) " + << nextHops.to_string().c_str(); + +SWSS_LOG_ERROR("%s", status.message().c_str()); + +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), status); +``` + +A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipeline``` and a flag ```buffered``` to make it use the pipelining. The constructor is similar to one in use with ```ProducerStateTable```: + +```c++ +ResponsePublisher::ResponsePublisher(RedisPipeline *pipeline, bool buffered = false); +``` + +### 7. SAI API No new SAI API or changes to SAI design and behaviour needed for this functionality. -### 9. Configuration and management +### 8. Configuration and management -#### 9.1. Config DB Enhancements +#### 8.1. Config DB Enhancements -##### 9.1.1. APP_STATE_LOGGING +##### 8.1.1. APP_STATE_LOGGING Configuration schema in ABNF format: @@ -306,7 +331,7 @@ Sample of CONFIG DB snippet given below: } ``` -##### 9.1.2. DEVICE_METADATA +##### 8.1.2. DEVICE_METADATA Configuration schema in ABNF format: @@ -330,13 +355,13 @@ Sample of CONFIG DB snippet given below: This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. -#### 9.2. Manifest (if the feature is an Application Extension) +#### 8.2. Manifest (if the feature is an Application Extension) This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. -#### 9.3. CLI/YANG model Enhancements +#### 8.3. CLI/YANG model Enhancements -##### 9.3.1. APP_STATE_LOGGING +##### 8.3.1. APP_STATE_LOGGING A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: @@ -376,7 +401,7 @@ module sonic-app-state-logging { Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. -##### 9.3.2. DEVICE_METADATA +##### 8.3.2. DEVICE_METADATA A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. @@ -421,21 +446,33 @@ This knob can only be set to ```"enable"``` when syncrhonous SAI configuration m No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. -### 10. Warmboot and Fastboot Design Impact -Mention whether this feature/enhancement has got any requirements/dependencies/impact w.r.t. warmboot and fastboot. Ensure that existing warmboot/fastboot feature is not affected due to this design and explain the same. +### 9. Warmboot and Fastboot Design Impact + +#### 9.1. Warm Reboot + +Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. + +A warm reboot regression test suite needs to be ran and verified no degradation introduced by the feature. + +#### 9.2. Fast Reboot + +Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. + +A fast reboot regression test suite needs to be ran and verified no degradation introduced by the feature. + -### 11. Restrictions/Limitations +### 10. Restrictions/Limitations -### 12. Testing Requirements/Design +### 11. Testing Requirements/Design Explain what kind of unit testing, system testing, regression testing, warmboot/fastboot testing, etc., Ensure that the existing warmboot/fastboot requirements are met. For example, if the current warmboot feature expects maximum of 1 second or zero second data disruption, the same should be met even after the new feature/enhancement is implemented. Explain the same here. Example sub-sections for unit test cases and system test cases are given below. -#### 12.1. Unit Test cases +#### 11.1. Unit Test cases -#### 12.2. System Test cases +#### 11.2. System Test cases -### 13. Open/Action items - if any +### 12. Open/Action items - if any NOTE: All the sections and sub-sections given above are mandatory in the design document. Users can add additional sections/sub-sections if required. From 3ca8282dfb64d9211d170553b95a23635118ae75 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 6 Sep 2022 18:47:50 +0300 Subject: [PATCH 04/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 262 ++++++++++++++++++++--------- 1 file changed, 184 insertions(+), 78 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index f9b42c5bfc..6b120b2d5e 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -12,9 +12,8 @@ - [6. High-Level Design](#6-high-level-design) - [6.1. BGP Docker container startup](#61-bgp-docker-container-startup) - [6.2. RouteOrch](#62-routeorch) - - [6.3. Temporary route](#63-temporary-route) - - [6.4. FPMsyncd](#64-fpmsyncd) - - [6.5. Response Channel Performance considerations](#65-response-channel-performance-considerations) + - [6.3. FPMsyncd](#63-fpmsyncd) + - [6.4. Response Channel Performance considerations](#64-response-channel-performance-considerations) - [7. SAI API](#7-sai-api) - [8. Configuration and management](#8-configuration-and-management) - [8.1. Config DB Enhancements](#81-config-db-enhancements) @@ -30,7 +29,8 @@ - [10. Restrictions/Limitations](#10-restrictionslimitations) - [11. Testing Requirements/Design](#11-testing-requirementsdesign) - [11.1. Unit Test cases](#111-unit-test-cases) - - [11.2. System Test cases](#112-system-test-cases) + - [11.2. VS Test cases](#112-vs-test-cases) + - [11.3. System Test cases](#113-system-test-cases) - [12. Open/Action items - if any](#12-openaction-items---if-any) @@ -54,6 +54,7 @@ This document describes a feedback mechanism that allows BGP not to adveritise r | SYNCD | ASIC syncrhonization service | | FPM | Forwarding Plane Manager | | SAI | Switch Abstraction Interface | +| OID | Object Identifier | ### 3. Overview @@ -99,6 +100,8 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a - A configuration knob ```bgp-suppress-fib-pending``` in ```DEVICE_METADATA``` table in CONFIG DB to control the enablement of the feature is required. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload```. This knob can only be enabled if the corresponding response channel is enabled in ```APP_STATE_LOGGING``` - ```fpmsyncd``` must consume the responses from ```RouteOrch``` when the feature is enabled and communicate the status of a route back to ```zebra``` using ```FPM``` channel - ```FRR``` must support ```bgp suppress-fib-pending``` as well as response channel via ```FPM```. Available as part of ```FRR``` 8.4 or requires a patched 8.2 release +- Link-local2ME, IP2Me routes responses aren't published +- MPLS, VNET routes are out of scope of this document ### 5. Architecture Design @@ -108,13 +111,19 @@ Described functionality does not require changes to the current SONiC architectu #### 6.1. BGP Docker container startup +BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. The startup flow of BGP container is described below: + ##### Figure 2. BGP Configuration Flow Diagram ```mermaid %%{ init: { - "theme": "forest" + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } } }%% sequenceDiagram @@ -153,84 +162,177 @@ sequenceDiagram #### 6.2. RouteOrch +```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. + +*A note on a temporary route*: + +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. + -##### Figure 3. BGP-SWSS Flow Diagram +##### Figure 3. RouteOrch Route Set Flow ```mermaid %%{ init: { - "theme": "forest" + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } } }%% sequenceDiagram - participant orchagent - participant APPL_DB - participant ASIC_DB - participant syncd - participant APPL_STATE_DB - participant fpmsyncd - participant Kernel - participant zebra - participant bgpd - Note right of bgpd: A new prefix is received - activate bgpd - bgpd -->> zebra:
- activate zebra - zebra -->> Kernel:
- activate Kernel - Note right of Kernel: Route is programmed to kernel
without RTM_F_OFFLOAD flag - Kernel -->> zebra:
- deactivate Kernel - zebra -->> fpmsyncd: Update via FPM channel - activate fpmsyncd - fpmsyncd -->> APPL_DB: Set ROUTE_TABLE entry - activate APPL_DB - APPL_DB -->> orchagent:
- activate orchagent - deactivate APPL_DB - deactivate fpmsyncd - deactivate zebra - deactivate bgpd - loop for each route - orchagent -->> orchagent: Prepare bulk create request + participant APPL_DB + participant RouteOrch + participant RouteBulker + participant ResponsePublisher + participant APPL_STATE_DB + participant ASIC_DB + participant SYNCD + + APPL_DB --) RouteOrch: ROUTE_TABLE Notification + activate RouteOrch + + RouteOrch ->> RouteOrch: doTask() + + loop For each entry in m_toSync + alt NHG exist or can be added + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + else NH can't be added + Note right of RouteOrch: 1 NH is selected and addRoute()
is called with temporary NH + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker end - orchagent -->> ASIC_DB: sai_route_api->create_route_entries - activate ASIC_DB - ASIC_DB -->> syncd:
- activate syncd - syncd -->> ASIC_DB: SAI bulk status - deactivate syncd - ASIC_DB -->> orchagent: SAI bulk status - deactivate ASIC_DB - loop for each route entry creation SAI status: - orchagent -->> APPL_STATE_DB: Set ROUTE_TABLE entry status + end + + Note left of RouteOrch: On any error/pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
+ + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + + RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + + loop For each status in RouteBulker + alt SAI_STATUS_SUCCESS + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB:
activate APPL_STATE_DB - APPL_STATE_DB -->> fpmsyncd:
- activate fpmsyncd - alt Route creation successful: - fpmsyncd -->> Kernel: Set RTM_F_OFFLOAD - activate Kernel - Kernel -->> zebra:
- activate zebra - zebra -->> bgpd:
- activate bgpd - Note right of bgpd: The prefix advertisement was suppressed
Now RTM_F_OFFLOAD appears in the flags
and it is going to be advertised to peers. - deactivate bgpd - deactivate zebra - deactivate Kernel - end - deactivate fpmsyncd - APPL_STATE_DB -->> orchagent:
+ APPL_STATE_DB -->> ResponsePublisher:
deactivate APPL_STATE_DB + Note right of RouteOrch: Publish actual FVs
(in case of temporary route only 1 NH is added) + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher + else + RouteOrch ->> RouteOrch: abort end - deactivate orchagent + end + + deactivate RouteOrch ``` -#### 6.3. Temporary route + +##### Figure 4. RouteOrch Route Delete Flow -#### 6.4. FPMsyncd +Similar processing happens on ```DEL``` operation for a prefix from ```ROUTE_TABLE```. The removed prefixes are filled in ```RouteBulker``` which are then flushed forming a bulk remove API call to SAIRedis. ```RouteOrch::removeRoutePost()``` then collects the object statuses and if the status is not SAI_STATUS_SUCCESS orchagent ```abort()```s, otherwise it should publish the result via ```ResponsePublisher::publish()``` leaving ```intent_attrs``` vector empty which will remove the corresponding state key from ```APPL_STATE_DB```. -#### 6.5. Response Channel Performance considerations +```mermaid +%%{ + init: { + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } + } +}%% +sequenceDiagram + participant APPL_DB + participant RouteOrch + participant RouteBulker + participant ResponsePublisher + participant APPL_STATE_DB + participant ASIC_DB + participant SYNCD + + APPL_DB --) RouteOrch: ROUTE_TABLE Notification + activate RouteOrch + + RouteOrch ->> RouteOrch: doTask() + + loop For each entry in m_toSync + RouteOrch ->> RouteBulker: RouteBulker.remove_entry() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + end + + Note left of RouteOrch: On any error/pre-condition failed
RouteOrch is retrying to remove
the route on next iteration
In this case no response is written
+ + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->remove_route_entries() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->remove_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + + loop For each status in RouteBulker + alt SAI_STATUS_SUCCESS + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB:
+ activate APPL_STATE_DB + APPL_STATE_DB -->> ResponsePublisher:
+ deactivate APPL_STATE_DB + Note right of RouteOrch: Publish the status and remove entry from APPL_STATE_DB + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher + else + RouteOrch ->> RouteOrch: abort + end + end + + deactivate RouteOrch +``` + +#### 6.3. FPMsyncd + + +##### Figure 5. FPMsyncd response processing + +TODO + +#### 6.4. Response Channel Performance considerations Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. @@ -452,27 +554,31 @@ No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be imp Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. -A warm reboot regression test suite needs to be ran and verified no degradation introduced by the feature. #### 9.2. Fast Reboot -Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. - -A fast reboot regression test suite needs to be ran and verified no degradation introduced by the feature. - +TODO ### 10. Restrictions/Limitations ### 11. Testing Requirements/Design -Explain what kind of unit testing, system testing, regression testing, warmboot/fastboot testing, etc., -Ensure that the existing warmboot/fastboot requirements are met. For example, if the current warmboot feature expects maximum of 1 second or zero second data disruption, the same should be met even after the new feature/enhancement is implemented. Explain the same here. -Example sub-sections for unit test cases and system test cases are given below. + +Regression test, VS test and unit testing are required for this functionality. + +Fast/warm reboot regression test suite needs to be ran and verified no degradation introduced by the feature. #### 11.1. Unit Test cases -#### 11.2. System Test cases +TODO -### 12. Open/Action items - if any +#### 11.2. VS Test cases +TODO + +#### 11.3. System Test cases + +TODO + +### 12. Open/Action items - if any -NOTE: All the sections and sub-sections given above are mandatory in the design document. Users can add additional sections/sub-sections if required. +TODO From 5707bd3a18de065eca838a378ed40f592c625654 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 6 Sep 2022 19:00:25 +0300 Subject: [PATCH 05/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 68 ++++++++++++------------------ 1 file changed, 28 insertions(+), 40 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 6b120b2d5e..8beff19bd7 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -168,6 +168,25 @@ sequenceDiagram A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. + +##### ResponsePublisher + +A snippet of ```ResponsePublisher```'s API that is going to be used is given below. The ```state_attrs``` argument is used when a ```SET``` operation for a route entry in ```ROUTE_TABLE``` was performed successfully but the actual set of next hops differs from the intent. When the ```intent_attrs``` vector is empty it implies a ```DEL``` operations. + +```c++ +void ResponsePublisher::publish(const std::string &table, const std::string &key, + const std::vector &intent_attrs, + const ReturnCode &status, + const std::vector &state_attrs, + bool replace = false) override; +``` + +Example usage in ```RouteOrch```: + +```c++ +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); +``` + ##### Figure 3. RouteOrch Route Set Flow @@ -336,12 +355,21 @@ TODO Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. +The following table shows the results for publishing 10k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: + +| Scenario | Time (sec) | +| -------------------------------- | ---------- | +| Publish 10k messages | 2.41 | +| Publish 10k messages (Pipelined) | 0.43 | + Adding a feedback mechanism to the system introduces a delay as each of the component needs to wait for the reply from the lower layer counterpart in order to proceed. SONiC has already moved to synchronous SAI Redis pipeline a route programming performance degradation caused by it is leveled by the use of SAIRedis Bulk API. By introducing a response channel it is required to leverage Redis Pipeline, so that the route configuration producer using Redis Pipeline with ```ProducerStateTable``` also receives route programming status responses produced by pipelined ```NotificationProducer``` which is part of ```ResponsePublisher```. On the other side, ```fpmsyncd``` does not wait for each individual route status but rather performs an asynchronous processing. +The following schema represents communication methods between components: + ```mermaid %%{ init: { @@ -357,46 +385,6 @@ flowchart LR syncd -. "SAIRedis Bulk Reply" -.-> RouteOrch ``` -A snippet of ```ResponsePublisher```'s API is going to be used: - -```c++ -// Intent attributes are the attributes sent in the notification into the -// redis channel. -// State attributes are the list of attributes that need to be written in -// the DB namespace. These might be different from intent attributes. For -// example: -// 1) If only a subset of the intent attributes were successfully applied, the -// state attributes shall be different from intent attributes. -// 2) If additional state changes occur due to the intent attributes, more -// attributes need to be added in the state DB namespace. -// 3) Invalid attributes are excluded from the state attributes. -// State attributes will be written into the DB even if the status code -// consists of an error. -void ResponsePublisher::publish(const std::string &table, const std::string &key, - const std::vector &intent_attrs, - const ReturnCode &status, - const std::vector &state_attrs, - bool replace = false) override; - -void ResponsePublisher::publish(const std::string &table, const std::string &key, - const std::vector &intent_attrs, - const ReturnCode &status, - bool replace = false) override; -``` - -Example usage in ```RouteOrch```: - -```c++ -auto status = ReturnCode(saiStatus) << "Failed to create route " - << ipPrefix.to_string().c_str() - << " with next hop(s) " - << nextHops.to_string().c_str(); - -SWSS_LOG_ERROR("%s", status.message().c_str()); - -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), status); -``` - A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipeline``` and a flag ```buffered``` to make it use the pipelining. The constructor is similar to one in use with ```ProducerStateTable```: ```c++ From ff9a9f4115776c43a8f8a2637f2f0b5bd04f1094 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 6 Sep 2022 19:04:36 +0300 Subject: [PATCH 06/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 8beff19bd7..37738d1c54 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -162,14 +162,10 @@ sequenceDiagram #### 6.2. RouteOrch -```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. - -*A note on a temporary route*: - -A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. - -##### ResponsePublisher +#### ResponsePublisher in RouteOrch + +```RouteOrch``` need to use existing ```ResponsePublisher``` API to be able to publish responses into ```APPL_STATE_DB```. A snippet of ```ResponsePublisher```'s API that is going to be used is given below. The ```state_attrs``` argument is used when a ```SET``` operation for a route entry in ```ROUTE_TABLE``` was performed successfully but the actual set of next hops differs from the intent. When the ```intent_attrs``` vector is empty it implies a ```DEL``` operations. @@ -187,6 +183,16 @@ Example usage in ```RouteOrch```: m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); ``` + +#### RouteOrch Route Set Flow + +```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. + +*A note on a temporary route*: + +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. + + ##### Figure 3. RouteOrch Route Set Flow @@ -273,6 +279,9 @@ sequenceDiagram deactivate RouteOrch ``` + +#### RouteOrch Route delete Flow + ##### Figure 4. RouteOrch Route Delete Flow From bee0a9e2f8f8c5acb7809a502c0b2d5e7f536ba4 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 6 Sep 2022 19:15:01 +0300 Subject: [PATCH 07/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 37738d1c54..a2fd7152d9 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -159,6 +159,8 @@ sequenceDiagram deactivate A ``` +This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. + #### 6.2. RouteOrch From 576335c4d9d9592443b7d951722b9f1f27f9f295 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 6 Sep 2022 19:25:24 +0300 Subject: [PATCH 08/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index a2fd7152d9..20cf1edd70 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -439,7 +439,8 @@ Configuration schema in ABNF format: ```abnf ; DEVICE_METADATA table key = DEVICE_METADATA|localhost ; Device metadata configuration table -bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, by default this flag is disabled +bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, + ; by default this flag is disabled ``` Sample of CONFIG DB snippet given below: @@ -533,8 +534,12 @@ module sonic-device_metadata { } default disabled; - must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' and /app-state-logging:sonic-app-state-logging/app-state-logging:APP_STATE_LOGGING/app-state-logging:ROUTE_TABLE/app-state-logging:state = 'enabled'))" { - error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must to be enabled in order to enable BGP suppress FIB pending feature"; + must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' + and /app-state-logging:sonic-app-state-logging/app-state-logging: + APP_STATE_LOGGING/app-state-logging: + ROUTE_TABLE/app-state-logging:state = 'enabled'))" { + error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must + be enabled in order to enable BGP suppress FIB pending feature"; } } } From 1b1d8c1627697e2673bf05140b983b2ed0e5a31f Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 8 Sep 2022 16:46:01 +0300 Subject: [PATCH 09/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 20cf1edd70..45e00f7dd7 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -145,7 +145,7 @@ sequenceDiagram CONFIG_DB -->> T: DEVICE_METADATA|localhost - alt "bgp-suppress-fib-pending" == "enable" + alt "bgp-suppress-fib-pending" == "enabled" T --> E: configure "bgp suppress-fib-pending" for router end @@ -159,7 +159,14 @@ sequenceDiagram deactivate A ``` -This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. +This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. Configuration template snippet *bgpd.main.conf.j2*: + +```jinja2 +router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} +{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} + bgp suppress-fib-pending +{% endif %} +``` #### 6.2. RouteOrch From 52d683ab554e7307cddea44c3cd2839b1b67786c Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 8 Sep 2022 16:47:25 +0300 Subject: [PATCH 10/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 332 ++++++++++++++--------------- 1 file changed, 166 insertions(+), 166 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 45e00f7dd7..4c7bf68911 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -9,20 +9,20 @@ - [3. Overview](#3-overview) - [4. Requirements](#4-requirements) - [5. Architecture Design](#5-architecture-design) -- [6. High-Level Design](#6-high-level-design) - - [6.1. BGP Docker container startup](#61-bgp-docker-container-startup) - - [6.2. RouteOrch](#62-routeorch) - - [6.3. FPMsyncd](#63-fpmsyncd) - - [6.4. Response Channel Performance considerations](#64-response-channel-performance-considerations) -- [7. SAI API](#7-sai-api) -- [8. Configuration and management](#8-configuration-and-management) - - [8.1. Config DB Enhancements](#81-config-db-enhancements) - - [8.1.1. APP_STATE_LOGGING](#811-app_state_logging) - - [8.1.2. DEVICE_METADATA](#812-device_metadata) - - [8.2. Manifest (if the feature is an Application Extension)](#82-manifest-if-the-feature-is-an-application-extension) - - [8.3. CLI/YANG model Enhancements](#83-cliyang-model-enhancements) - - [8.3.1. APP_STATE_LOGGING](#831-app_state_logging) - - [8.3.2. DEVICE_METADATA](#832--device_metadata) +- [6. Configuration and management](#6-configuration-and-management) + - [6.1. Config DB Enhancements](#61-config-db-enhancements) + - [6.1.1. APP_STATE_LOGGING](#611-app_state_logging) + - [6.1.2. DEVICE_METADATA](#612-device_metadata) + - [6.2. Manifest (if the feature is an Application Extension)](#62-manifest-if-the-feature-is-an-application-extension) + - [6.3. CLI/YANG model Enhancements](#63-cliyang-model-enhancements) + - [6.3.1. APP_STATE_LOGGING](#631-app_state_logging) + - [6.3.2. DEVICE_METADATA](#632--device_metadata) +- [7. High-Level Design](#7-high-level-design) + - [7.1. BGP Docker container startup](#71-bgp-docker-container-startup) + - [7.2. RouteOrch](#72-routeorch) + - [7.3. FPMsyncd](#73-fpmsyncd) + - [7.4. Response Channel Performance considerations](#74-response-channel-performance-considerations) +- [8. SAI API](#8-sai-api) - [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - [9.1. Warm Reboot](#91-warm-reboot) - [9.2. Fast Reboot](#92-fast-reboot) @@ -107,9 +107,155 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a Described functionality does not require changes to the current SONiC architecture. This design follows existing SONiC architecture approaches and uses existing SONiC infrastrcuture. -### 6. High-Level Design +### 6. Configuration and management -#### 6.1. BGP Docker container startup +#### 6.1. Config DB Enhancements + +##### 6.1.1. APP_STATE_LOGGING + +Configuration schema in ABNF format: + +```abnf +; APP_STATE_LOGGING table +key = APP_STATE_LOGGING|ROUTE_TABLE ; Configuration for ROUTE_TABLE +state = "enabled"/"disabled" ; Enable/disable response logging for ROUTE_TABLE +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "APP_STATE_LOGGING": { + "ROUTE_TABLE": { + "state": "enabled" + } + } +} +``` + +##### 6.1.2. DEVICE_METADATA + +Configuration schema in ABNF format: + +```abnf +; DEVICE_METADATA table +key = DEVICE_METADATA|localhost ; Device metadata configuration table +bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, + ; by default this flag is disabled +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "DEVICE_METADATA": { + "localhost": { + "bgp-suppress-fib-pending": "enabled" + } + } +} +``` + +This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. + +#### 6.2. Manifest (if the feature is an Application Extension) + +This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. + +#### 6.3. CLI/YANG model Enhancements + +##### 6.3.1. APP_STATE_LOGGING + +A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: + +```yang +module sonic-app-state-logging { + yang-version 1.1; + + namespace "http://github.com/Azure/sonic-app-state-logging"; + prefix app-state-logging; + + description "APP_STATE_LOGGING YANG module for SONiC OS"; + + revision 2022-09-15 { + description "Initial revision"; + } + + container sonic-app-state-logging { + container APP_STATE_LOGGING { + description "Controls the enablement of a response channel per APPL_DB table"; + + container ROUTE_TABLE { + description "Configure response channel for ASIC route configuration"; + + leaf state { + description "Enablement state of response channel for the given table"; + type enumeration { + enum enabled; + enum disabled; + } + default disabled; + } + } + } + } +} +``` + +Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. + +##### 6.3.2. DEVICE_METADATA + +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. + +Snippet of ```sonic-device_metatadata.yang```: + +```yang +module sonic-device_metadata { + import sonic-app-state-logging { + prefix app-state-logging; + } + + revision 2022-09-15 { + description "Add BGP suppress FIB pending configuration knob"; + } + + container sonic-device_metadata { + container DEVICE_METADATA { + description "DEVICE_METADATA part of config_db.json"; + + container localhost{ + leaf bgp-suppress-fib-pending { + description "Enable BGP suppress FIB pending feature. BGP will wait for route + FIB intallation before announcing routes. This configuration requires + restarting BGP sessions."; + type enumeration { + enum enabled; + enum disabled; + } + default disabled; + + must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' + and /app-state-logging:sonic-app-state-logging/app-state-logging: + APP_STATE_LOGGING/app-state-logging: + ROUTE_TABLE/app-state-logging:state = 'enabled'))" { + error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must + be enabled in order to enable BGP suppress FIB pending feature"; + } + } + } + } + } +} +``` + +This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. + +No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. + +### 7. High-Level Design + +#### 7.1. BGP Docker container startup BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. The startup flow of BGP container is described below: @@ -169,7 +315,7 @@ router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} ``` -#### 6.2. RouteOrch +#### 7.2. RouteOrch #### ResponsePublisher in RouteOrch @@ -362,14 +508,14 @@ sequenceDiagram deactivate RouteOrch ``` -#### 6.3. FPMsyncd +#### 7.3. FPMsyncd ##### Figure 5. FPMsyncd response processing TODO -#### 6.4. Response Channel Performance considerations +#### 7.4. Response Channel Performance considerations Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. @@ -409,156 +555,10 @@ A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipelin ResponsePublisher::ResponsePublisher(RedisPipeline *pipeline, bool buffered = false); ``` -### 7. SAI API +### 8. SAI API No new SAI API or changes to SAI design and behaviour needed for this functionality. -### 8. Configuration and management - -#### 8.1. Config DB Enhancements - -##### 8.1.1. APP_STATE_LOGGING - -Configuration schema in ABNF format: - -```abnf -; APP_STATE_LOGGING table -key = APP_STATE_LOGGING|ROUTE_TABLE ; Configuration for ROUTE_TABLE -state = "enabled"/"disabled" ; Enable/disable response logging for ROUTE_TABLE -``` - -Sample of CONFIG DB snippet given below: - -```json -{ - "APP_STATE_LOGGING": { - "ROUTE_TABLE": { - "state": "enabled" - } - } -} -``` - -##### 8.1.2. DEVICE_METADATA - -Configuration schema in ABNF format: - -```abnf -; DEVICE_METADATA table -key = DEVICE_METADATA|localhost ; Device metadata configuration table -bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, - ; by default this flag is disabled -``` - -Sample of CONFIG DB snippet given below: - -```json -{ - "DEVICE_METADATA": { - "localhost": { - "bgp-suppress-fib-pending": "enabled" - } - } -} -``` - -This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. - -#### 8.2. Manifest (if the feature is an Application Extension) - -This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. - -#### 8.3. CLI/YANG model Enhancements - -##### 8.3.1. APP_STATE_LOGGING - -A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: - -```yang -module sonic-app-state-logging { - yang-version 1.1; - - namespace "http://github.com/Azure/sonic-app-state-logging"; - prefix app-state-logging; - - description "APP_STATE_LOGGING YANG module for SONiC OS"; - - revision 2022-09-15 { - description "Initial revision"; - } - - container sonic-app-state-logging { - container APP_STATE_LOGGING { - description "Controls the enablement of a response channel per APPL_DB table"; - - container ROUTE_TABLE { - description "Configure response channel for ASIC route configuration"; - - leaf state { - description "Enablement state of response channel for the given table"; - type enumeration { - enum enabled; - enum disabled; - } - default disabled; - } - } - } - } -} -``` - -Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. - -##### 8.3.2. DEVICE_METADATA - -A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. - -Snippet of ```sonic-device_metatadata.yang```: - -```yang -module sonic-device_metadata { - import sonic-app-state-logging { - prefix app-state-logging; - } - - revision 2022-09-15 { - description "Add BGP suppress FIB pending configuration knob"; - } - - container sonic-device_metadata { - container DEVICE_METADATA { - description "DEVICE_METADATA part of config_db.json"; - - container localhost{ - leaf bgp-suppress-fib-pending { - description "Enable BGP suppress FIB pending feature. BGP will wait for route - FIB intallation before announcing routes. This configuration requires - restarting BGP sessions."; - type enumeration { - enum enabled; - enum disabled; - } - default disabled; - - must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' - and /app-state-logging:sonic-app-state-logging/app-state-logging: - APP_STATE_LOGGING/app-state-logging: - ROUTE_TABLE/app-state-logging:state = 'enabled'))" { - error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must - be enabled in order to enable BGP suppress FIB pending feature"; - } - } - } - } - } -} -``` - -This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. - -No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. - ### 9. Warmboot and Fastboot Design Impact #### 9.1. Warm Reboot From 7591fb2fb063b48022d6c2293efb0ddb51a730d3 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 12 Sep 2022 18:06:29 +0300 Subject: [PATCH 11/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 64 +++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 14 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 4c7bf68911..0665487705 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -22,6 +22,7 @@ - [7.2. RouteOrch](#72-routeorch) - [7.3. FPMsyncd](#73-fpmsyncd) - [7.4. Response Channel Performance considerations](#74-response-channel-performance-considerations) + - [7.4.1. Table 1. Publishing 1k ROUTE_TABLE responses](#741-table-1-publishing-1k-route_table-responses) - [8. SAI API](#8-sai-api) - [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - [9.1. Warm Reboot](#91-warm-reboot) @@ -74,13 +75,13 @@ The following conditions apply will apply when checking for route installation s [FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst) -Consider the following scenario: +Considering the following scenario: ##### Figure 1. Use case scenario

-Figure 1. Use case scenario +Figure 1. Use case scenario

The problem with BGP programming occurs after the T1 switch is rebooted: @@ -105,7 +106,14 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a ### 5. Architecture Design -Described functionality does not require changes to the current SONiC architecture. This design follows existing SONiC architecture approaches and uses existing SONiC infrastrcuture. +TBD + + +##### Figure 2. Use case scenario + +

+Figure 2. Architecture diagram +

### 6. Configuration and management @@ -197,9 +205,13 @@ module sonic-app-state-logging { default disabled; } } + /* end of container ROUTE_TABLE */ } + /* end of container APP_STATE_LOGGING */ } + /* end of container of top level container */ } +/* end of module sonic-app-state-logging */ ``` Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. @@ -244,9 +256,13 @@ module sonic-device_metadata { } } } + /* end of container localhost */ } + /* end of container DEVICE_METADATA */ } + /* end of top level container */ } +/* end of module sonic-device_metadata ``` This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. @@ -260,7 +276,7 @@ No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be imp BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. The startup flow of BGP container is described below: -##### Figure 2. BGP Configuration Flow Diagram +##### Figure 3. BGP Configuration Flow Diagram ```mermaid %%{ @@ -349,7 +365,7 @@ A special handling exists in ```RouteOrch``` for a case when there can't be more -##### Figure 3. RouteOrch Route Set Flow +##### Figure 4. RouteOrch Route Set Flow ```mermaid %%{ @@ -438,7 +454,7 @@ sequenceDiagram #### RouteOrch Route delete Flow -##### Figure 4. RouteOrch Route Delete Flow +##### Figure 5. RouteOrch Route Delete Flow Similar processing happens on ```DEL``` operation for a prefix from ```ROUTE_TABLE```. The removed prefixes are filled in ```RouteBulker``` which are then flushed forming a bulk remove API call to SAIRedis. ```RouteOrch::removeRoutePost()``` then collects the object statuses and if the status is not SAI_STATUS_SUCCESS orchagent ```abort()```s, otherwise it should publish the result via ```ResponsePublisher::publish()``` leaving ```intent_attrs``` vector empty which will remove the corresponding state key from ```APPL_STATE_DB```. @@ -511,20 +527,38 @@ sequenceDiagram #### 7.3. FPMsyncd -##### Figure 5. FPMsyncd response processing +##### Figure 6. FPMsyncd response processing -TODO +```mermaid +%%{ + init: { + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } + } +}%% +sequenceDiagram + participant APPL_STATE_DB + participant fpmsyncd + participant zebra + + APPL_STATE_DB ->> fpmsyncd: ROUTE_TABLE: fvs +``` #### 7.4. Response Channel Performance considerations -Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. +Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. The following table shows the results for publishing 10k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: -| Scenario | Time (sec) | -| -------------------------------- | ---------- | -| Publish 10k messages | 2.41 | -| Publish 10k messages (Pipelined) | 0.43 | +##### 7.4.1. Table 1. Publishing 1k ROUTE_TABLE responses + +| Scenario | Time (sec) | Ratio | +| ---------------------- | ---------- | ----- | +| Without Redis Pipeline | 0.641 | 4.27 | +| With Redis Pipeline | 0.150 | 1 | Adding a feedback mechanism to the system introduces a delay as each of the component needs to wait for the reply from the lower layer counterpart in order to proceed. SONiC has already moved to synchronous SAI Redis pipeline a route programming performance degradation caused by it is leveled by the use of SAIRedis Bulk API. @@ -555,6 +589,8 @@ A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipelin ResponsePublisher::ResponsePublisher(RedisPipeline *pipeline, bool buffered = false); ``` +The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when pending tasks. + ### 8. SAI API No new SAI API or changes to SAI design and behaviour needed for this functionality. @@ -568,7 +604,7 @@ Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keep #### 9.2. Fast Reboot -TODO +On fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the BGP makes active the previous bgp routes. From now SONiC devices restores its work. BGP sessions set up and bgp graceful restart mode ends ### 10. Restrictions/Limitations From 919506084633dc9d6388899baa5e9539d0db3a27 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 12 Sep 2022 18:07:30 +0300 Subject: [PATCH 12/43] Add files via upload --- doc/BGP/img/architecture-diagram.png | Bin 0 -> 96468 bytes doc/BGP/img/use-case.png | Bin 0 -> 23827 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 doc/BGP/img/architecture-diagram.png create mode 100644 doc/BGP/img/use-case.png diff --git a/doc/BGP/img/architecture-diagram.png b/doc/BGP/img/architecture-diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..3da51a135eb6af278f5a9c8e1fdafddba7ee7e45 GIT binary patch literal 96468 zcmdqIWmH_t)&-gb4el-hg1fuBI|O&P#ytc`aCe6w!QI{6-5RHHcjtA^z4sgE{eJ)6 z7~Rcos&;kNTB}x7%{h0dl7b{693I?<4<8VvrNmS|eE0+azhz*d!T%|{@~{N|`RJ@7 zDe|FWg5VJR1=2!TUiia@>S*{EBPj6q&j2Yc=MNu{`oQnrk1N(Gzz-kZ3#7$_)jaf1 zUd>GK4;L!Jv$9^t#*)?Mr##EMey0 z37DQXiikikGWAe0@c?uYzv9GgmNPz4^o0mgFd~;t?jKzTo9f_ooH`o-&*PaxDLb61 zSsE^GZaIyOlKF~XV?acv zkFPz2H5PFeI1(OA{;UW)^XMq-sOi(FG75rT-c%NLMa4EMB%?$0*hXlmvh+f2LuwHr z^MMZpVge)G{jFMDxT*dS$+Ok3ax@&7a{Xy})Zg5gaKrCfE|(Rs1%P`!evM5R+2agt zoztOyGA|~95Pmx6T?XWDH52}V<%!>RS|>q%$%F6Imv0gj%tnn6&vvB4iHJ&H+`l&1a3N=_V2 z*i2)AQCm)GMvFvl)NPL~uyll8*{Rg<^?T=t|3;b1w&rN&Lw~Zui%TekpgvvokUue< z%ci#h*Tvdo=EI$Zg3l=N#>*V`xwOv?I%NmLWqtJE&M}(q#Gvmn&&7DozMoFR{Th^l zk3qvb#PVdcsh?o%L+Z)%z5&mL4fUvlCii9Vyuypth!n|7q12}2YR0#^!C&)|aaGqo zcoV>pzDykR57(P=Ij!o6Lt*FUq(2;m}F9=BhhO79<0CRZn@0Sou*J31MkWytFiNx7Jq?3l;!?&P_cOYYmo5D+m|fvT_vju_CkHm~#GlPs zLWLK*K=e+ntvMQnkrhsXSdXlnN2o))Fr{4Vp_6t>mBM|oiE(#9i`z2vP91WQ#^KO@ z?=>zbcEadPC@Q*Q4DPeOC92giuutS7KW7->Ocpd&BKMokhxjB!&PxIxTBTMelb(J? z;uaR-)*Je#e&srMf*jS$2}=?glv{Jh;<_jWA)U%qzq+sxabGE`u7>>XCV5}xMCkRv z`Wwf8%m9E7&A-&SJh(C6Q5>JQLpyO}O z;LQ2^A6IGf@UDt)&eR3=xOd`AVx@Mqw>(gYPEgmT3F>e6g#{vU#XB=CZJhgIBJ?s* z82`0uor_By*+Og(1hy~4+M8RVtawzkELV0AU> zx^Y&F4b7El9TR44>HA8go<)3>ynQkuU~7Z)OQ+6hPc+tZ-_M!c)7^l6aw)@giRSbe zb$9^{KLFvxT~`#GpZ;;rAg|a;)Wgq|eW&BkoLJ|sTaC1t563((#su6vTOCF?5>#@V z2)!vuKe`-+v=wCgyFIyAyRci=mH=as<&MnH$^$`;ri=7Mh8MN`sBN^E zh96UqA_bIGSps{>xtp+9f0`pK?g-T?s7GnL$FvRcH=YAT%K>+vvo{$qxOLp;+_M^> zp;#R4)Vj#xJIoTgwn?`if=M;+lTI7ln<%uzVIlOu;{lpjmaOm}ODc073OL6gU8dQ) zbNY@)eQGbhWFP}?h|=a;+W_PQQ^XXW_*xyqbD9uJy<;i>lgNxEiUP))phmiNg;jpigV&|{&3@^Uz z@P7LHJ;VzSCD_z*8Gpi0&%d_~VPi<2rC-OXlzqz*;`SW1)??T)(GAw>BYw)3{ro(v z5<6^0sz%Et8UXL7d)#AS*6Fvg+0)6Z<3Qm>;Hz_CEtHyKg_q@PAu-HV=s;+i(6XIM7doTN9e%`U$2{C$LcBCn3~m{jBza_ z$THp_fl7|XmsHX}hYHFVQM5sJSessD=vlG1Q{loHf?0UrYtx1hB#R|)zO8@FmB8YT zK#K2f-|hR#NniFth-nzYo78R$cqxQ8#evulaV~_;x4UX)`)hrQ|0IJ+^d=={;iIpg zv?$Cosocsq=IBx&#e=n~415ktHD0fIDBqfgf4$o}TCG+oibte88dKObOA)iWCNPa) zfzFE!rK_C#kv_{T*vHb{9hpMz#zZ?3R*de0FH(i9`eZ}{hg}jVSjVy>=o%JY?@E%Kf9gs zNLJ^{Fw7(aj`_;3ZbtRL#A6pdnJQ&1eUB1Og+nrbVM}`GeZSwmy?K1IJ9K+}p0ZZ( zt3v4Jx2a!YRWkQ$W`7yvKvzzu`Nu2N|4fb1QnaSjJ~e&;YDv70T5v=0km5&Xq#iom zELbQtKN9#OrOm!4cAx%}meBv>O#mFQmFVxyV;KKrlzvDCAEUoXPwqbpUUmuP2ve>0y9KR?>4iTqQThD(?BoYTwiMXnIh zirDel_e9zFHxFtc*!6+yfa8CgE*X_eW*oe}UH10dzgULiZx|fRfG`proi#KT-?_UGSlV(tH%e5^c` z{Y{|5&}t8TKA$zE0S4E8UQgFQ0Ngn`E**16)^Y2D@6d*v>YiiG5`|x^`&Ab9@p;d~4^2hez5q z-iUO1aWC}qQ_1=Ta}7v*zOMA%6Q&nfjxX(z>`wPVz@Q--bXd&ao_oau%3krG|H(rQ zcF@?qx0$_|k46(`@dY)35(Yuv?78tBxAocI$U%ll%ySK*zy3WMRR6pTX8ihV8C;bx z9`z{&VEMp~3$jAk(u=uk7+exrBVtqr2SK~uuUcfAtqGe&SAq3-vVbI-D~;f&btBVc z&&x`?R2uDcaah%Go{Oyt+ZSU`xHC%5yt)p}uN5C*NNLq@-aEJx+KWirMZ@ibQh4%{U5qE|oqTi}ggrCbPL=*Wk-XY7zx&H(#CBC{$Es(A zBkz-xH$k9=BBs&j`S{WpH=Ia)6j)6UT%C}>Y?!>?FEM0JZ(z**MDEJZg{~gxXFuFm zFz9;W0i!%S6KvvMdLbKIAa5ZX!?5~l>E7`6`lSH3;>B(Jr26QVAGMm8-YJ|(gQQd! z^2`=>eIqTjC;_X_ykXm5H)$%ULLbionK3(l5o`2O!ib7lT<&7jrnL=j!jCzpPt%#$T zWoT=Qk!M1NgG)G2)i-s35+ngl4*m6s9a`dPN^mB;<2R9?i(usX?}|&Dn+d8Xi8f2S zQgnaWens;X|K}Fix#;AtRuYk)>QBB2bX*yI>Yx#hHCnA>FN72C;bF&@Tz)oq4^4JO zyYvrFHoSwMq8C zd|CTdTYJcaK=f=Z(u*y1DXwU^BtU8N`<)pJX=bDp=A-dKBw8OrvSJ#Vc+p6V`W8A3 z9BQ^`b*_i<0+lH7+1NmR6OUM7gW>ovX`e+s>CAKpStIS%h@EP&F?7_n#{69P!3~zK zhgz^*^2kM_Jg&BQnhMZAaz7lztv*C|+?U2G{?H%xwJdq{I@+3jws9BG;g(>iVSiN( z2~SXJr$zqKD{hZDV^AsAop+<5t7zHf=Vn^cq|R|?mB>)~JyT^MeA@M4_Q?g4%iiYd zjxzYand6MzP)_W3zfN%5ikips%ByVT*l3x+ z%aGF#Rz)jRU68enSqfn-#&^`%)*UeqrRPdM3ct3}aTM^SWyp89mb`eiYf9O zpqodSM~y9W2=DS0{~GEvNHd;-5bY-OAaqt6*h_}m)M+q@sgCMrxx07SS5roZYUUixn{0YpRu5p>QjfT;JhiuH@Z|;A-@#0mrBoU&2 zCq4Fi-6>w}f`YUU4xuzVrzIxe%+_Ju9I;u9!u8vTt624>GkZ@SB3OOT^i(37S^70! z7^bGboNDwhv)ki+Q&(3?4w?)5=5o{UlpM7EnMMZ=HPII}`kGYflv?9s3bk%yP4kCj z-RBrJyvU$CIE;nDy!OKW&Vg1@$x6w|`#hkH6kSUou*Gz)Q8PE68GgQ;oTk-CSifW8 zi~?icBBzcThMQelCCJuSQR(E9a5w%f%(-A>wKAKhjpK4Vu_F|x^5L+kLLLM)sncP7 z^-zhTkU~2|bEN$)CwX2Kw>l|eKRqth^pHp@B@F1QEo{L@tpPKqq#bsiu)pY#N?1>l z`iN)r>xlB?g@%+`Zgr_lUU4a&{Mx+bS%ohi{hI3qWUv7nyDAO~-qgazkEbM-jLyV> zY`Q-gzp$b=4I4ty>bGpGVKX$n9aM@sXlAY#IV~^*TLA)8jV9F8yd4(?G2^eR0&o+ZJ12X$?|Zp9=T0%jYg9k3f|TEg3cN3_&4wp2xf=FOFiRRN zbT;Qsu@`SF3quVmmZpS+-9Fu?Zw`Ry^AO=Dd0%@x@ z!>8y)qs2gRD!Z{u!7IA4tTj2_C|~7Kz{4~RYP0OY+4+qiPwYg81_r08F~DOR+u5NL z8#nRuP?3y~%p+Mw#A*ns*16vH=YbgY)4te28vuTdr-1sd8)L3BgAINa zK-h{mxd~#rSu<7zi;N|kht}V8fuWHCH}$H;>v<*rq!)(>zb6EE#f2lVg^w*V{_^DsC`{wR~gIGS8E zpFw%1NzynHerpj{ABis>B{E32d{zotL1(Yc7cI3-Wj2MLN~Xq-z87$PTG=}tlaSVH zOacj+79gR?Z4I)WDb>Hw8d-N^mZG zcg9jzn&$@uvSv;dgC1i={8m4w9>Y|y7|{6IAt$#$sMdm?)U;gB>5{smP^)=+fUfzX zZrg2ux~%h3;LJi#$WA9!e++`Dm6ZudxxG11;cc)P=go))=McU`Zar^({zLQUm(SHu zu_4KzF*QtfQ_JO)=J43pe9gYR^1|g#aYG@74`$<@Y2!|qf>X9C1%)W;S)_?WA46rR zS!?AJYa%A28Ji;edRmL@=;L<|#MEVf3*0NOv#&)D>O?b8Bj1mAYDM93CtdA*TAtLS zcNTa4ZQt5bj}zMOe!~aoq+i$0AAVwfjHQt(i$mygRqS`@4jS^?(VijEt85a07mrNk z%B(^R&ea^cCf%fo6!F#h#b@$zi-Sif=Iaf`LW8#Z0-U;5UC)T#n*QiXh?k#%e0w`w zE-x}^*ht_&c4Zx3Q09(8xnEmEBCCHbK(F{+9*$_=AlcWxI}0WpHQimbpjBidP_iMB zadP&W-i*3P-V9DXJDs{xwTu9ptej85fU)Bb-=xtf!NkY4^(ZZ5vh8?cn71}&QC_Sn zLi|#i%7OXA3zD%P$|Vb7YSq}(tC5#U#y6UtNd2LN7!A5~-RmK6FDJ6*7igEfvE+Ae zZ3H^O8i>e_g}{>ZESRRMNn_5JaS@}PI?OOpaGrP2e}0zT9QBR-27&N6G_=?+m?)tX|` z>F7?f$#H!!J+IZ6*!n*(DfsNxM7;^ssL61%%&w3TwGs9ywx+m5oLUq+v(>S?8NL|A zYoDz?=$*%6=<=**u!+FGtrUi$Ev+wfStR* zz~#~{7fw5z0&3Cj)gUQXy5r3yo^9tf?j8TPFxs=%t9zcrF|ue4llNW_4WqxDx~O;J zi8tq+IK8;oh;m|fx)mjAtPkWVKT%z1lb)tO+MgVT@c_`f>6A`JSt>VB5 z9?S@62_P8j3enC%EkKZcI*8V$NWJU!c&{jS@QIZT@PNv&vjL=B{7t|DKX~fc0F@ND z+^kjq+1HN*Dby!ZdMT*9QBZ@#h~uoQ*DUR`NrT-L2X)j~%_)Xw$V5k(RsnwTptywF zPK`~r47!>5Mj&Io@4RSiao{v9YNj)$Ys;CkRtGYKXnRY0QiT^LbJ(xi2Z1AA8q`#; zPy1+YmiWBp>l4FwDR#R86Wne&P0Z+Laa6Hm@=clF;b4f%yVfFVE$c8GhE@LR3K8u}QIFW>8Ejgz*QTG~QNqpdql> zjG6Rjw3;sUcS_LXn1}>OH;Kh|0b$vy8Ps62%g37T&hkAzWAqr1$frayTIRMLN5FaC zKSp7|nDp*7Vnjb`gvyguS?(^N+|7HwL|xOU6x~`zt=TyK3X~E2ovTB+Dn->quhNEf z&pH{JaW>83ywSSrYR2wqUJtP^zT#<5v@aRL$2jYIbqTxduwQpdu|c43PD7s^Qxz`~ z(BN-^73qZKk)hHL@*iww$j+O*6`U}OZyY>@TzC$8A$(~QQbK9wn|u4zy`B%ehvBWQ z<4M5fF#m)*T|Bv~`B^kS>U^493L*5yoj9uGz@8iqZcc`+oEq#kh}a-I`98thTm@<#TC{zdC=qROK7~H}v>1N4pw-c0 zH>;q+d=lB@YMCglF4oUF;;@<}b7J`f)7I^Du_5Ty_%ZsiUu*?`2(vw|KBWLN~GyvX`uv#sUbPEI85n$8PG%M#WKApZ4s&B;;WHCjRM( zg*Ht!JwFq}ez+!`bSc`Y^4MnJ#6;`}RgG<5a6F*nuq~pi{p^xWwy7^tuuQ%;n={{L zwN(fJ8}qJ}x;=U(kJi@=`(IcPh*4#y{i{%@^oT>^O_Ei4?1X7leI+G4iYU55QdoxG zJK|XjTihN^U5E-AwOCS}NFme`i2f0PwJ~dphktn9be(!9XM0%EyObZ!2O`sfv|r6s zL`n6MV^y^fH?|-0P5MRJYe$K-K7Hp_mb6A(FK22Nt&y1(xFvz$#!1Y~VBuu_3UAO9m0L+cXsc@i9TfAIVt48|~& z==f2s`cXIiv6M>(nTcNDC?I?)v2mBIlL5+|b)v{aC91}(-Evm^ z@y_~B$aeMVT_Gv5P5o0uAO?FI~hV(X6hRIK5YbuDbk*cQz zQ6r`uX9h<~1xi~iW#=QRr|v45`aktzrJGc$Wl#BRiEO65dRIcMT4OTI&#ual;q$(& z=Zrl`tHd_XYz!uQ3`*>yBwSRC2}tJURH^fJ{Y|^T=Q`hYs91FKYlcur`=&>S*c|Z| z0r5D}9t7YM{`*2rhCJ~1XtO|R*fQ!5erjri=&OowLwj{2W4A~rPrW-|xxT#!I=>8W zy;7o7#9bbD)6BaZB?cKEe%WfkW2jI~kNgx&z%Bt5s&MQdGm$AmCj6N`GcABKbQXTgN5DX6>6 zyKmnLWUcZ6ykTcnl`3%DtL@dI5TymnNsQG84Q&)QAlJ2${Y{yf@4hc~n|df*&>(8H zpW5yf0ne=eh4AG+ngyuLEGd^qPUDtQfB-0C4^_$5A2brHtz)#lRBQ;&<&6eQLf?9h ztyG&p6jI=!9&WbiPIMb;k#pfz%+ibW*j+ZSuXEVVx^Z4daP&lvG=Gh&5+@JIq+I%Q zQXvK99qE)3w>gIFEyOX}(M$*jW%jG5pM}jut+>nOuHFXggs#+z{OU+$3aDNgpth<_ z6W1srb`sJcXyVN|A=l+wwew;15*3aNrU}H+T#4n-u7U2nF~;Y#(GIcZrLc@tDI66! z(;s2-!9H+S(2S5mIOag3)347JdaVy%Z0fuad%Ql?E64M*5l`;?V7 zp#Mb#G;xNk`!kH|tJh;#J=2;8rF=dTJIu%w&4ch@;W+ZUy8%#q zG~Z6q%cX|o$Q5T)Wb_wRjbmS#bd7k*u|Ln0uX!`&AKD=I{*sLt*{Q163>pda@ z7rJg`8J)Y2O|<8ECGA3`(i=%1O(GZ3(VH&*hj5>$b&&2sRtW(5uklFrwM3tz4y}m`(v|T%Kv|O=u3as|~vkg63o|?SmkORrA5dh;) z*d6~6+kaBJ+My#NB6F%s;h4!a9mncL`Xkl6x6aM0oz2+DVt>Le#YAlq{7| zB!6bHx7`>Z5!Uv`D@&X6p#QfA_Coz*1$Nmftp3%wlDkx(x8#oo4!TXxDA=cj5Qz8?&i1zPV5eU&pIYr!<$usVTemx?(lL9|j z6dM$n9B?XWDF9pW|6LSyQx1p@e&oGSAn|tgeZIGDq9r~6d~q1{NgY@X|Ca}_p;x@{ zP}KoQtvdqqCpuQnNk}s`I1Zow)fA9`<;fIM{q$E;Qav&ZK{yyOWy5P%_b|SW$8$jW ze^q2=;rQA1=j&gfPjsRHpt0`np0oB|WYos}Z z87A|cDT4U871xn|=+2osmGv-_C+j-gzo((5n4JanM1j4sKdq%qchPYAWs>2)qv6xa zQyPG59|egk=}R8N#7Y~k113h3bEC>--mg#!fHMI`Q)Ot^t7H9N40YDz(|z0xlG~Qi zrTB@W?$4iBQV2-*kHpLoje+(KFcpGQgXbFLCAw|*R+f)d9*++}89G@T(GA=&DH%3@ z3Bv?%{6XfK^h%X`{^Y;R-Z7s*^?+G3-S#T~T}B_!z59dINp&Lj-o^|_eCHh!NVh1U}aPfL2U18IJk@f^vVdV%Rpme2dlZ*P6UttDpHvJy+2af z;JQ{XC0C&!hcg%(_N}zzoV$pMB&y&aQu+93+z1cm^SjP9crL;j z(Jy<&NnH&dASFs+#SN)E5>g9|?}JA5HgP=o&ZRGwJTmZ{%6v|qosg)SI`yuqqSB+Cz+Q_Eh_ePOTTDzny7pu^5KJV%4*X!}JKhx2fJhm)kn<<6uXB!R zosbHbtXC8V??BXBhgW9QW>bN2fK0MNN}3}|U$(c!Wcb0h5dIyd___B;>_r=L58Hsp zg)8o_o~#v~fQIYb_h{;5PV=?(-6|8^i)QSxe9AQ>nz&dp`8ZE}iXioHRtzIT=f0P<$ z5EWPFZ(CP5En?#7iAYyZ#tUM0mfJKR9$I3GE6uACbYv#LY;?VY>h);!3}AlLMLd@6!TFb$whM#*R27(3S%jV<(sUgs+I92`~+*iF3!glUd;4z%}>AA z(qDWbgk3T*BO`A+12`%0IrCT@LBbRh z>myxTrB+L>44CPPrgVZSKf%H;urtxY0lRHL`rU8i%T;62R#*1I1B-qqc5&{v zS2uCe@ASgdFNr`FvH>%nh-D$LGUB4yPj7#UZp+U!o!HKO%Ncc){5ejIB~{!M_fA); zzxw!)%bgc}F|P)2tc-QmXS(%Wmw~AHDwZlPDb?V&R{b7Uw3KZjfpav0A7cblXec94N3d${e&-tHlu9U~6(y<;E#EvbJVBk*U_^;wMoV)e1S^&` z%+k*xIiVH*Rr^h!AIp5+!&U>+8b^YQ$v+(maoq{LctnRE@=ndD?t}Bx0PU~1ds7UP zJTkOi99%p!VLTM&BW$B1QR2Kt$*(f`jABX^1HH%5T^l>cw@z(6&wO7-H5po%W?%~@GamL!P_Tu zp?3hIJ&0)5sUxYVR%T-lIp z_J0jtHI92)|EI(lT;dhf0w8-vnfPN5UZINr??W%Yinf7{!s?*^AV1wW(cI1vs4OO;mAPQm{BYdhxmlrDMUMIjh4tGc7|gv91i z`hUMf3a0fNr2qiN{mCuH6 z!KTg()5z{STGw{a&6!)J-r0>zvbnyoin;Od*FHliky>`%c@H=ZAPy!ffg6eT6ZKRK1G zS)P^6uE6!oE-tJ^94(;HGvX2v&YkYsXrbM5%dg}4jxj|f4?;3P`ikm#eDgpmj9bY2 zleYw3rXrF>Dy|X|X{DbXcQ5fU<_bc&_=Vm5D0Talm)-%~_7)l4Q$s)AKQk&M+t%ud znb#mJxT|eiUkMY0>jl?_@&fvOQ6623-y|(2ehiK8w85TqG@^K)F%EpB_7*!*?!#P< zJ0C2P&9m!V2vP^O1#k{e7=u&UXt;C zyEqqWBdW4ivKo=7@Nf&+^9p`FWUyI!+h5=!*z~F-}bsP0^{@4QGD8w?Z`<-+?5WMbAyPz9^7)9B2(;3Q98|%jwG(MN9}iJCODM= zV1TzthNx(WX@(Y!iFDLjdlbNU-8R%un&Etc!VSlN{Ml?^b;$Zm0v#26)2wCe2Lv!&!kwg}Zc!vKgTtyEz$L?Y3JI^5kVcnYk^=QWQ;;(3(BI=Jw(l%w#`w=0{Gobzp&V3!w~dtugP} z_o#!6QJQ!@KfZN-{h8yA{IF4^*xhr z6*5zSO3#aWE|(Cwuxb^e>5Ferk;LdVi?8qMj~l=icl>Md>5Twd+*U_&8%AxfqrXYj z12@Ia=CDL_xOo#8I5r{{A%HtwBC5eJv9H@bpC5RDkD$jtRMNckB88a%rY_Voti^bU z!hABMm!hsG^qNxL(3G$47j&{WYik9k!Fz$9RcUdPV3LrNi>Q`J6Zq^%>Xs)H3dYhI zTJMW(bpdA!CCW0Zo`Vcp4Op#bz9cL)`ny#cjr}YvrEa^I+VsJ-yB)Aw0_E#p$By)p zPE~bbYa@aCLHO?jhY#gmZS-h#jE%l0)7lAPOp(K#hE(lVn2t1Rq5Y%)yMi=dliOX> zv;tcTw1yHsl^ocDGxL?t+zY-Qbc3EuUIpgsRDTkTG^)=M?F2{KOLB*g7G?%)3F5BR z_5`s;7N8c1JnfKjR81q5=o|{OV_)rN$BeW+Gh|KzPr_*$eDscZXaK?!#H{Wg2}pnY zAdcWU-@F>)41}9}!VYLyht7~+{y7B9JE&sbMA%7;Ge{3}j>;S=itt_u3Tq0_Q``Jw z4ZpCwSb|He0kJ+QP#Q(7g@6RsV|fuD|uhp^2{{rQu8yzVDZEFlGi@EmU}!L)gk`Tyj^+J z15CsEo7jR5bg-0G`KmRnnIKM5%9# zl_^6$>cTndc|$R@fb~!;sxsevefaDHvyz)_k1cV|!6#)Bun=iWY{nWmDp(%r0}fJG zj0FqJ23kcil9YrRNgAVGvPtvgA@Gj6grA(ob1@G2@X6M_;p|C}-FbjUS;MMkWu2#KF zgS=gIEzO??rCp@dFEz9!*;8TlTG^hZ6pbxj5;wQ2I=)LfwyqV|!jb+k#{4(K5XU2D zDb^(U+0^w_SyyC8SHBH@JzYMWuc!e5v|Pxe-cupFt?NJr_E3O6mjF+$smGY z&EZ3)sZlmCp1Jj{a~<&kBTpR zU3jC?9Zd8SnMW->WJkW`IagXoOKMzKui+Jh&nMa{&d+aD%zI{Dae4d1#ZIHz<@J}X z;Kq`S$cT-gu3>@bX`bP93zS!?-rSz3WAo=%&}%m~{zNTz^lkH|k>n~BgyIlJFX!O$ zhCwH1yPwUu8Dq$Weg1XUG89t#;NQRxF9qXWu`b15q27#9QdS~H&+F}k6uMh5`kmSp z%nf5In(tBtjL9+EG+iBKi;oD1M!^z+Vh4My7nMFqDw_pWj`SiysP^udYFva2Ve_KR z)vI^uVU<6du{%b{3KX38)9-X3=g=QBtK)f+$ag6#!{ zl2F*(G_;en zbvIi&;)A)VWZI6F<(OHNESbtSTyms+xr;pY6#Qz@`7v<(F9UQ?%{ z4od>4yvU>a%BGPhPwdhe_b)V7r0o_TOvfm*$AHYELvqx$F<^S0rg9OFAVR<36mpqj z8d{R`Kf9vaOc2-Zw=l{6Kujjf~0cwoF=^zrT8RToxCTUR=>FUXvcL-#JJ3A->~ajd&# zdtCIWzi3ktnIu2ya$MdRei&a_%3c;RzYZwX;5KC9MI?ySs!Sb3cT8uSec-xJgUbf@ z37P_+=z~pncekDVH4yuY`6k*ZS$#E=-#j!?V^HHjdc=7TlWe+SsudYwotPUdD~w zsoeg;WgC%t^H&*OU)A}A+}a_0MI9vKA+xZ}7gJ76ZofFB@^3e%!!L0M>5`c-9 z>^9eAfjK&(9KtWZbS}`=+E-T?m^vB6S;N6+e*CuUPn5-Jmef5v8!`yS!byBvJ0^9!@lg!0LT9P*V|59CRrn*BF7(?#(-%Al(& zSutW2hl^XuCUGzy;91TR^Ap^p%?Fk#*d5!6Ng9mKPQtT#A?%C?_vR~^;@XZ5+~fM8 zR2^m9H;>&1YM!kk>-VOxu6z3LBrE^Y_2IG|`g7oCJeuccWH*;x{ji`|eYAdF{W$$x z^Fj>9`aUcpB9&{fL_Ge)mO}JAXAuR?s8W(HY~Fv2NQI)PJ{aU{;tvbS@L_KMg#T%a*$E%Q`DPz-SdX% z7{$|12PHot{GdzYZ2gD5#kBlu{y;}W)CqwGvz~iY_F+UMMY_2T6e!92wA-((!3#{RN4}6%x ze?$0b)TaHC$$i^1&k00SbT7ZlCBVg>xibYLv|kFINWf4->|)GM)!}1D%JU{SqRpK) zE}l_`Uf!ucnTRrcsoDj@cjjYhN>fX$Qw%`^r2*&aZ}2FVlwcogc z{ZV#3L3M=M=6V@!IELzKPUE8OG0H<8B%^jNJ2HFrUp2%^y4Z)FnaU}k$?0*Q0t2+d z1e%+mcvf)+j)jCS9->V5qB2Du^g+mQ>ouGXt~#+Dh2B9eyS`^4x`(b9Yz0MA{0z+j z*Tp(@Yi8aC@EPj&0BRGMxH>&^{IphUvo*CxJ~d+xTcwR>TjxO;{az1EFv^ju&5-Mj zjMJir z)f&1ZUAkG)R;r$><@&iP)my9}ieG41qI_KksS+|bx>v3N}N{py5(R9yulvf zUgb9AA%L~XPu%PgjUUmQq&>V!9oY_LeXt(sZNI-DW7OwcE`T3k?R7IdDb_j~me5hm z8_`kQ^3)w!D&SA@B$1};4CQ^KR078NcWe=PcpcT5e5|$62C)t>!}|6zgJCb$IvTeG zTwa5E+rk$^rGX%H1=&(m{ z!lO4?;0Qa7>)A&7632SNO!4vw^haFU&w|POz=b{bCbd4CHX&V^jp9+RAUv3|NZQCL zc-UIYG7eCJ3IR*^cUYVOUD9kO^OIfTQl-@DooJf3sqG7%OEc7!iKx@Pqc*dUG4A{w z{F=0+)aHn^!J$}U{S&=WVolpZYeI^M#Z+$eCT7nKZ1sj6FF9@o3CtZ87ixw{rlzxq zo!^2K@BBMdo_N6f!~-XSmczcnK{vx$kd$snoPqtc8Tx{^x+8aAO>$_sGSHlwC5#V; zus&j9wCB#|*(pG2RXO+6&P8F$$YB2ycsq-n za=5o2Cz^CoJ3Q&85QuJ7&wLTG>o`O>!Qn)1(&$EIeNZx$D!I-5qTpUH5S$sbG`qm0 zkG&_i86-IXICA^AZa^I9I;c_&a)IW^mK)qtRqZ>Rsx}ibH9X$siub2SXs+CPPalP- zwsXob%eS#jeD>eGI<;V}e$}OD$aFVaZDwol)JuLQhKEd646`Fif-;sY;u3sbawXxa z8Q3xAa2u+LZWuUzdx9>@9F#Lk$NMhqe%E*;M?e7ExUTbClmtl66$pN+A$GA#0fv4> zD(%#M#S5Pate38}4RkW9a$^s=+}j+TK@=pEOHmz4^r*SH9{+=k(PBgDVyih=q7|i@ ztn?j%xnI*&EFhMWV6$a4D2Zh+5$u}oE#;HWq(DT<86~BKs4zT6nGzm$nZVpKFFu)9 zWqnp6%x>jlakDQ+e?p!ZB5C~8Sn((%>~=in*xW`>`-*Ii#N#bKAddSwW&5#uWl8v6 zV@%HkLL=w7n@_KV4{XMmO-2q+UP`K_C#c`~y_oxFT&uaZuG#8xd@fMM(0 za0W~0NE0=2Q{tY(T|^!&@B1H`Fht`HG7B86K1kxxM)SXeF5S8(+5AE-pZx{LxT5qz zecrfR#SEe2!w*-Lff*HAylx(?ngBQg=h*qW%vU>}|A(iq3~Ta#-~NgqASECr>5wh~ z=`skB5G14#Mo5na38lNFySrhmA(P?g>ycw0z|tN`Zra4Y_?+j?7%;TQPZ?YeJ;3oYuvrUy)dA3(TdZsj)#>nHzd&&!Q?rLixn78?4fJ3616)w`EY z=WqwbQ;L6Z!0UyQ+#SftokZ-oXWmH-rhVNUgMPKC1z`MNZ%saqy;j5e?nen|>M^U2 zp1yJlUuGxtY6~bg+8f;<F z6xIZ#U>IhV;K|4EeVAK2l1mnNKgZaUBpHE~%!SC^r_hYvBGybM7dJ?zGmCC#vOtZI zKB-Y6{4kToWIY@WT?QU?4t3(JzU)3p)5Uylw2gIW>F&;FK|7Rh^G-@TZOEZj5j&S7 zQFtaFn3WXMX>27pqAov%Vo~!t%;E*j`5!1BGt6^|7Ax>w>2~ou_z3eQ0EQR#T#$JC zkug>@rgjo`Ypb?)(hqZYs~0+E$uBI4yR7)fHijz{=&gwT2AQnJgOI9Yq#vfpRj{?0 z%_zqU!3dww9{xoJFK{EVy6%KSjQR9;Pg%RK|0x~}K`l}}ID~1m4dfbrfAHMjX#~~> z+e!(GrAYT7lj1L_Ot7~0V_j<>Ii{-!uCUkg+kBec)BTgeZz-x7O3=ECMk)G}J5^7l zZE>*Xi=*mny>J+;y%OYH!SW0?!(@U>LZ7`tj&V{_eCqH#S?r!sU!bApYfDJOSQ`rV zXZp14&!)>8cf1r+2A_+&gN0_}vkb^jaasaq{KHof7qumZ&8cpw->5KPhoCp)1-%P z0IFh^+!AOz4k}d8F$v1d*+zt-gb*(_UvJ!%r9974cv03Kov$DvXc1!IYdC~q-GY^p zuYj(IZdnCWh=eMwlE&C3D=q>LVq})AIG-0?ml#Y5X0^p(unY|v|E&0nB+BDIfb^3&5uGhzlvoRv!MgP_N5@KTb;Lxs4&E74qNqzWT`-}yCjR$2 zLXaV<@k2U`c|$O24>*;9UnI=v+n!T#YMszXS_E0^o2a=g9*1A<%RixV1MzL$;*tIGE4@d zu392KV03dZ#ayr>?P*MA$JP+3GIX@#!MTajX^~I+3}~795aJ6aR&wdqEy@?D?U+M9 znrmEw(7w%{8(c~oroGZW>>KL!x^=e-wDnR~VM%P{D4&zs&VQX$?lb;^-|DbKT5Y$U zkiZgEV?Ut#?oUGP-QYFcCahMWg`FHR72J#;kdnabw&5kLdmSmDuq?RDP;Hr_KQXRd z?qj?d!&O+#lI-x`AbiCd=w*#l8F9mKZ_?=$2Yr^s*bf{w@9Ky#nRnuQ4$6EG58kmAZy?P-{EWIyV~d>iO~57inHw6@N<`T8n6TwplsJuCqr0;b z|Me_Sr~qEyY~)?RdW4#%fH-G;XyY}L!@vi{eKP%Lh2vi5vm-AK*$s$cG*M-neaKK= z>+ycz^-#vwDYcVQMORyA=7e#_!kP=36iDl4`zU>yu+h4Yr-W}r=O)uSzKL?eLxGm& zo@7@FGVe-be=1=5HP(yU?MjdQ=cRUeQ@Th)z~~q^^QWnZK7aA2lMeLN6F4%8lvzKM z2@_|SdDH`VmH7mn8Xwr}FF+H=#)1RFK=GAWN3>jwha?M8T9tYDiHR24QPR#4beB`?5ae zt;?!4UwH3n_=z^XVab}Mn~8ZC%I2>-Pwhkg0b zLZ71cQ<2{@#z+Iy!j~b8gVtD`n%?r2d^EDRbBv$kha~5KeP4aTk1YC>4r>l<^83k; zxIdETfi*eM$?;^(krc*sV)4^zQgC`}p@~?y3A5S;sQw8*e|wi-Wyqxh{-$B|{`$yx z(0ww#mgAR}aia2EVY?lmF)jMyawXx^G0PsDhf$6&@pQRa!*k2x&j)TE zAgx1+*P6OqC&dks4BbgOGhUJ`(|{O1>yy^e%R&0fYiG8sPhzlLYa-=yI`Kwo7QfV< zQlm*rT63ul*aG~`b!eZdw79wdvGl0GL@700pH}s{Tg@BfP^!FM2&UhrL(k9F&R=(n ze2aSRY>?-jVeN>bvHBv;)cU5uTv%8?aIoTZh6HQV_3Sc2z~JmTk(vq?HBV~Nf(nKh zsa5HrH@XOZ8_Vvt3o3sCxD2zPE{G=v7kXP{zV}Yik1R;Y3btu2EvX|mBv#TjsvSGy zQA?FPLR>EI#bLft$OX}xxMW~s&NH8@#p%B|**R2X{y`21r2dg0|^0Mh*QqThDb z@Au$3Ho$WG{18E)jg&Jk&q{9%r~ zZH!<0fA`wOPFEP=hxxa?NJ%qFG67HQVg8<7I>_Fez9MD(5~SEj(Ap03Jv9k{c&g^J z-*z9Irm!99`j(4GP>TJ^PRh)3sPGb}yvcuBV*zCw8!fOJ2oKEIhw0?)_oxTPa0sPv zMdeQhlt>#nt`4!S&(JD+MPC*{IG{8HD%#5fR_~;3^PO-DzV1o+yr-+y^78*<^Ef7q zEuybtTQw=%>Z(-g~#^4zuH6mV&R{lij&??vpd_ ztklKfFTmry4m#g;^0K6_-4b+ss5f~kf4e+`_8jX*cbiLX@q~zdYmvNAxT0>|*KtuQ z=@;m|P^bBT=EUNDBI9NIrFa-viYMadq5QRYc-?K{6){W4WlB`bY z1cAk1gR7@xLTLm?5BGR5D-CwBBxgvo)EO4e=rv&zWW^tckPd~veixf^(~&W{?W z#ug-s3fRhdR`SGP#Y0B@;ejw~<<|%~Nk;6;RJA~&w#=qGhc?|`;j7Lhy&M)DWR0mV z-`|D4SF9g05KYXR5@%oB?c^d|P?0@xxIl$l+M{}3Mb!xJ)hC1*V=jQgN!|}wzSeW4 zR9s|@sanfk5GlC5`XVPPz8Lpobw_b9O_wZl)%0SdR>vJa?)6?s@A1ZokF3^j*x9#| zr-dS}#jpE^h6Nc82Npmn@jNABX^)*1qvIl$)Z93Y3-@wKYnFGiKe5jw2Ik%_zz&Pr z!BMqVn>-aR>2k*7i3>l*<`jP&@GR1N^B1nA*C=$St@cPMKH)4gr5#Bt_4)0Z9wv}} zgsnqcESo>vG$=Wbc5HV-nzl0r2?S0=cPy&>AeOHmq|7X!;mR?{Wkhahs<0=r^?mvG z(mDb-)Urd$){wDOb`h0ZD+c-uGTWw^pJ3O*!(S+wNK3T%NCKWE)F%zwADK!yQM0v< z>elG$eRLNUgy)j7=$&)tRIwsf%%?dx&655Q{%4Va8ImH9IO+MKx+mLERr8N!gviuy zhE2ZWBVeENSvW46<0~$3P7I3(xwVi-Rg!#b(R%^e^pn3cGk(P%=%So+r>Nt7EX=zb9OGMK)YS(BsJB|)e7VUo(? zsuV(eHjbHS>#e+b>f=S_>$Ill?JjzK?+J~rgNsF8MfX^)!24%sJjEa4mBNRKMGC2c z20G@zg)?jsw#`q4JU!&$)fc+GZQCM#a8?5$vV2Lpa>@7|RO#Y_y_TP0V)50>QJ3s3 zEwHrX3m#Ec394s0CmlC*IwY44kCKC-XcNwT5l{PjnnwqkTXjjicBx*czb8Z1E5t!x z4@l1E^)>hr@gpS7k}YhL#Zmmqb9$*T3~%3e%L!)ShguiDhR9lP#ETT!Mc;rfve7vv zvSBYLmo=PKtH`=9vS7wAwWyLfhmbE)`OszQw?0l!zZ}Y`o0g5TYduiE!U%h31`(?M zx~J#_PBA1z|dkzWi?F2;G@dEC<<0}5C0-{ z%cntxaaJwwiMoOk-vqDH%%+LtE4Li}u4W&A*KZEpC;V?)oyhVlfK3yDt;yrPw=j0~mxSX?XpC9fV4!G*s0aUf}vkOS9-|v-@>` zgQ}{qxlT$%5c2v=#5EuC+^TViLQIs0IRXhwvH{t1LNM2HO0(^iFs#jOll2z!i0S}l z9=GP=TUkmlbd@xaEV|Ct7JOuJs)n9u;RRM>K~{9d>P@dm?1=Qg`czDtS;9y5%jS!D zS~&slGdSxo;j21=3`t%y>Ok4cEhS1*wXwe1d`4ut8R?)z3Sw@&YsLXeE>e#EqEv5v zD0cdh$tdfyZzJ>E1s&gx&iPF+CAw|R&IO+(2Hc%Ec{DT9)3TAk{2Hk>++LCFhUc3j zLiSg&^p{jXuLewx?RE7I)mDUQ5+m#{gDXG#Zru40?GbS|q6*qQuZJ2?1u37Foq}~z ziJync7I*B>_iA`D%VLC~IUkwRkRIBbpEFilm4f6`PsE%BK_8KIsj%4cL}*|FPgD)= z;1qQj`^fph3-y(3D{NN(641J7eNNuHu5<4gE;`8FK|LXx#4}0pYw~LQZ7KLQucpWF z&YXP}L0Pxr({AHick$tOa&GBq$rz2P4T_fEsGlJPql3g3Trj_onCtmyi- zbEZY74BnT#n@yH-K_3zc^o%NxCkFa;e$6Q~Zq2zoVgZEZx|PlxW3YBb1xB|H(9iei zV9f)bIW^~e<|JkzJ=%WmJ>|yKtn3w++nj`CMwqUmrTjd!~>&7N5l`(-|wR?pXqC( zCPy}?l|~Ire+0-*YVq|Avq&Hivp z`pQ1P`-8q(kJmniRUC0h2ESyMA+%k;nnq|?iY5lf+%djb=QG&$Bojv_7VIy~t!vPO z`)eQ97FFV4e=6e;wRR5su=q_=(Q7~Mu=nnSe{hxCzFU6&8{t%=;?lJNA4k`L`dhz? zx$~M!3+aOrEQXfp8UKirhTXUPq9A9lL%!tTg9UF1gwOz%#fLtmoh_$TNI2W)apkO~ zYk1|_p4f2%F{u~o%BBtNl2JXL2WVtePx#fQ!f1o)u})Yq!qsxKiKrgjmG#VR$Kdr? z$a~kds7b6*5&7;maUR`toDNL~&Az%gV7J(MVVEEnoLEne%ReiYvuA5mnw%(3#dx$PX6PHR_le+lX6n6~KWOsZtcwd=^l}7nW5---7{MPiwJd3^9WIbnAEztW&M<(S z-=oKKM6u))*ygg~wm;zCqm)12E3fAUvIQvg=*4XgRH)4J-$|r<4q;*PT2p1vS zMz2wZ|&V*;eRMr5xicv94rLs^Y2@7U+8dj z7v4_7Xr?nBwpKOp*0qQ9`RkELJf_p{-%Mi%D;Am^AaVn`SeY%yyA1eds((^r8jkOb zP6;jR6VJ7#H;Zd_^-h=7kNS_^xlflxy0c5u}-rQq2m8WA|N+(Z)}BH-+X?9y0Hk2XcX;+{A2OUK*p zWn@M@(B;xOil+^*{d$RSM#6xMwtZZ1+dBv&Qps15up*PqPN*Dj?IYe3RVsL6klhM6 zPTtbhV~6YLC)X4q^r#Esc>jB8PYkE$F$sa!U2Fa%P+sOOOc%h0p8pI;>yV$q#*>QF zWWxvBtx3b724JQD$*5W-wc@r4_d9lTS53t~C-Ht&TgZ)SvWZrqr0pL~T8LJm)Y1A^ zizI^7p8=iPN%GINZucmBu}?EGS&71^a}X|CJQJG089<~Yt4r)f!fLMXAvcU}N!#So zRtz5NOblyQThyh2ppQHQgsSyeddL6}Ku!jsd{)Xi6+;Iohy-cPR6f_L$C?D~+1_Q; z`;rkVW_MCx3VI1+^v6zz2NVu!`Sr3>Y(nQDP;?7LvceME_S4tMn$-9YwrA?G;e`mZ z=mfuh=50yY)E3l_rhu8>LFLi98G!7>3rsNRGi?W1`$mJWKinOJ#9)Qo4Y^Lv+>X1X zE)V1DQO=m{zgeM3kwcbLZ)(Ku>;m^unH!c5NE-mf_PX|6PXK%tH75N46`?u^?{SM7 zBYE;_k-+o&NjvQm{tucj@9e8;FuWuDM0(J@PCria^6-aFBz0@tE`QW;+3g0%{hk8a62$cJddS{HDpL2#n_{t=|xe2s5+k%`=!9zHJk@Ank2a7k7?mk2H zy$1Y()jvKC1y`&}dMSQVR0eN)0k%*MCf-QJ4@okcD0X7mQ*^ncCdG?&!zR`%PeVuG zxbkSbJ<*FkB=X$QXSj2RRG40PZoK519Q-Bh5N0qSus5PKuI!uBcguXb_EkX^~<#!2`Xpk05v$LAfWM2w0pb zGbB!5ZSy!|nF<)&kIw7fI&>b8nH{$NI<5VBj-RE)P^!H-yY>PlkP0^typW0k_tz6y z`5>TXF}#WdwbD6Z;M_{gt=R&t{gvJ}OQBL#14n^p+L$Ag{m}RDX-LC#`^!binZ?pJ z-VE@=#i-%_!dpIJxo+yu(VmZH8fRZ_5a>M&a14yvyOB(JuT{#s4`EjVdnSr_5qlY4 zIzOf|)lFcBsPrNMo73JC>G~C}t*j!a)S``cA44nLfrAUGg;ZjV>Nm%^1%_6)c5IoS zn0|PI+FNlyI>{E}U-kNTrhJTi_n=~vJo9iczlpmC)@q#9@(24SKm;OkC>XbRBP8B% zLnGdBK@+!h9ecsNZi#K1JH4FR#l@oTY&W8rp5ypj^Tn>MKc;|WLrue}IR$?j@f#6( zZ$ze`VExS~T8XG$2DpGaaQP+GWT5;JUPqMHjF(~kYrb0Lv9($OS>DArS3Xe^AR%=+ zOy`Y=G!C-h%A?n17m*uZM)wp3hXV};n5fXNfbkGOGn+8Txe^c&uIQV@k}x2FeSU5g zEyeMK*pK50YJZ+tbI$;=f-d-~4MA)soXE%2yurLBlv^qsxM3)6BzYo)#&VaND5G$g#H#0hhHeT z+xGXb0y0k>G6 zVdarl1~5yRl~MKNn{;%D&jEkrXzs>@KYQ=MhQ}W1b^l`*i7&GiHI2c!a~MK$NG5wQ8?0XEr|fK4u$E&7NMpqez6_mxlL(G5 z8wzu~Q;hwsr1cHBJHG_GnNEk2&Ib&1!@V{rC^- z)w3nD4X=@U9av-ohB$Z+Kx9`$^jP#M%@qa_gD5~r=^A@c+CJi zWr>vX_}PQqW4xs2`VKD%l80iKJwFkfYu}XqD&Xci!*vAZt+9qTsMFO7Yq^y_v;UZY zZmogAc|XsTLmDaLg#X_JQP6$A)E&)PwXX)iy=3G!bEfdp^nRUdDUF7IFN+vuF^M~H z2Dfoy8mjEp278Bv2x*?Df0jX=rs6lXaI0fpS{~Bag#6v>tHdz#JFpx2l%9~^FM{FK zuLDvk_IhoTF~zW03a^5@_nooQphBL<#)HONTNJM)kU6ttEe1{8FK=}hM~bafW~<#s zGes@M^}XE=7*z#sP10Y6LfUr5{fF#2Wp+|kUtsi(FzmVJTP`wLpcbZ-(p$;LX86eJNYv=f!4YlQq;%)B*$avul zNz)0T0E5s`SUOzFd;rgYP;jXA2|I*~(~6fE8gwj)x&Fdx@y#AFp?1}DP+a&i)Z#_+ z%yf{KJv)B5XxsUWU22T*^Jx1Q_+K`!dsEhr%FdBH9mNc{(-;UTE=c z_*dY}hx8*uuu{H9>ivn2uXDx{SnacY#iK7mkFKJ%Imu4R)M}z$Iy7EpkHT&vi-UVP52$FM>Og9AA+N>hj~tEv)=Zp&N% zSD8!eOLjNlo*4>zc=Ap7W^`mOBy&`mkXDn^c-{F~_qSr3?KkoI?pJ&=ny3b<|7ON# zBQ`EPQw_WURBmp=+C>Uo;ogiYm7K;k3*Fa13TygZ82CC0GKJ+04+W)Qe*~%A4 z4XqHKqNEY+YipnW7Y8J~n`qt312`x&GOamp`j{pKCJ5?;UwIkrI`skHL zo9JP&)d}j(4eoPqCqjsutftC;u@YBJxT{EJFq-MU*PtFgBOfuuU`cMTU}cJ3X1w2# zz1k1(XK&)4m3zN@JU_z*psuR(XqoMd#?g($-e{e4UbkK?y=%9bztaxs?hVs?y=w`& zYQ=y$zI@!CgEUIysb(Qc#O}zdBY{o2TY6+nH)JOq^vXdkqq%%mn7;|sP6xRLLFng* z9jYq0^L|1TLy%`D8~s%ntD0tR+TMSiPpGM;D>I64$g(Wd*5as!rZs8CLPSRCm&X0M zKjTK=J|ngtcOX3?qp;|9UjH z3Pn`+H9~guY86Lo_r8jAiWf;~>Xv%~hib8jktKQ|Uj(GH9+zd3z*u&<&$x;C6a-NT z(q=XQ%f~=xgOSK&M;!dWNf?&PF)oFEsltwntM6Rr6c56yvBRvUM%!G@Aw+j4Q#w{; z(?ZK_BU?ZI5Tc~88e26HR@Q$%(=o_JIN(Y0N?V?N*OjFWr;|R1jG-diTw_V;&*PnK zioFR%yxvIy$(@Efxv*KDO*H2-X?XY_etv1?WK9IelT4uZxF1_czqVfV`}^jbaH+n% z0KsW26iEz@SKROYzz2RRp7lgPI$YH5eJo542vZdGB9g)l|9k>N2a;R;h@h`SL+-gE z?AGJGls<;1W$5=N_bB5PXUH2h0ma2)#2EdxhG)rt@ify}S&MVs6y=YczI=(8m`M6M z;S@8ocFFY~IHo5AA`e2ltZqV)4y&Y3lAJjL?+rA{gUjFsIV8?s^d*IWSuNnh*9Aja zN`jHC<8A$TDB~}bFz>t}fM`x|}QK9iZ;7*1ZnN`&j;WdHWzFKw&AFs(<4fODbS3`gdI<78gm0x+0 z+T9*I z*$#OsUI)GiB9Gq~r(%h1ygdfU=0*57j& z*0Q#o;(<*<#VFosDd2_+#?$8tfA|f(9F_;FA`w!d+E#!7#As8E{r4GQE0-S~?OX8q zs`fOz)Qj!P)YOZ^`!1Ud27JTeq&eymZ#6B#r)DUInL^QI+FaD<&n~$Z1y0I({b^5D*#xWPS7i0hELgPGv5Y5&b3TKR}!>w_c zz9oC6La2Vv;%Xx@JYN0#;4TZ1K3S6}?ZqNpq!I zlL_w?WVp+W4&J=I#tRLdt+ixrXNNG_*pvMGz^NAp&jBxS!`-e=soIXywRT1^E~5g= zjf&*{Ks_N~AF*oNYm6H)cmH#okm#{1pnfu3uufWVm7UV@H9#9B+*lT2Jdsy}ZO1E&}(!mJS-1y&nAn z@#TUDHXo}@i{x9!CKyA&Y2~WMgR;(v%&N-I<-~}_K>(nhaE_CUSqXB-B+pEA^J?Vd zgdpbMdW8g*0`5zTRBQdNmJX7gmH@(_^zfG$`IFh5{cTB88A>fJ(Sl+NNmpa)$8tqc zY1;UvTHJ;U7=a%KI-0L^=EDZN_Pb|c47R7rq@ZkG!t0kf0yHLc?X*p_6#H&_dGkOk z#ytxoix0uXt`BrtqvtGqa&Y#^Xk9I%qUI5anqxt+jif~}rf%x~*6#6#*@oPdX7Xn{ z-7`HEPrpk>w7GD9yOoL591HRhV69kw&B>q2^+w0!{@{)7XjB~j!o(_C|8=V=a4|A~u&nx6omR1Zwb|Oeq2oShW%XAW{@uIP@?EN{>Q36}7pnxt zM8}P>L09L&l6U5bzVpAOSDmZw2#SdvQ(Jl(?&H)X*6V@5msU*e)GPOR9Q^~i_8ofJ zDuChHoDd8qG=o6|h^MovAaZL*dY1f#Iv5bs7*GrB*<>x=k&D*I!enOmE5VTnN{FI=G6woZ%wkVg4jwp5DFrMw6}huq&}ZD+;DlBCZ+Oc)<9|S%WFS zISP;kNRsKc@6V$EFRn8#KJP?cuhZIY)(!w*sN8v9*Kt2~Z zf~y0IX`rI=*UZRkE-j>1)MwQo-!(K*`JQpysW!H44M#z2TFZ0ez66qNs$Y_8p<`#M z82BVb<=^*OKa1Nf7Zd5 zPiN-vZCT8DKnT&XN>r0Rhx1k?VAKNw!r(e@z`cT=V$@L)RfMStIHu-=!TlC|sP~!F zqqwQYad}zQrsTGXG?vXu!NZs9hD+ymYw8ZY&Pl1&&e-b7{LYgZMWq0HW~pTY%VY>W zoM~x%G$Q4vdd_+sU12Z zcsfAPRXG|}X9B{Zy!6W<%bRkkKsZ?Xf6AfoXE7u=?gF0f=I-s{+yx{0)Qk zswqJoDg<0S9u!DMcGY3%_Ao=YVj_->7nQwM>j&? zw3n9ezX2AIP`JVGNqEFMBfcLrUoUp!#RMN7a^@pJIs@h~13^KwQ zGCy7RUI|@~f}r{KUztKvO``!z9?W__Us?0z`(xNlZTvlW??EKWbXT5eQ0kE zhlA4Enfegka8!BUWTw^DZ+M#*NB+D^LS)#_D%P_LhmZsfxANXIf^bkR?4etEJh<(* zj%A~U_gXvt*;&NZ^vgwlW~x8vVgZs}*3Q7kKgV`fhs zU6%Tke|6L6-DxJ7ih7QI-)hMBM^|#!)YYyk8{z|bVXI6a$`Zy}OJqMLL-=MSkxU|( z?<_8wGbg0;{43*7&Woo$@4!7Qm#a!fxh5ze!Db?;hshi@#q2DgJx6eb0sqcDyqX z9s?aI7wv#K&+o=Aug*31{g;OF!$(`>!4}^j0IL%&7#VPG8uBE1<&9fi?_n~&Ul_P8 zw#F!{dMR722qF0QCCHh|h*(SKZr7+Ba?2T^2{5(osG2OpFa3x^oJ1m*heiwQLlal_ zDcVr42>8?35UU-og6_p1F#m&Y!xk+C!${XlzZW2Z8}9F_tZ@izWq^%UY?csz*VsRk zO;O8+k9|>(wpxbMU;b&H^&2PTG&0&SY(=+Lub>CYV6GwYlD5rT`^p>D=QMfHj7gC# z{SKGtSAJ9}Ziwh^n+J0r(WSt~$IZ*Lyfm4ZqG*+1e02 zh4n$*b>1(90}5~MSLPrZKm`am_l-@?h8JIsAI)-NI;Gg6TBQU4mpc@Jg^t#O%r#cc zLRHE9DLn+v+#;$kp|jU1Ay8kj?MsQt)mV89!wI<*ezMs&8gvv!mftWdKs`-%Ygf7I z0`OCg93WFTe;y+&BAheufCl_&ospr@z#QKr@V(t%je{G~_P6>o&#D)MEt=b%JM`96 z@DyKaAbrp%o$|Zx@GJ&m-Is#l@#Dkfk?Wy1bmOG#^%B}Uoj@L4#RnsoC5&KOBGLof z?GuSoTRKFd&&qRRSiIlE|3vKP0PGl>l-iKL3@wpC4_Ax-AZ!^MF)}-i_R&U6I$eXr z{C}?(@xhV0cUreu3;c$*`RQ}ouiWYj9;ae}UjE+12*LB| zSKT9j!*kXkol)X-`tsTZxbsmjE+ll;I&IV&UX4S)mTTvRZGPRk$Tzl+G**L^##Y06K({HT|@_^VId@PO`XYqTuhHzjK}VavwDA+7dz!{gvctUY>-0*61p zPhMtU*hkeYmDvGhs(WH>g!4_6#okQi?}S5gmh#4g*qASBr}tn^~UZ38~5nXYqLv-=Jg4Gsj7L|qqNe~TaDErO`t)4D`WJGkX z4oS%tm$urMBp=tOE|uD?#P>^Gz|U$~ZY$Bdf~^jjBL}d|6E$eQAK7sJp?Np867$6! zKfT}TP6v9IHhucY7Xobv)=y;4)wHOApYHJ*$yx$ZM#0S(7u82LG>7{x1-z# z5w%Wp5R!5npX`S~gw>D66FB~kMd-gHVnA}4QRcO}psMs281Cbz)fN&&h1Xs2@X)8# zHZe6_AhBYC#)|cv@*BrPjK!O^z*%-ECD_CQ#jP(=#V4Z!QsImeBU68g-%z1rIQyV3 zm=s)yE)`j+>TPv?CFlc~_PKM`wOMhiAol3iLvIZckVgHHi}#urvWW=_YcFZpj66)p zC7!n3HufwYO5HIRN$9+2XY77&kX^8Sj45kXU{6BD2s!mJL7gBg)*x46JH8dc zNtNaf$kN*CCkD5+YyiW#zYXqvcB6n0Gh-57_Jk}tkMa^}p-Sa#dpzN76xI`|K&zbu zD?plKQY}7zuv;%Xn(fYwgV@n@LRT6UMiFy7o=Q8FW!1*yWLU|NxX?e7eO~SPRPrJv zH|4l~AV+zQIkF!=FrimmW4(8e(hXT9d__>4cy7Ov|A`+zu^pF&*L6ubvn$nAoLaO5VWOd%NEnyym8;Urj>`~ZxLyVB z;%3#d;}wK>7 z2XK%3M8)!3&x?-pjXeJNG4=gL!S|ql_()tm5o}ZHbtpOOx8E+TZk{VR=>7c9C4Ct? zbGm)fDSIA2%3*) zxD3p-V*GEp#kV$x049Uq+ALF&XJI(GSF8VZhk^OaPQwNec=%yi6-CqsDU2|?XC6`a zmW#va(^g}fO**fJgkj&Et$B&6%^~auSWv0bR-YON*OFQNvR-44U+swtfdhsF!fqRR zD*yKKKhI^iU*ur^A>JQ>=)eN1IQuGY-S}+u#Q$8i)NE%8UzFBLp*7e?_MW6?!{;Jl z^rHV;uSTyRIS4I6j0>MhjJVk3DT_wekW_pG7`(<`B2KukofpCIF_bEXbOpcRn=do;Q3v-r61*1^BW-y8Wj>`nDlIsKr&(LNODvd zTiHQpK9U%6q4=EVu~uXfGBk$j`;jMP?fTt;y;#_tNsdCew@FSv^ex8a@vFyTJ-Fc@ zf+|KC?(B0jBi;`$K4>dz=mz|}F<*;!i=_X4n}sLD0Piu5@e6M{$yi=8-nVPfh4M+M zic(^P_EGqLUwDh1W+UT7FTS+h>S0QPn33Bc`S+##^F-FOaYrG}=>fd9KYZi1%63Q~ zJKtPZfk~app(Aw?NQ72CI}gvuH=QACMvbzKPLl-^hIvYVr}O7-B_}`bE z6M!{dIKRb>Ffg(U4K;9B3qp|F$arugw6EQaG=^H5{_}DHEmC0}sq0Y=+9`y)b|QIw z74pXbVychwy|Z-tKmX-}!5>zrpE$x5u~w*y+YQypw4!ob{&2Kz)`_bLVFyRNtY+|k zrTf30S$Rbo~AK_<=iwJTXxh*blT}XYtjFc95XT&J@?b}0cWg0grDjy$^ ztXYoeMXy!*;*IeqbUV8ORs2T*SSGKT1Kis#et0b}b#%bV$H>PsjPOHLKe4+~V*M85 zc}4|_|DBP;jW!G2fQpw$q?z1TSAfChy{)&K{c?$-v z#0EH>fEFj#5eEx-21pw%Gjpu6Dw<(6em`qp`w~I7|4Pd+a`p?G#9XsFMj|(y2w}u# zfjsxN{^I`}#hVk4zUgM85R>59N;G%Z7qoRLVP8-22W1aC$W{tla}`JhTbiyh4howv zu;=C1tlq|&VF4P`xVlZ8`8(5j^6Dr+xOy;^klBEm1W(Gvd-_tW6N z#@q*k0r6%gsG5A%YFn98+y2oX>V%Bn4Ndg0uynk!bOM1tK|pQ_*^UgJC_j z$}hY4_M8*1Q=Z3$i^;k=wr*BEK0E}VVv@3+BCraQ?rZ!1d#-&S6hb$oxcdZ!krWjo zZ#iL$T1%o;M?;HG;6!7xTgM}XJy|7X-R0MX{$tlNy8_fTZpM)RO>xKw?;&~>GXSE; z`#bai>v01{`%MSH6yo=JpVTxx86)&?J0|3&K*m`W)(R@Q$O79S`Wyb@!+OJt{yK`j zewW4WOCn-XPH!=Wmblu7a1v)c`5!iN8Lgb{znu>)Z%u5JYKQ=MppbKAmZE13X;Jy# zJ`!-s#2dz!R4b&l{L=A{k)Yj{qKPeEEyABw`<+vQO~F3> zWbCollT@AJDrsh4XCk&-Tna5XxN2d-+l1tYX{6KuqB4Ky1bj_|0o(o>6XNbtWy>(> z=@kvmeDCOZU8L-@zVEuquQ>;dYxx0za1cP?8&tn%mTs)_J0ilkRm}c3(`&lF&0H2R z!pSm438C89v9O~Ia{b^ZDNSk_nnnL5N^xjFmjJa)G-Op>T%B6B9neq|Snow`#(bI0 ze3A0AC88hPkKa)F$GmJvQ}%k+jR6x6gPW zqTefwX&fSj&cRr#o-U&4pFjc#qO!BPH>(HYv*na@7QFBYV~aVR-_ zfMVjFtU&g6Ht{zz;W)3x3x)A>GGE4K4Ygc5pFbuAsI>rD7s)iZe{^Q9c($6J&^1t@ zv?D2-1xC+93XAE=oi^yoG?MITz%1WfjV$Tjj67>6EfG!$1?bAZM5;&V-^S)@z9eEB z&TSGSwFTF*bHT@zwU9y0@zstVL#47wbGMS5ArS5&$!eDYgfa+_);vudP_JxyZj59a zM%i+^!Biw8C!>7PWVl6bi{!jkP*y1QH0IVYU+GtM!@3_Qf3ymrp)q~B zD4MHR;IcviOJa(#+sGA%0nkwASP9z#B7SR?`za(M-jwlQjrRFIWWdI|)pmIrv&@{T zDJdi@-<^yJibLNdX&mcPaRRw-+`rUek?~qkNYRkwmzwe@jZb9n{!GfV-m6dX>i~I+ za6uN(Ue#U~G4ZYoJ%#7YGp0_YPzXKNF&8gp?tYT29)ZzFoun*6lo&A4zJaY)AI(y@ z4QQP{l!~YC=8?QY!D5Diy6Yh*>rz*34hkr&-nuu?D|5mN8$4*<8GFQ=+yLC4e*SdD zj&3B;*>%i#L!51*2h9K?mkg*c()S13yn2F#`VnFzuz^RZ+A(79eePMGfIt^eE>@#m z8fw-hV$Tn(S>>%F5VxBU{vT6s84$kF+VrY~&|F|~-1NRoBjfSB)tsNreI zmpwj{y826ZH~37_MfS1GNDXl7+t?^^*0v2x&8m7|ly{u5)yW}*29!#95pw^rIE$*S zvjZZT@zoOvo-)tl8VVCo0PH?R#TtdCj$>1g83 zZG=yiDX`$F>;BD7)Rfl{g&rR?gpB@+V|p z^r54ph#@YtM|-nqsA%tEpC5J}kW)7jXoZDiNY-MBdKt9<-G?Zn(wSsd8C({5uEiLq zc{}zgs`*|T!D)(YCGjjOS}uO(3Zf1UZE8#{3T;vOet!FU;uCE>Mv z&S2w#twZ^L`-XlyyiB={TbBVgF^ixW8{Ze&6H(24ga$3?*9817Nql%r0z~0E5_^W! zT&bG4Uii(%2(68XxI9XI*jA;p@mgmk@+A8GQut9(0T!N=IEm zac1gl-$--jtE>oE?1)atxjMeO|DfDvJA~(^EJF{!X{=T3*$$?lF?OWiTxc7*W42WT z+QAj@Np3P9)TUvRnFXYS(6xsrqu#$C5OYsoY(MRsAu4@)?h&t0qEwi9>|n|@Ps|_; zmw3pqy4!vK&=>NaOPcm7OR-*dAuoRPrd#XOhH!vs6F8vglM^htpg+>KerTup&XIP( zjTRz$eRa3k75Jk;ZYFOE(33D+20-AUuRCoYaJ&!>B_xk;WO6CiVgEmSkuAdvjNmZ3 zrQ0!8Sni1g4KC)^;%K^QhI`J5$&6te0OW(R*-f}(T2Y#Mox!Q%FOmz85Z?D;cvUNl zWxdFwQ8TR0;hHmK1+_|y>XuYE+vT&Q{PX+~$@Ry(ELGc5&5|DoU$C?%u$T#JRSuM0 zkK(^*wsr8~Z|4EDnJWBiyQMPRY(`) zUy{AUAm;DY(K{N^Jl0J~DZZ2DRLE}&2_dD1Ftt;kcnL%O=6W|n+b_Qr{)B>(hCQq{ zSqX3LHv0pC!#SW!(VEuUrh58ES3Z>p@V@e^Fx5CjilK!56(c2x|3v2hEW4i;zPX51 z5D$X~s0*4_+nriSeF-MeaYb(_?;Rj2@oO^6_-dkIZ(jR%c;G@Qnd1O(^N@v;<6Y6Lp+n+{%ysMG4_awHXI^vO&YT7`-(2(aQ zEo*P7L`>~7*kSzl2>dvB>q$m7`*^)pCRnp+blrc*;Z^BjM(mpoCGp2ZXV!t*e_E3j zP=Cm&&%@v&6M7%HZUA@g93u{J28 z{PobjM!Vvj%_(A-MKk2B(2i~!V0we$J_WwDaE#p{EB@I~I-3s7AbSjpt2C1Y$DI5o zjFk&@_)Hsg>LkV0z;u;ihvmj6Ol%-me_+7^ii_s>V9S8nnEk;w^(znHUjE0s21ufo zKyU_l9KiV?X5=;fzcV1T^PKF8cmAfh$p8voNo&#TxbzE50azW768Xl~qU;B-;K2hk z0b71f-#J1#uOdyItQiyTB8f~8FMx#<4_fyB*8lBfe?c?QO6szgUt=1sBDA< z(J}#QopiCS46#*Eaqv+FUDD;u=>Xo&p#B0!eG!I9e$|-LmIa%m-(!&Q-_&=-YtWNH zWouc|U)*qGtY~MEnSPZ3lY8R7S^YuY15iw83ltly}%P}+O z1o7ofS4=n!UatqY%1#=zh}Cy&K8MJz{{XhB_}iI6#O1;G4IK4V;cps(-7Y+8%i4~! zfLsu;6S!~-i+#=}&b0b>!%w(>%NaJpA>g=YRXCu--cG&bDO*nxGW75lXAa3NjY-7_ zDQX`sN{y{v%z#Z()3~I5hKy(|CnsBl#R?DtJ&;2e5rXD)3Pw9fl~Y|Q!~Y;Dl7*Ih z{#OubpYY6RU-H21Ep5d%1!nVc(l@1@`9$mCz;~m+6m9?!jY`HGm~gSNxeS``dL~Pv zlET_X04XpjoP@#2f+_@kff5OavtH{L0A1tg2Q-G%SLdT9=Wb%j*xmmp>k?7~5OKDw zXaMY0x_}ZK*1`G0n^#Pmfi_Y@XHl<5d`N}Y&kqmw0h+r;eit`&(OBo$C;p_jt`CvX z0~E)pncPNdey3nGY7~;p+A|WsJ!=MAf=bjQl!{k@SDH_0)ebinD|0S8*~rvWxOeVmwjy*GK5hP>`LF{}xn zwE%XL<-07h{sF^ORBQK~TKS1bzVK_E7}22@m|H3qS1y*2qbiR(x7C2637EsSFaKK# zwn%B(4x%!jY@C-P0f~*7H!JapHg_!GwL4BsolBFx_KR#r5uCzR57}9K?-AVA5cmMv z4^A(uiGcu{O}*@A%u2BccO~gdsJqhL@xk|%6B9BU%WsKlJ2M#pMPi^uk;_}VSVOfP z$qhevhOq=%l@jPHv?$2Mruw6kQZaFf@=I9fvzsF_`bx2=auaSw6zQ>#ZtIhr$EW?e zQBQ8vkz@om^|I%s8MC|k$NBr9RtcC>s&*A`s?&vsucO>z^C{#*WVX_>uRwZfDr;2b zIEBGa%HX1FvcRDb*@`*e7;0@SRPn4vxm6gY=P9B6oX&G6nnE*{gzV(9?Ip&6_sbsK zS_Al~ox2^T|FcO(xsxWUe7eKNKV^#nfw8>u-XOW2#C45CL5wF||2Kt7Giy(Tto&G)AR2LF6=B!;um zrnx3;n0a%Z7G^_VMoL znJZD=p$->Bt+z#`>QhC7wFAC*9=+H4urM z=-Kfphj&u!hIK|p9O$;Qj5ad)@lZuX@d|x&7eLn<@1a`4f()8Zbfs~rGVg|t;;Ko} zn$9M3amukEP+DK2n`0Zv8zC@<^A1v5P3?Z4cHHN!>6-y@Cz7m`xZrG;O%JdkokoVvLC`(QWHSAExXOzqehYDEm}#+Wpq(k8JJ{RZA^4ejjoCtkZ#m zZ5l>{az>ouY`(vY{Tp-RN~=#0;R_`g>||KK`vSClc2?y|#Na zHT&_Kbg2$ZLUKAJF04#DCNWq1_DSa@3XAl8QJfXu7qk9ooE9DJJhYqAWl6!E?jwmtz%86VR0SF%jsDhZR_HRe}bJVzD4Zg?!dz#1z zD4BrLVUCqOD9ZV6r0M&m);8K0(lu^;O9(Q?pnpoxEJ?6Xrw`j?#6VYO8 zgLCZynVv+Q$#VRzzq8uuOosdX0A@g@A4u)cm@T94c4$>DQICJ?7ML5@!}D__3xlTo zwqW@$I-T(hWnhUVqWgmKs4XT8z^@g433wNs6e(EmgI``ajZ?;Llt|A#>iXtEyYSY}3PoG8OOcAqv{AM>NKaV7kZN+xGfxKlo{Nbzg-CX@h!9t|m zz5p;IyvO-Z-6rn#=k2YsFI5y+-bPD}Ges$(&%PdBFO8J)y(LN}(Rg@x=?RVSZb{XD z<5`qyL*qehvBBd`lw@BGiyMn!5z4oV5z|{#e%a5I^JE0ok?FxF9J{QS_pi-Ss%e-C}jefGIYe2$A)&8Kca z!BMi-Q^oG=R2tmK05h~(%A5AiAyMo~sSGPKT^xtS!FHfR2D7Sg9vGhvP1}- zY7l4etdCF$4WZQ2c&u70qS5^k^77~PTTdGiH%~t_wV@4h8vqx1|pmwk)cY0 zs@bUuk>BytPKUK7`+eyb_~x@CZ0;q`R*GSWnk|w#f`AGn5R10LyT!lZ#usfWuG;@j z(Cmg#w|LoG22+Yzlmk~~{9>(-&A%2~I8z_3U|#hoS8czzh9}Eck`iuXT`3G(WW9 z1;t1!vsX2;{u(jWUj|ax25U<9c{Dn$Ky5mQO#+|OcwoNfVwwEjafRRKsG6bIsLq^+ z)wtV9i@s-#!{k7)mI&d{*+TJL+I1{D6?>3we|q)j1KIDogLtdW_P;_fi6o5QeK#wS z`F+2o$pyLzMa3mIy&UR*7ty0;>PDWQOaaslXk^M>DE8sy`I^bAjay0VTw4LltAkR> z$d9kxzn8vIwK|N zWwWqgC7NFj^BqcYgbdjZk`ZRqSGbd4ZfkLid2nUL`)l3+y^X)y;$g;;8u#>zs#2mX z16kX#M(HT|j?R^ciircs8^Pmv60DJ^ClkfA5S3)(*?DT8<5gsurL)gP;ym((*bTpL z#}FC#=z9LfnGMog^kpF~7qy7Zdr)fe@XOT`@V5O;i>{|IC1f{FUdYy))J{^KI(f^K z%WAjg;c?qXcaO>aVK~V4BbiQ#!6nUN9k)~75eIvl_I8BA44^5Oj?=iDJQLD8^2Vc` z7SkdsKWnjA4&b(>?}PAWn0l-!+4ZNI^y#?cqXAZzlHngwH-e?VnO+doD~I(z=}m{d zf<-xReq)AJz{duW~! zMxx1JrhvQxgin!9quZgH7;*b+c^P90Hho=MtP+9e5e?u@Op4U?O*Lou-s6H9#)Z|>75Yks0VHaSe|uQ1 zdl=M<6-?{S!G}#lal5|(-)P4UD$%{xw|{stpamoTXmEaeZss`eNs0xJ4QeiHiK0oM zyWt18cR&l|RPLMiM#j-fzm6O_>%r06O-Jd!52ho5O7sIZ2AI5DxCFe^j8NOQRJ0VB z;=4)I`1mlgqJP}kNwYv-FjbceXB5^G_?^SKi+n&Q+GJ_Wh`YCCn{H#?wA!Wur0bDZO>*AMqK{Bfg?GO=L-?-H!wFdXf14P`xles!tltv zz23s{j-XP!*XkSsbJV0UD?v-qG>K1lJXC`TI&B?091TivR#DS>0JhwfB94J-s#XcOxL~vkG!G_$RUy z{<9kixoJF8$h9eUN z|3##*1_Ya}s%x3@-*E&z-j2P_*9d1kuVlNEaeH|-RhQ4WB{J{V%S({ZtYBq3KJF0w zcMAYQ2#C1J0rXxV{X^K1x_bRea6Cuv7VjSg!MemOm)e8vw8spbi(hkZI{A=ZI0g?r z7?TlfU+qG(z;Wc|e-w~kvr1SEy9ob07j-NrMr|>)r-A1D-d7Xgv!hBl5jMMd#?b{6 z{`Xu!)8-$}sdRij7@V7^%1ikA#aA)!0@auPL<4&h9S`xImU=WGaQ)Ps26WKv#-+6F z*F2Bt43z_N*Z))1^kPx_-yfly-6hrsZyXg=Z9m|)lqZmvjD&YVnhv$N761D$rYUR^ zztFn2)FR$RW*_To!h4fGh~(fP-%Ha=Gg>V8!eQf{r<{_t9xcKfWrXFwuj~Q{(TiXK z#hQOepm%kK=a2u`DDlq1zUEL!9+*@VfcT;q(mVC3QozbZ7Kh}SkM&D=dk#0 z^rLq5sPUT}{k2HsEfvQ$96l)%zyJOMAY8P#hp> zQ&Z2U8tbnA78+S?XL94H((BB2fyF0w_46yXepUK@B?*ns*lm4xnH0D(HeB(0h0OpM zANxQgO_$f4V?SX~gEJn}K5;3ZQb?Y}1(>a#*f9h8CD(@dKSKfFA>*onj*qo2s;wy> zGGtz#LGD79IqoJ)GA1^&9uC_y?^Gqa)P8+6K2DThf&AYjlv`I0>)cLgxK9Eo>L%lF z*)wD~fUh%`?5iguc-og?VRYwEVI#L%-K-5fkNv;rWp&aoZi+hCp`|SSo%lF8|Cd{W zksvSPsAz4G>qEm6>8z_KAcsNHWMrjUy%zZ}X8t-Lkl7_Vq`n4eI0{D>i1gBtOSaMQ zhz;}qGf+-*70S1prb^u;SVEcn`;Fd1LPFVmgQ+6aQa(ukq9R5W#DNYtryw<*Sm4`v zC4p==1qcmdf2v{J+~F*$nbH0>>Hq%$Ho9{ciW=C<7$xs5pFKre;+Hl zOZtG-sSQG zHX8+l7Iqs$aU2!6FPg8lxSC%x!$7CT@5{3pG~KzZ7~IXlPton3{~Mxn`ZI*saE59< zz#Y52_0rzM<=A4L@?Fx>Uc7{kC)-x)D>`)KExTT!^-0fp%M1X@#Bc|edPOk`d9KZE z(!Ut9F&7oR*|Orcx3|bLdKx3%=W4yTqO0ws=Z;T21*^=qajy#y|1&FZP^khV5cUo1 zlG&ILPiNI{AD1$xNBld%UFKZKIwD#Scw)2({SC?!RNWqMfeu|9Z2V2nRvr$=d_Z4n z;Ef}8Cl**D5QgZvq50gCSDiTkh1)Ift7D*r@~e|MkZvj&J^_^ z%oO(+fIH!O{bH=F_1Bx-x$1QDnc?)fk}>jsLLm_65>U{8gUISMnxR<#8Mn}o`M_ec zrVz^Fn@SAKCPe69MC$VWE?+|IJ4+pgsO4f|v7RNS&!?9ETRx{`xw7JNYidEkV!Kt_ zlN}x#PEULZ4S*0;5734i@O%dEh!}@rmx3Zq!mI9di5Ui_vzwld@qaqk|MxMHSF;(G z2f0GfCcOv#27CAIaWGmvzMI zVXb+mY$6m4Y>2oWN^{#K#~OXg|J~!t(F`FdzBm=eKI2RwH)-hgZwUhe0Rzb(Rr_Mp z!|_B?rUIkxSa}yv7T9-IOrb^*7Jx-bd~)lu(e_%1-K*_S5C+`(GZnH6UMl{B9Mp-1 z{>!Slh@@LAG-vLu`iige@}NLKOJ4y5xi&b7Kw9bbF_AxC)EBk}o)6hnKb$!!e<;sm z!PB;2P5ukhR%w!`e=o$--Z2q{ffK+hHVM@`U$WG&)qJt|tltD`G58~|svO?c)pb^& zYk1`qdXDa_gIk^YPJT%-0ul?EK+dHeezlfdr8ittf;C!_dOCJ9Z+=bp_g;o)5GcJd z>1S}o&HQ<8A3Ho6OfYoQ9EWxf2B3zEJ8Y%uy^JKGjMS};Xhy7~=0JAP#n@H&Qt z)!m73%T6yQzm%$@TmNC7++)b>c*Ptl3<=TyEB$APrQCr(`Wz69Hrm4jJKrjR(pk>~3+m7CaB<*WW+>-{A`qOQ(d!kAjW3lqy zkh`3JM8w#dYjKh1^XCbh?tJmnvLJ~!kds4E6B`uuw(Bzt+YydSPj4k)0lA&_5`HjI zSg%z@fm;PUn3#XEnw~ftGjD{h4ivo=$ad%Z`s}91QAbQ@pnLk=raE-Ik<+8CJffjE zBv23++A&cYF4K8bd|YfSe7D~|f_L{6dHUJq>!*%m8^U9N35`ZoVY2C^&u@oZJh8eE z==+>?-i+Tiw{d5E&6gKK%NzBlp6)V>85UUTSK6Ug!WzzRnW71*j!c=x1~_u+hOb!D z>G>3+HQ!n%p(xc!SDGYjX7~&xYb>Zb(}7-=^#^QJCN~WgL+)3FQ-v`gLHG7JTM^w_ zctMmv%9A{oAkX(xwgF z;MEo+E`Hh)B>o0c+7|sYg8q}mFnd%3e$4CIlI-fjjGw-@9aw!MA@rDr%|8ZVhaiUCIHi8w9m=&N&rxxEJ+ksp!b!mlA0K?arOyDJNv# z9d^1&9D?WY+M)o5C0)GU`PPkB=*|s+DLGD8*A`FfgrC*lfbdb>vRkTdd_-&26~$|KM1 zAxKVrS6nFJTm5CO^l#I{lbH*NEjehl)0^w)rKH^fgQ|>UK2M;)ATOIC@VRhXjUdL; zZHq{~u`GUN6>GvSGGneLR_}TQ9>(jgmAJOSIV-0b^QIJeS$bpu&9@oF+1_?b5{Erq za-bSk22fmxie{fbPgM$N`94}iPQ)}R0O!hny^fjDPX5+NNbRFXq^WR>Pf#rL?oLBUQ$+bxSxV6DfRrS z5}TB-fd@U8x4*3j%lg3j zlQ%y9l5C@-bKfkI;zU-*9w^q^V5sb#COM3S(gG}gEMBM(}ml3*)qUf%vb(eXJ@|@7O za&5SNIr!mODeIg>VU}tMbf@rJ+s$XKiWN{P^!3DL0UxR3jI97>N(!{VsKHE%iH37W zO5;lSRAMP=Z%^yi+_IInymx$^tJCS|;&KG(9W>=1#y#pv&%pv*JeE~^5a4Q)U)OLU zP5DVz`#1QzhoAyZuNqZ8Gb}dd3)FKnG}XqrAQopy%!e$PR2dUe>H_?27ZqgLre-Q)?K2JK*|q2tI58F=<4E+vPofJRhF+PIg3 z_hCO!1}YENJE35Fp<4Mnb^5F~oOJHP)@?2sx1D0fw z;qETLPx2W)O6cXhgI(JX3#2W$Fy-Hz`?jvD)lFBhp?ga(?7-K04CO=Z4RC`elsFrD z=1dvf!|Q{^@dh(FrNx+R*Yu{=W{UF;E$+3}{6z*fHjI9rLa1oy8_7W;qe-2EpHehE zk6jeJ#CCbNd#}Y=sDO=pd}xWsrvw1r)5+DnNb)lvvR=DnA~fnjHxBHhse1E*>RWN3 z%NQ2gp>Ro*Hu~eteREK^lxzk6m>l#;{ZO(pR>52O2DVvEwuNH;$=8*2xZ&xN{B`uF@d>CwCVYxP`oQPLHy; z!R0vS3utB9SMUh%-!I|mzD>L1+E=`%vc^W~tSh-adwi+}Lp4AQ^WDx715h(lY&+>B z%jrd`0$L&2(5UFY*-lYD@!mi22l7P^V6k)XTBo3B%MR{sUj|2>mTj}HXpiQy5Z^s& zQ%^B|`1<*<$A>6wQEq z1+{7a(x&u7N!%xRwdD}T_JLt1n@7L7IK>#F zK7F*GT!PODQFEnX&Ooj+>~sI;os)UZ__-FxtQ8Fy%K>W~WW)O_PG<%4vkK`@)!Mmo zQU$j`#F)6E!eWi+$q5{{4SOV?cl`_k<(*Tdlt9HQa%95^x{(wRqlxymmYJ^HQ*pO! ze6B^tzGgUz-9}wghe}eO&p#Ac$%rU(BSu3+|7qwzPEMX1YJNElQHC&a9uR-vGz|i$ zpuN5umG@efTM<9j7rm3kmvV1EZ%YrFN~K!>%Uha(*y%|xSMwlD?9{)tF&_!k&S4r; z6-r{JPY50$iL*DRd+~a6ZAlkra!^rCc5qcCz`HZzW;GAxb~>DLm@F`AIZg=XP!5B&U0_nAD;;gK{_7OvUM5A(h1PvH8L_ zWoHJ|BL8t+^`m1cZq~>^o&>hKMA$^*MvMsQRaBZzm0(T+iHRX_y-%!7HnK|QOln73 zh?L#Ve!O?U&7DgE+zOb0h(KIM4dLY1Y;nWE;R^e;8$UJZdd(cZx>?rtsR-8^HcvRN7xlJpKopZskDk!#h zZkUVBFpDBri*b@f&B{KDpF1)4@Y+Xc_9xyJ-_UsaB1B~`(Zi@;b)unRvZhv^xKPRr zBosTxxY`iP8kuR~3-eJp>dAf&a-AcV9XQtzTTJ$Aaz>@b(q1U?*VEg%YwQ0h=M{97%OKL9+`}=^*8|{L;>}{HAL**czDPI~f*Y z?gQ}G#6HCNYAOq(T$5>XGhjxtW^N-ptSdw0)5ce$#DR&pkO2JPhTZF24Pwe8(p+ha z-L!5kO|WBnOF*GDV7ShRXP~%V*f2!bM-5EaNxUR(Vg%QlxR$}h-YOh|uf${B=H~bP zNatGW)j>Wghg~m$eHTv@Y@bxlP78}>_szIYlD~bK? z2n#D>JQ&mT19;sA1>%>tK#**Du!r@TB^@pTxK_o{qEWMJ`B0FylP1>7dl5(Gf7<_Z z_jGd&H!uW@HO6l$s8@&E?u2;KmnjanYK$hz66xNT;6<9R;FO5a{Hs?Lb z@$iYe&nBRP-XKh@tp&FWGP*tS-+5}^wPsW{&9rJKwKlIu>ubuOSoIu>JpA=!t%%>3 z(9vOh^XBiBOOJ>@9zOmLx6c&=z2}h;e}nx8&E3LETp#3@Q>=rKU3*>!Dn9cEyZ5{? zX7~6)(U+sbx51Shx;8de557jX62lAnQ>Jh-VR;6zgOgKEULC0JdK5UI33`D+X-!IB zBx(C{`l|B^B0t&2V51h((gFfpcyd2na3*Z06?A4wcv+?E=vRbP~fd8PFCOx1K6g7S9OP zZkJ}*S8imh$90U>qS?96#z)aTs(XAZZaJ%tbW?~e2AWWbTGK+OIPBlLQz0b5Lsba; z?po=E;RVNkXsIMnVv97jvhrV=Da+u^SQ7M>YYTdF5V4ATjR8M-VkH_^Kv34s?QpO0 znN|Of=4vhAyxTDA9B((LpWJ*KV0!tjMU!|~F^VT2vh|>UdO;@(R_Ju)x8-7OE1`WE zr5kwq<*^(w-q+XCglxJ5AD>7f*;Bv+UkEpds3xFe+%w+m)jSCt)qjw4CK*lO6bth$~2W~IW)4VCy%!fu3w zv1+pb()_6JiujBeh2q3h`PY=-rq_65gx>AO4txk^AkKSF#?pbz<~X`{LOJcqw#CCa z!H8-Rr25(Imi1%D_C`Ib=FmcLZLP%#+ot%uNlW-{mr=EFv;!|rOtSY%g)+sq-ZPs>AG zPxj%kwCDwTFM(2;%}M?|vKME)@)565dA(znijTGJw1|n})s2Pj=a65&Qyb%J2ouub zDNm2-U!4~hHlNil3EMiYFNd(EMzn8(v)X|f%1zg{U3O^>anU|S;{z*3fz7rq5b!;^ zmw-AnKrm^Uh({havC`RQ(=3!Xq|%*(e$3?!t@<{p)MtTuve1kkuj!7bTil?r1ifW} z@;b<6x|1b$LXQ?*hHE~_51U^~>X59-W(5xYJT)Z&F( zUkXGn<_z zRB<7qu)@b9Vkcg6{tC=c)4e2$^C%>LuXNK-It>Fz;;rsd~X%tRAVYWeGGbQYN0T`w18mj zHFf*d+8HBbzS?%97}PSxbskMm${6dmi8|Pph`C&M!mzl#c9r@Rw%@z2SOjQ%JzCxvATZ{RI54*EKl1nsMvo#Tf0nR5$g)ACvNzXoNFSC5@9v8oMNT%{_Zd$g|jV#Bm* zz%M!>S7Lf_3_0qLE!n7`GDr5XAjiof{O^!t+kQjMGQmR;YRqdmTxV-FsS`_0Sc z~hZKz|4DkIY& z5ohLph4TFQq&N@%8g-}BU}z+aS?e2~@%()@h-N|f4fk0GA?WVDI%Xj9;awR0-GdF_ zk1I%b>X<|qk8Q++7sPn!B89>WC|hUuJYV;1v+y2&O|QiO_y-g)X*LpZ2max)r!uP# zVIT+^9UdP#)dnRVQ{Rh5_hJss$)k=ceyxuD2uwDQsk`T@?#v43h}!b(ULsQOyv|V~ z^6(B(0`du&RV|Cf?dZ!ySmoexzZHoY#8am>a@P0MXz6tL9#hb}BPeU9(;H6RHDefq z8xU<%?8(uP3FiA@*Y2^soDbJ5P>L2Phg8w(l)F9rFnDMN*j;IcXI(^F5&5Xj*+IP& z-5aa5i+F14itjZ{G%Kp`#=A{F=GGk4o7!~H-V*~%NX+VskshxVaf;7vgcvJ#GvMgt z3*rIl7`i=3R8E=p`YI9Ic_@K6NCJdLAfb?S^2Xz&7SKTx`4-d1hooyb(qOkoL06rkUG&>fPGpre?A|W|xmWnEc?}NEXSh0OlP2n(M zP!JwL@Z7$5-KdaIh=|)SL?bxXAZ!(oQ=xNIc7NNhh>wZu3F5b_Ri+N>IH>72&_#|g z-{A7=Bt+`PE!m_6!~11$XnXo^uL6n!8%Wrw{U-*Q)z}XOvzBETQ;58i@OaECWlG0` zutrQ!d_nH9&sMo2dXa6C-XC_w8MjPLzn9~V>+o6WPnIkFdfnp6fqE1CY|7+DnNVEI zJF_|r`586S{Js=~*Db(EiS=*gz6a0;gc`WXoP+vLD!147z?r}_<`3~9D6i>^B4%~# zLlo{%r}k-JfRlJx-sq%)aYb#YzNNf8LbOPvk`hUTamCWJc!8%v4+HBidsfL@|G-3V zio{;6&5FsWQ<5N3b?fvEgo)$DfgF+89d!h?d3C7j4MdZ-dFqW5jQi?oqs-FjdM0@A z0SjYl@@(+!Zyq~azS9psO008i1}pAu#z*d~FNI%i|EoUZ1JJj=?_#sB65GMRWqxAG zV^RuAlT~pLDHZ$RJ0h}w47}i#F=is&;MCdf4>|s1Y+;J4BKn$56TOG~ix{S9$>_|w zG-E59US9LgBm`tIZ5_U+(A1X^s!^QIwvp0SZnuMJL}{#17#gPT z8BqgIDu$Z+$Yp2!rdSY_@w*9a#s-FekT%3SNbo%&BZ0hSm}Afq^qSbEYzml4Zf;LyJ-~@EVJy8ZnCO@Zh z)-J8gac@-$D*g1QxLICvhQ*v>W?aKe`d8ERZ}R~SS1nfi5~XQb>jbpTvf@b=s}k#! zW%CsBQ4aI#V~^|gi8>e{Hi6eUJ#3jO(SVtMM|e;x*DHep*%rV2_CKVB^m~MzQ7g%= zGf2cnz{Gkk*DwBnV$M`XSyStY62A1mBOxHdrArzv(%s$N zU6=0e&hPj<&pY1t`vr`_IOptr_TFc$x#paUiTB4NI`@dpETjiToSWE8=_q~6^0lD^ z4RNS(I}Pz~-C?N|NuHaPoiLjuf7ZEY+r~;Fv`$`-TJ!C%R#S)4C`dT%?l)i1R+O$lKc(Eo)dA@JYIN zMBF^gIhz@-;+vtIvw@pupQAcGlNk%KUF;dg1PO_xxjd+- zQ5hufT~ME`K+S?=2ezsB@Y5Z8^R}Dm`RXg(p*=F<${<{m-_6dDBuQ~ihj=p5zkkoV z{3N5~!d#}EBkq8Plz;dcT(PBhIGdx6ON4I!0^x8jY8tUdkDjry0u|uf8EkDZOjIJb zS%hrn|I55d9M*RietmF*c8lW4-=txtLF>_)a_mL}J4qnJAxB`@_t~pzIY2(<#+%P* z)p%Vahhk%}Slzo~RktifhGS!(U2RIZ3f zFB2x^0^O|s?`}(xQy%-t0WxxkmYlW5RLd+H)Dw@m4U&G`b1E%XoF4pqHIf0Jbk=QQ zazu+=ei{zC<(7gq&{hpj;1_mCBAQjiGGW;W*LN-Nv%X6HyBsn1WZ~YOX#U?_?LDA* z%>;xRHRUH=+=WN%>$}-mhI>%~!JjNPjHLDXqdQiuEuWiNQs4SU(vA43It!7y=K+vS zbd^KF!}(?H$ZI3cbCvXVIOjKcn`WHHmnNC$6k+`jp4K&P^&`7^(&hOmR!YY93RrUs zSZ*QKIs)G`E0wZkDtfsQC5CbOIp1zL$TVuWnU5ZtMVIZ4tmg$N?6w+UeOQ=pOy0(F zYjZ!oM=Z07nrM~@)nIgV>UV{{99hED8DYK(`AqsO4w`HcL+R(p&B4=^U26*VsG^>#icTZ(O|-$23Vlh#`jEuMD~ zzgW9$Q2z=&fCH=|dlQu9tl}mgP??1?gux6yQg-e9O5!EG)llnf^uf%XZbSh*#1|#o z2^(-0)WKOTlQ(#3|1-nrwBPF%UAb zsPF=wAuzkO?!s7d_c%R6<8@K z$>#5k$wXQqM=_s;Xl1ZWh&97RhGV_9a?rU<=*!(NK7$Lc0g8*`2mXBz<^5V;Z8MPI ztg0DxuEun^Tlh3&3fvj)hpdrOmW-rz3~d(O!C3oI85T2Fy6D&+5M4O+u-MI0%nkYi z##A3gT4x|vAMF=r(Ae<$nFCeK_?2fgJtntHZt9m_hfSW)l?5ROnMn z3WU&Yg;?%QdCt5|{kT|!b)iR}xVI<3_<1gp#!)b^E|h&C6v}1;4x}oMbYqIL*9y*z z*>8vqN0&Mle&9t`0)dtJ3~ z^NeRKtQS$K*)5)h?{dx8fJ&+F5TnPc6ZwwP^>`Qih>uoSF!{s$&-iGHTlbKU3;q^n z(l4S{F^iTv#<5Q88=$^!bVXSds%VTk3hjjh%3$L_De7NVrC<#o8D+F~)8=a}KlYJunrGEL!H+L*Z< zA0c_!yRFWuLa|i_g>7^1ZArevoy&%|+b7%agXd6$`kD)jUiV9UK^sd+uY##iy{FpL z8CyDZ1RD@9V{BZxYl4Op@mQ@car5B6keZzGc*4xL^TtqzGb1%HB9R7({8>n}8)F>B ztE(GMt-$(75^eNwp zitILx^VDef4U@&7;!A<3GAwnjpbz=ci9bb+U$?L&wbHp+_rEJ32z5>5)Bq8k>hh&% z2vgim#c^K*NNVtmU;0VP-|Lp*FA;j*H8ninL`&n?3E}`L@kR4V1A@M}8#qn;=0glDf^n@9nEdzUu}5z0{4U9 zADpoBmeDhw7xF>NrDz`a$sEO@l%B%x*rR2+E++c?7Klz;u56MbtA33lD(KqT2&A8= zPJO(193B5)G)-&a9;Y3X*4;$!EsfTn_pxm^|9ga!3G$_oX2zCiu?W>G5~(UsD9#|= zhUT>P-3LlqwcU$fgk4;2Je+qRXGu_zG)TV1g1n&-Tr--rY73J@tR`xO125LaB^br3 z;x~vV;Xc@W!T}-l!ZTsq$3GCmc2%CDPDuG`PELAZky}$v;Ku?Ep&6|7=JGpdwVy7J zqGbik?3KY_Wita3n@6}RN!Yzxz1u3W{<>rOHbQyK0)7cRkWO-%AX?KWTrDi9&9Wmt z>Nz>uTT48QaRy5Idp06ydOx&Uc#3Z9`kv7=Y^+8@AF=N>6qCkAreq|SI_WYDxWjkAgtV0n_!Pg*Xl)a)Stwu~X;P}mU z-d`V=+di-OaYc-OrmxT|F=304ShGK#u%$W@o<(U3Cea&F3Md^R^8u04?R+h>p@b-ldfE_IR&g!z2eMQv=Girh9bEAAC2H%ZG$mGE!Hapam` zQX*4HESH=X6B_8lI!8eElQW-Bd*eZB z(_f0bJCa^-8@?cGb}`FDgM9o`otZv|tY*`L*iBB%9X;T;=Je@VlP(-l`bMHKjAsPK zXEGPBN|%=LMn^#K6K%jdVzAL;UyE+@>DG+za*6P{H~vfPAkG^;sl=?z1u*$KYU z_z~_7R`Ed7r(|kfpBZ;_`3gZFEaDgJtN(dMFYt%LG-}BP*RS1g%sSwk{3X*L=R?Iu zNwe%-PEMI&RWTU8i6LJ*^j!BOJ*!rNHK)Q zG`coqFiInJtp6dJf(|W#lr(wW2Wqu)5dtICN((G4t;Zl*r;f%cJI7|94z-$r-OIWt zxoE*RKgV^n+}H9rq0ip-sC5-SCK5dXP{)#%cL_Bqi_I2F zMBHu^DaOcy9Cj=o;>Ow-0y6WFbS|`1R1pD5jbk$K0r+kh_yO3rW@aH#L9Iq3;Bg2X$pDmSM;!qre}0Y_t}S!47{~MkcXLPhfwM?=?)qR2EkqT6C=p6~S!040 zV@7IeKNH*8Bf$U{s?yu{?8nye@HYDD+&=jI!(9zt;JW548)M^csgwSU%b5=8H={tz ztmqQ^$trlNDF!-1*os<}veZ!u5dLUVd%L>>v9I6BnDO`?H|3&fIVJ=M@;Z{_nfNt- z*q>m5?aKu${hOB%-ESV=9a~OjYq_oMB0=IChH*2wDo_wY^?KoB5ZCpObzf7oe}^Y3 zUc4vMax209K+}hf0R*z6Mb}WU8~K#qBDd3(i9w&X>iJCRD6ktn+#F*?A}xe7U2T38 zBwsZ>kDrHi7aYbnux$wM|7^*&pNddeaHDkOzG=8~{9S$;G2xgkoLsJq*l!|qCG<6q z2Q`4nY5qZJ&qa|`OIR#5w$3`HE4tQ}$m;H#K1Cu2($rr4rgjE2KmQxpbGyGcAtyZD zsPv@nYV5^@a5#pEWW%Npn{x=I)226DZ|WkG=hx?@bf%IPJ}hhP%$IIlpN{D2?et618tbF{A%l%rhj4JMwV^NC>XiNc2{7Ac1rOY z9PY)gy3bTs!|wT;wj`PLCFMp@ZgGu9~pb*`)vIy@ijOwkR{n~ zaO6^WR&vb@M_hlT%wd|$H}YIxqM|BjA9WB4Xmo& zOT)#Q*Cv{wCFT0!oo}`S9Fc#%y4OA10IW&YB~ueO6Olc8ipwo(q7*v!PkN<-p_Z|N z1o-RmI8s>9XBL#iWUm8=d09xKekRsijEIm=G(=vv_1B~l1!u1Enl1Nj)cK1B3Vr^4 z+@CG)Fdh)L>K*$`%kKEolo`M<5YpSX?V>a1Zb1^W7GW1;Dvq`Nlsn>D!e*$T zJQX9_lSH%`{WU|a_`P@e;KEDjjq*W7tJ`z!*`N3WXs&|B4f877c)A7&OwhOPm7^Rj z9*KFO;)XoJl8$N-mme;!>L)X~S7-(c|3fgEw)M6UUqUH8o4(@0SX3!Qb<2*7HcRiLrzi|z zmuU%n@0)&zS1QtPW&a8&;nGlwjit}G%^YG?yjV*zOq?ejnxZu zil|yT%g+*s_jVejOVzG>5#Kz?It$HAueg@YI={^K2mt2z&yeey(zVaAZmBELTf6$H&9LBuGHNPO`Vt0!GkfI`tmRdhFGzU2p>rwGP<33e8 z-Wr@=0Z^zIz=mMZK<>HqkxRiEJQZ;%Bo{51HVVYAY*BT@A1FaWZoLo%AAggTwUIec z=daK|N9HEgGQyy=m864EyT|!TzYcFK%-{)IA-sG~)3Xq(=61Gy5scJ4OGsVOzl33< z!!z%hTr$1b3B8wiYiXMXpp2P)WOh~x85KhK5AB|}DpsDT#uEP`rH!swp*@W`OR#1M zY?6unBB}g9gRH^*>kyCYbwrJAgt0Ph&+u|VQzGh-Wu>Rr~&i zT?+5I$&9QPwl_v{)Uc!}j%)qYR@a$Urd7>b5lOj%vs`{MKqTG13c8UfM)a~WT#80( z@)2XLbJyU^?i`=?pV;Nq>|Y4;H74V~m-|pu7eq&(HnLtc!FFg2)N}tz?HWl=5TJ|a zhl|{qDhq7~Z3a1qebPtXL{?)}m9AvQ|8| z>B=y=ncMxaT3!=hsVvH(B?P%BVqeJm{x(e^F8z4*Jk;P^A8-3SYt`cOPbXCVlN?S$ zHg))m?K-4$yTr%~juV_u!c2V*hW|k9^W^mNOr_m!@&>P-GZvczP+q;%xG=F$&JIg! zxF9auWZQZWNh8#a2E;?)0vn`gfFfveLdy3gL07aZP$}ED;fX<9*LKy2)k+D9pAW1!a)YqhoP!nC^eYQO>-h6jTVCtIup z8%Ecc3Wz(`mgI|Cb*AZ7&V*d~g{Gj&ClzS&jo;0v( z^>S>8bP>VsF}0XaL7eMC7myA&NFs9gFA_pv!2Y&;ZVZ1rMa@!6`2Abrk1%eN758BB z>%qBC_wFW?kF_*J(~$`qx$OLq(WvqLTrnr*47aIo>&8zNs~9X>`(ldu62c0h+Ka{9 zCnz@OPC=<}+*j;A_*;Kxq>SlY6~Dw*sn{|v(pPtnUF@NymErgv%2T#mB>s@ZOyZp= z03@ew8QHZ(^+*7nHYY~SDXUJy1BH!5W8!QR+X^_z)bPB>92->h0c0(TaM(G*4IHoDP;SaO zCU?pirr=H6@-zn0ic#Pd0r(U<#Nfu>YT_rkAWJ2Rh(@G$Dzo#dJEjD_=XVPqIU8&d z^^^eCE21^MZ#2AMVU)5#PewD(tY& zaOHxOpt+(~f(+2svijz>)B5K3`Wsvnu+URiO+D|mc2S4pcibGtAnOLHcYbiEb#K(LVM4)=lDD*$p-WQ))-x} z8h~>Pc?V-#Rh$4YD^J_+YG)7u)LMnx1vmU5P+A+wrg+n_++fANu$;_0WsFOtS) zeT6!o6)!BV(K8trkqyl-{Lsmngku~JueI;Ty^~K}ig#s-teVBsRv_z8%O*r}>ca7* z5JH7YY?PrDd)}Ojhsg4|{zmoKZUV~g*$_GY5w6BQq^i{^2UlKn*BS9fDXbEb#-{`! zZ~Sn6CmJLO`O_e{#+0%;*H7cE3cpq{*~%wgOgRd#q&nqa%GI(Hh>}2<#s&dk2~Nzp zZQnQ%X(;Z+E7+?g)O8Q@ghta?{!%ZM^~-TC0LJ2}BwW(KW##di7?E3%MQy5li6!zi z+Gok;SGNfEQA^Vgp`1e|$w1C%=plw@9T#Qw4ZV|mZNzut{-gWnI+{;Z9@L<#*-4y| z{UO$4QR{>U2`?lMO3elfkTuJ`Cu)t)if|1$8FInKMU&Pzzk@JjBO|7kCTBnOypMH7 zwH0r|fhc`aBfEXQgMeJx%DEK#Zgew$IuP@c*vN!oh|H2-8DP5)2BC(6c;1PAaHm|% zKu9!-pyX+r<%m@JfhMde%(P5E?>nFyZ97Cd=Q-m}Pj$z-1ykGA4bqh|4W?})omR-| zo%+-y{h?3`Aqh9Ufu@BZ+CtJcRgH@Q3F3Ht7#DT|RZvEncTxV(rl2O&lCj3!K?{cf zDoe_^?hlA_w0G+6q=RaX7geiDL)IRyYwj4T_}NGIE|yOl z$)gZ?P%n!R$q|A%enLX*Wc;f{`=X19RY%2Db-!(?KDIhTV9Q5^xDwlB8 zk${N?!WXKJ?Tj1j&*pY^CFIIcU5%0d`f8hsKYiTD}RQ`F- zU!p=OFYWpTF1~hhEgWA2EQf11?a~%g(nwtDCD#@df5*+OWxh7A#sXT2wZ|@H7~!_O z#q7&6P(fSXJy}3jIQcGOyDWf-f093#eLQ)?TodJtBZQEv@tx@YfyGSI#H+&#<}CY+ z%V?t8pdHq{yw?9j^i+F`<>7XK=;8Dr;Zuuqa#2XNY}_hV;XH-evaw1hi^Kv1=6O)m z8gYag9!%zQOS}wDPI}QUHY#-WL(#Z4QtE=VSr%_qpG7fNnK=&I;%lbUZDI0#UDBI>CbQ5)6R3){AosxCa>%AML9p!_ndq+&~)lJKxn zZ*qlG)8ng;rB4~nH0DX~lwkpXWB?#e{Q8|;qR*)vV7baTVP0x6v~($}*Lf}d@!)hk zZ=2@%DI1nwa_NEsGZ#)sp7lkd^@ zg2i;pfb@%Kvc2{DP(^Ii(j>N~H^}Q65rvbz3VR7Kq)`mvyF=h;N5lNBkhKhgiW$B9 zTG;q%sfnBFnlhG+-J$_lU1@`hyt1$*>R0}DbTc1jkNuwL_N zqy$vJG?!w0^I&~I#4R{z-uCC(PH3vR`t_^q0_8^Mer!Hlu5_hFGo&(tz~r|%yU)K` zbjTod2!Z3&QdQ7#jTt@>sybWTJk4CI>{$=n;MeqjIieDBzKPMY+vnUDOwCUbY8HK! zSo+~@VyNP?E8)K0ZmKKGj=1crl`I&@|DrO-I$t78hxAm;4IdgJo+0fo5dzsp-h2^L zVceqXsV1Q*eZYvc{pfR`@E95#dcbSTfhR)Xeb?f$Y&?kup<;n7=vTP=zKyRO z9I4_m5|#9ArOh?bVkxq7?i6@>2c*~~<2GjHf6(0_l!3EzYWA8P@N(h`8SW?dzN03*zKSB>GZj#~^=iWmd-555vWI%oK-GVW&u4x>Zr z`&Q?+LHy?W;8gl+&2T8QNm(hRm?BZ$h*|-3-~*$y$Sk=3B`*~ABSflzpk^<^z`3XN zO0%c?PdC9fR=^Pb^PVh=7paD+FX&d!;){v>jbMYpUjqk!N*4 z@F(VIUo2{ss_EzD;vPslh`^a3k`Wi+D&%fS=+S#2&NG^8((*Ndigy=5@ScAJXM ztMQPKdtMWI_M1#`Og`RRWq8_l%lsn2>C(=;{wsMYv)TuPVYwzs{Nv4N=lQ0R$)K5n z0~e<6;{sjsAm?Ct|I>b?p=~kQ+`2-JB8OXz6&yE1szA5Vufp#1_^W6?&*$;Fc(pUL zi!~ZusgFqD+j^chfWWOf0pI&f0Sv)k83b2v0Spz(gA)f$50&(_Ud5owEc*9Jg?Vd<3YjC- z$Yqy1%sl^9>&^spoqV6)vJ@Jx*&)m@O_{6mI$Uo0?PW$|qN+DRw)KjP9rch%QMV6S zCE$7W9PlpLo%kjj`ao_;431cmzKN@6GBR{rnUC~AX%GELY#Lo#k^qgj4-1^ zM#zEjU~r)_)8~`;e-YaK2&gK_T)|4tO1mY8?`ikJA1r^w8&olJt2OFqO5uF7a4v3% zA-iMQ%Ev1~GX^SPc8vU7&*~JtgDJ>|78yH5c!Vd1o_&$iMs7O*Yfh)C(y-;DGbcCB zm9F}fWB23mL4-pUc;&LsywboI@Y#K1u7`Fl z&5~aVZ@EaMuyu;##`K89oAup~sg#$yaPV83bDH8lzs!00`A=r8wSgDBcKL+$ENc1( zzm=HyzWp%j{#9}zjx4`e*lwkP$u6UnW3flDypJ;(M!=citPjZdoXfCU27M$qTi)3DN?Nz zS~EPkqo-;e8mhiXk8uh130BF0wa(XLrt3!!uwF^^jSoF9!#X=eb3HG6W5i=2{gVMf zdNaE;kV&FqVG$Qs3uw7#0#_>8a0nI8CFojz0@4IzhyXIDtJmmzG|avW$2+u=2e2su zpEQ-yX+31;BMtU^PHR@Y-NghmfMsKC_1UgF6%N77S+}JeWm(-C$ft*0wl{+ZRKD4p ztRWPkwc{ld|5Z=E0xDC|YoI0xu>Lqgb+|vu>YJH&2&pZG{TBYr)yR^QT66Z%x)Y=L zqK@U#Ew6s!&BiJ!Hy1GGJGaKN!nZ1Ohzj~CR6fJ7dZo_OO4|{k;oAfwhFjoEN~7&) zA)OsL*~i`R--s4^AxQ>Ml@^_B6Op1HXv%AoNH~T(-ojN;lI5Lc(I$8aNomVzY^N` zrO{S^<%?cSfNn77i+;W@JDrRLml#Nx1HD_u=Z}2=EUx~#<#%ocFkbCUKBSrmGrvTm zbh9F+PbBzaP`~SMH>5x4t)<<)_D=LUAq;q8u^+?XkXI6omgis=y!f>dsvm8+45yzx zs18P^PSx0U9YHJ!E(_?UT+9gT0QQhN>H~OHl^%#A$C5L5jlm>m>?=vDm5?{h8>okq zJYuDe;{FA7IoR0~@-{)szmS&$UR1pD87w zcASb<<7|sV=23Zl+_SqiB}dyvrw65=BPKMmna-mP3wWhdp_55V$vEfr_nFMI5Uev? z3Kt&5CaTWc6es+G(dA}rbD#Hz1?pOFZOCZKnv2Q*Axl2=dhmcQdj5+*09tp>FN5en zTf9Y65YHD#d5AZA&Ilp6`a>`hPe0JAlpDxY(@gS=gG6)gq?szJT*svbxmzTEO{b{| zPuER(w`p2>;8%62qV*6MoTR_9Y<~GJZYDz_wuA#20lvKu{VyF6ki;ra{&N;30+6FP z0G~F<5zLvM9>p*xWLRBgg(S#&2fxr8TH+~q7pFS!$Ec84vZq7%R0_*D zWnrR0pDjlva*UkUY?t?Z{ePq|U^!&v@gdgWf#)q+;r$5KO>JaQH!={5(!j^_UK5>f zN9~OjPWyG5e{Spm9CUTx59MH|FQr-a0lFyPh|%I%(-MHwR6-yAUgN3GI7Z%W#o3S$Kkh%pf(_G84jP4L z?tsblcE2O(qR~uQR2*neif_!FBm*kVqM7RU?}1bPVMLF2M#eD&Rtv!cbAjDYhvI=v zbZP4Ea;BFVHF82R{}ioU6r_HU;xwrJ_o*^?z>v`v0G3&N1(LsnqCn1O*SJS>fTAu_ zX(&aJmtkxjM7RloY&Zt!c&mQ^!ZWd%^zomC&ukxzI)FmMPhoH5E%|gp@2*qk z-qtf~f{V+6JYO*X6Z3TF_m2xzI(Z%4^`|dtnkseoiAO^f)|y1L5(1!XcK6wGHT2zE zY9Jrd2A@X1vZssc}7q8-e2cN)hZ!AkB(sD=^%Zlya;>p@$V_ z=wd|{2IL2h_{dWA=1W)yvcgfg_-(OjHsdq@slc`%LUMJ{KYJaoT0e@(4>0H;$Z8W& zGKiatRm)#FNxo#Euf~+!pk6j%9D}Hac59F7P%(AzAZH0blOXA@3cXrN%{Et0H0WP> zyED3C10X`w-Pn_1@UL6X^ZI9QJdbO!nl%!G0P^KT#pP-!Om6G>CT~z!8UOKUtR}O; z(*Ol#avLCCo-%9jQ&LR&aU31f_AK2z1R;dP}n~2x| zkF`Q-X)IrKc7)dM8%j^0yccN*m_~O;tPUZ* zlVNduwy%p&BrCO%`P|lcbI!Bf&q^;Xb3?^usFz5*oTpMY_o+K)oMGR@*e`8ilvglj zKDPpCu*w0j^3-Zd_hjC9HL+7P>!b#GmdROy!*q%l^-`jc9R<6Wiu=b-lfmCpk62r9 zhy)ykSCV9Kie7WAu_l&tai!%9F>a{t+~QyDUhwA2$Lv5+qXyT{}C^V(i24ntS-w4GT z{w*SWA;`eld=jy83+Ry33kaoccg#2#65LK+HfjkK-b_*^xzD`RjSXN1AH|e~QdTC? z7xuih$26>5gB9|@7E6v;?Y5m^qL&6e$)X+M%=Z1N$xLFQqDmIdpBuIIhN9_P4jNSp zS2X3Z;=UZJ*4$ndX$pR@kPfy?R6kT=ou{imgK!q9u7s>8df`wRIV|V z0iHWxz>t?<@9pmB0$|jyV2h64kEemJqbeV*>4av(Su>Ka;{IV$>GWeoxkwkT#t7#4dRs(Ij#kJ z-cWCw9UClNklq{11#DIkOt?%i-XYGaNfhfhYsQ39#4+G*nM!^$!x_Q=Jz7XpK9&f} zf=bbM$}BsB^gZq@0>M;`rdw-d$OHVNGoyi+=+LtHpu*Ca53GBNVeqgxe#iPq{#42{ zH#?3<7&jI?MujGjk8x3iV+>ffQ)N$Y7P{ME2dlj0IZRCbllS1}h!Wa4 znO-uLPl5jU=AGnEndc?9h6*XssTmZQhuM`YADyz>v?5wa+F?jTT-kl$N+bRkCVj^M z)>yLXK)*Y87p^_#7p#x~xiKq0EZ=k>@han#6Y`FrtTE&pYIa&*eY#O7yX1s=&cD96 z(0`a^->+ANCNYGi;ID*h*M=y?xsf$D&*W~gpYZQia{eGH?f}c&`OCAxn1}w5}1{=lJmh2kly49h{p9fGhC>FXJ z3ZjS|o;4z4Sp2BbDeUWrN|Mj)_uB7)urw*!j|<0GHPIU#3n))*w_nPBncF+h^(Nl~ zObjUIqBvpm{uD%v1UbaM%$taQqJ+a6?slKtu*}G@e{7i{%(sU)z20;*bGdrL!+o$+ z5N6NB#^K<9vAaD<6K^-r{xrOV&n3-T-B);=w{+7g$@0k=5ItF%Dc};*kk+I@k=FQgNELcPS z-b^BP^A}$9O#gM;8tsJhx!KnkINnSVE!WDl1$)VXK5~M!xXp=3|ec_M&t9wB@Sf zU&Y&h5wA=a!n_Jl;(W)(%>`R@_J*xn_)Z-P&&QFmcY_TgZ~1B??0ZnWQbUZjS&?~t zus0K!g;X;?Uk)F*UVndz%Ag*@L24kT1lr)@>7YAw({iSlbmiaZ;*7&>-;|Q*?6ZCj zo4CY`NkEi_tkr2$sUCd1ZGW@x&j6w6GI%(eBFC~CtM5+fC-*xzg4;VTELCrqUPmQU z&5*_NxppJ1qJ2J5PyRW)-NH{FmDZ6|YhYXHk}#f2*|vn4!zK+)epe&X9~xxSBI2|j zfU+6`7@1rJ6YPMExRkW(qv1u1HT9uR03(7aoP^Xpbj3^DP+WCe@TTs}F2<>5`8H^J zg#CX$HQ@R8E%%62{JNq0iitv^%wF;%&f-o5s zq8T*V)@9zk<~<>18YV@$+5#({AC97_BwcvxcC0;kNG7=T&dmBqgx?#gCt%~Wt=Z1v z{1H}Na7bn{$wm+OQk&FmRQr7~-hZi=e=4-z%Wb_~g917a+_vYRMl(ByTeKG`IL@N%P2@f!@tX2xbg|+3c+?HuTmA7i0>`UQZ zMmiV*tp3+>{$!6mxy?ZT8g_n+F&JJ{~&i}%t$=%ueB-tJxk*dpq{JU-ahQ$5&^Sy{94kC3pgL8H?(GJ^TTrc{~?q8^DyR@2rL=+ z?eDkWaobCgcn23bsWlJHmu!Ea=I)H%G2GE^Cb@V1k5KS$@Zjhz;9mWywaxVM;?rZW zY%UjwNb<4e-M+uj%ySK&VjO>Cq<7SlxxN4Q9w2K7F<{lFe^*Pc!dWvqvMHgJe|1nz z$hx~xSqmeA9vpp0?sW0YPZZ*17-=|B$wH`kOB%6-Hq`G5Kppal4$D$l3}g?d-g-Ch zpc()7(o-O7Q&Xf;oW7GI=Af_cOcLq(?Y+(|nJIf4j-3&o3ItNn@pR`ioclI*`;qDbg@!M6U>R00Jv6)p?V)jr9D zUrZ$2myQJLk}&!s`?pOq0PIk2iV+AeMuaxT{W{?-#0yd%t@eZiX7`ST<_>$|%u-SP zRK()uUU48zl2{7n(cU4x(4^6`f6MIu&Lo$&yILpqudp$JTGFs!;5&ld9w z>#2Z{{`yuF@9$ajB}$~B3(&#=oWp;XNgqh_>n9S%oXh24PvmQekJdl{PIJi$&R#ig z>a35C7{`mgJC-I4KPPf$ zA@Ma8@qhU>S0(Bq3%gQZTojvEuxKw^>jcb4$VuK5x#ArCA6Nio`8_!*la}?w(_BO_ zjvqtwC=-Buc8x3`0z@z2Csee`Ak(fH;k5&&ip6E_aL{FYq&e`cN? zL}z}0ZC(VwBH}w999EJz?!wJ$c&rA^qkN)S4BO;m?{|ouR=oz$OFfP`sU>e$y0r1>}BXSD2Rsx|2crnFLq5 zke=7T6;Gmy%nmvq_a+m_TyH+oXg1xT+WvpzlS)Z%r2Ba<6NXI^skS2}^$~|MtFl#n zqVW5c>W_k2m;z|X!8qQ=JbDuiF5g1-b6l;w<^g~_lr=e^^% z zIo|hi1I+D#G)Owo!Yky$==?U%22=NF(tI%0BLRN12&KNZG$hv2(&`yv?T)#d1$uad zE)|!qpgzmT=)sc;Q9JDHx)%c%BR7d?o44yA1@KThuMS0NzKHqKEpjm5(bgx)^S zS|lU-MSfZY4f2!u0d@g9Idy-C;4CWook7nHqif%trbxD+5&AuNnp#--bJRbJJKWH} zQ2fVgT|=sv+&ym&*y!SVV&Daj%b}C`!ucwjZLZRlzlJV=pYda9X8_R246>$3X@|kU z^+|}67dPRu^RGAHo4Z>QOIz%pfRU-ou=&Yh5QnkV@6d({iFp%|FxA`VXTcb;{? zKkpe@Pi3_;;CTkNRlu-4Rw5)kvPT>Evl`d0e9|tNd;QsagDmyWYO`=TeTT<&3L?cY zG=(dLvC(qC;%;c#>E19GtC`UB?k-EEfG;ut@Qyov_XpRtb8>rkziU~vxuwa>;_o+} z1=tmwuX*OXE`?uAxh$U~YU@gk<2Q@!yb=29ob?Qd?afUuJRS4iDOl*fL<4-&>H-Ke zH&oxoCpAX}0>#MV&cjb06?}jPSyVtOMhDO+TJB|c7DOT}T09hllOC9oSG{0h;yS@Q z5jc6$!cRm&qcHUVnB)&%s4*&?Y&qs$>&H>RhqLml*{LgW)o&(Et*s@Bmw?^^UgL_b zz?8)p0=wMzq1`2Z@o%cHsCtCS%Eg{g$D1f^?T=cmq!FP5&>xQID00#ebpA>O*a3(n0|^yKQFI~w>I zt3ylP_rOqxeH*qyytxo+GJaPsbUZM7ud}%b3KbFWwcL7e-1A%+X`!1ZCHtnY=`bsF zQcY*AMjc=m2(Nbd1pmgcbWYu*8CCafE8}-?a?}?s!fLc?Pz6F=sbZSH)6C23n>o({ zz{LJaKm=U;mS5Gwhx&SJ@2fBA^YvOBgA$RSw^`i0<&4KKYjLY7w^mOdG|B5@{R*== z{?a;9k4jdK$dC_Jvsa10??s5-y;^>F|8fU4Sc%}YDLc)i+qo%&j&;8lvpw0Z!xXIk zgL!jhl!JSB2eUnX<7Aw?@z~uBxtZDM`GCu_Hj>uJbcLxEBBeN>eQ+n$RHiv#gzJtp z5t^k37~v(={r1bZW3g2@4cBFrQWfJe-%DKC-I3C13D$l0(yGtfjS&j09%C8c!1>9e z9hx_s{@%^w$QrHkyDrkff@hA^Nk-LdFt(cL*&VHNc0CSPnU;oH@jRp}t(}VaeOTa) zce`#$;CQg>J>vEOS#&cU&J*BHK00zJ{RM`KWhLzMd!*9vXy7u%dnz|;*3VjjDxhul zk!c(*kVeTw2}_?0+@%dm@q-iXer{W^r&cTPscKvAzGQ z)*QVwb2xrlW96AogFsBFtePhX)Bq{fC>~h%+M!mj&^DU(=}J zzPc>VCYu<8liLc1QtUGMB82YdwPSkCA&U>Ci}&u{OW7vr4Yf_)R0z!)plnRJL94ec z#eqWk&%=~{*=~JAu^jjxMmS|LI!Y&Ow z&-&5rACWtsaS%PCn7Ml*z^0x`*8df9a(Nd?kSXk14ENHUeVQNkQr>yh1rYeLM=m=q z)z;SWCyPtVDHN+o%x{wT*^}#ACnpw-j)n6X_VjBK-@l6v&+mtC>B4vfn|=slUP zCAefjk;YN4@Gx%+TUtmu18ERCsdIyx+A9dEoT#!4b#>-~Lw^|vd+fA*Pu==p&vT0Y zuAh+*rXwrqEIHBd8B*bp;O_clhFvXz3ZHJZU~9zbW!&6~m(R?dR`W0*!2x~RrWbEc zr)4anJaCVF{3ROE@b?FCV^d2}NjpYhjWQhv?C$885b;TdiiT_34tQ(f%`ybu=-y(T zz#q37M1+rF=Vopi6G*VUX&t}+iYitiJTtt?LCh?*$F9KoEVI9np6_M9Ql` zn}cK<0OKIV^tXY?0BjlgEhPf3_C#`-PoqGN;y>O~*6U8lCDRe(YA_?))q&7Y0&akNUCUI*5sBuS7Xgk3OkaQ*~#; z{^nN=qh~}TfFEOhRsXbTRa?2Kwupy|OJ%rfKx6ZVfmsA2V(zUyqb1n1MPTFeo98wr znaQ%`af5a@WFK?#7 zay9dpprrvWT4LTBb_V`0$l$p!1#SR$G-VO#K6g*1a_{=`%vrAmlK9^S;w4Y|04I<9 znV4|7Bs`}`2^z+Z!qZ#0{v#V~wd-u$UQjwUnkic8BC_zIK>)eh5y_$laUZLg@7?ya|>_Y&5TrcB%=o+RpkV&rkG3 zI-W<(zE)I!h2UdB;^0EPvz6hdQeZ>>U?CJUHBA7m`c58F>1dRAwf-7Tnd`MoN!LGL zmLI6Dw~xe?xUQb4EOiqPHQQSqa9DJiR@>nT>HrbFAFzHLB5#S{gE@j@)GKtQS<6nR zyF9ijy-mgz5}A7&ufOv2>Cx`>Pq9g*+va=aOS_joI&SNF9oPU?78Q8#%`~Qey})}q zfud3>W-GD4(=jxw%*R#KZ;v^o)C=xNU}bwF^zuqRDWRVx-r~liBJ6OwABoBv091O? z^cYYsd5u1#+FCmw^Y>N~%eyn?+rr_D(|f(f$?A3XNYu<4*xFXT=BwB)D{Dg&{BKkbuckyd}#d)1qY)JuXj4%H4P zaUSL6089s)Gbh97=>WR@P+&QLPWSNq<58;wlWuQ2(pE9bvAa`p-OeY$Q*^8%sYJUNpwt_F5}9=t;9UKB8YKk3;?<^E1`^8=f{8nqnQFoqquCT z#=i>T_P&bpejMiB*$|TKzeH4{Y||{Y!kcIMBI5t(|F7=gO&awl`^9EGXnNtMah>nq zR{RlE8%1Zr&8a5YwI~r^lI<%dC{M@8*)&?2)Tr46X>UtHaeN2%*dGhBf9;$y1uX_< zlP@@NTWg?xgwL}&e?Tn_5XYZc3QN6ZuGR2M8Qc?M7W1B*Sk%wN& z@b3Wt8IfG zx-+D5<RR4>&Ad7VU61rAig>ONyDPITqNV85~1{+kDP3m?p`~s8+AuI4Y3)5-I zcC+msl9&7E0KFe_uN}oRLx6ceR|DxId)ZV3zy(O%cw*x*D@HPoSW+kppEzF9Akgu? zMnyl@Pz-_l`@qx|`I~$ybW}v(;xMm(;u63z@8p9=YmR3VnIyF7&gS{XH!KEU?S99@ zM8zXQh`EN~6`-@W&l)O1>c0Lfimo7S`@X=^S*qPTPPiFHT}Gi^P6W5o_CN?z8zf2? zrLts3_d+l&!QIV_!<3qtfG6l;s0>rBW;07Dlm~Nq=S=ovh0Km7R5W_LHPHWMk%N_^2oEyUkgeop5G8nS}c_^rR~%DvK$h zhoXfs$DlRn8ib#+(Z3P@vb?MS+tFlO*K;!b`sKPyA2eR2wX;>#&mtCM$)bTp??OHY zf6{;p5z!n*AW|_nyS+YYMiK^RkV;x)q^uK$&g{-)LTyUnp)*Fu9F`y z;MZf0#LCdSAO6(}B^RMj@P_{vAJKTj`?`tn&R@&aR9O)_G_9cTP?ehfSFQ8xVVfU9 z-}EM4e=Wu+xb#@09bIY-uN^#Z+R@+s3%p89Q%z`FLTqM<80|-nw;&dtsWLDTQ*%#aBv+>8prScV-il|5?oL(nf6q=}S;2VArQ%_ezuP0f4&5Nad*`#oXo+XUKJ7q^{wGd~ z8N=tB?;N{+SckjrE!2PC7CKE4aY$!@f(RZLW+Lmy z&=2H>4QlHCy8r3BqW@&LH@kH}!d*#Kk-hk#jHWk__@nE}`EJoKtIa#Ye_0SLdD~w{ zjC-()8#dhhHtpD6_vd*}+IJB>DO(ty*(XK2hH$GGod+*lPg{9y@nY`t@>@Jr(GDqp zg?ZLZlj_g2L@*>PIQzfn~%+XQLdVMtP^^}wB~W#nCw>AX7}_x@9x<1 zuq^vM(5XI3Ugz=(uB}GA`c+O*gk>uBV4nNQvE#heFn=<4Hp>m92zZ4UioCU6!9cVv zLAZ)83%_A;_WX_68dGnlJqEDxFdulfOKQxGZhuImUE}kns~lPBL!1dZ^=sIuWAfN{ zBjZ3QzB?O9)cqu7x`)~`zhVX^q-W#lW>WM1N%z?hRHp|D;^nFQ|BoqvK~d{R)@q5o zC~yE2{0I%DtgH@7{1BiW$yC1D0bWjle@w~n5R&)K-!SFf*dfl71+8V|<5~MWY!!EA z9x5Wupi*g158EGTFTa(c$d(uG4j(x*4C3U_-^-tjV#DA5J4NSxtFhiE2=3sP{qz>q zk*EEOXULN94RAn{TD);>X!v2!8~cwfq&1(Xk4Gp-WygwM`PXsLxHS_aJ(x(*02OPX zZuSIxk91qpeX=j*615~=MWYtt^a>3%k@0X|M%6v_0J)15V?01q>IQRCA(Dqz{PT;$ z3hTB=id5PJngHq|k<3SttP!)MCh5qd_UV+k%$u*;E%B8*lN+_75J>MJNe+0D%u*b= zwp&7cf|r;dnku05dsp5~2U2+se_6@_vkDhpwo|;7&IYx#gSYD=!~(!`(gyCKmv;a( z20}3|ZS<+m8i@jt=T+RuSAX%2F`!&<#nOzXJ8s^9S(9pfysEU!K^AatTz;`PoviL7 zP<(UBOtNV$3tufY(mNOB3%R5vNk85D{kxOW2$IZ^7IEdK|GAY4TzqiZz69|RQf%pD zeQ^Cpk#L1~*NBUw^(ASaJ&dme-?v|YrJ_>NDS*JhaAj(44!*{acIrrTkzpST8wr^m z`4WluWvj6ZB+Y-&Vv~-FLx7HdkFBCug3(xgVEpDVotwl}JIF;iT2fARI>(%QFDa>|2o#=6TVj_v@Bf!#Bkyh@58$09JRr`MHR}0g!;rvK+sp|3E8K~MO~Z&Ht9f|H zwt4R4WH8##^Zrm8p89)JVGwNX;`tZb92YIY{!QV+o~)uG$_SFzu|5$bTQ8Um$K4gE zMq#5BI7z04Fe0e_TjJDcmP9g>B7prJ#-|AESd66MsNa0RaJpY#5cF3(UNKoZmoiyC zuR`|nCvARdYjn*y+HX9rR^P@=$`0Jki;}nKVx0XS@b@S4;6n)Ioud0=K6}$W?C2E;kpp~5Uq+6^XGFi~U2&vCK43Qbj=i-Cm*fC-_fV1xklSMV?ss1y|u zpZiVy0%V?#tgGs*eK(4&!XA8vrp#KMhBQtp{WUt_v?qfI3gHcBk=?P}^0*HQq)=Iy z0DSY$*N*aaHL|xS=@~EOWPMI2dcweD1yam~9l51lF$BNLD;(s@K>9ry$U_`-txIsF z?zn)!&w>w+LBm^a;t43MTq@g7FD-@q3|5r*#BHVvtQrfzwDg-V9y-ll$O1U=blMC> zBGY{PP(Wb_!(J;@xHFtjV%oODfS#PnSgA5-@{Zu%y{0K~*)<)X=6|$EHty7sGNHb- z#smLs3Z~|z_<5l}r1e9C@D(4%#J7-z71N>uMJRz5v|(Zg1N`db@XhOt%bcJVCZc@D z28)^DPkpNs#0BGuU}r}1Y{RCjKB%;Yq{UX|%VqQvc$@f;6F8*57{-Tl_`zr|^`YGE zQ{1kCzbPbbHkJ#AaDTNUhi+Td!N(T$DWDS&ZYUZ!i#Ur5C(0-`-m#edA$Oe1?%%l(zfnnOOF&yE?syH6M_vb1ik{*28&Rg5Q5S1#?kqzA&sV*Lv-I2AZN17iy1?6}t-~2~jky)z zPf$teQC`y430d<4^I8%`x)X;Ne;bMgZw5qom0=eURbjx1WvdAspC69_y}|)p{6efY zHr%{f%5g(a;_a(dy30lA^Vta&kh&4B4%Q?3o6-zP90jyJWr=kWnow}?Z_0{W> zz`%IqE9jFB;XRPoA8_SA*)Vi^_nxf}CVic00!VLX`TaB&en&`gbK8k37P#QvRDt-L zA1vGJmc)f6Oo4s^&iAl3E~J6zo(~Io{2@f z$>@$Kwaf2Mk%*es?T({2aZZ72d1n;coueeDgx~grK0CKMOSWRK=6RO7U>TD*_wE4g z3yn3)AkbDJ%R=Z@=%o=w@80sS9U7rV!i9@_?fUB02Hna0LyXAk8tvWMRTc4Pc!{$C zZ0d!uALp&}!!`qLmwe?eSBGx*WcBg$H}SJj7^TZ; z&aBS?iLzk}h7< zh;nlhz(g`(J5k)J$)#Jy%-JKk@%`PZ@5QbX^Hc%8ZoFzqeXZ8S z8z4-pw$9mo>#?Ng!H(#VP;9ZXL@Dkr2<3poFT)PunfFW1VQ4XC_FQq3L=l@*7Qq=M z9Px<-!k9oFF$743EmCDidDfqJm5p}J0(JqK)1{+Y{SnhOy}^=9)pPmGp7@3Q8|z%3 zYQW0dO}D4?nZEWV(>33)w8*|!uRdSAE6Bzzg0bwXd^qe;jGDfX_wk@f#H-<*WQoSp zLjKUy-!--JZoAf>l{k}eD})Qz;lccxmo`>m6ewIIrd5vZo0hL8*4XAI5A9`({fT{t zYW6V$h%Z}#CcuvyDR2?$k=4W8W#&a2O_z2o(9#eFcfK3lyxUz}pz%hno5l|0d)OMr^T zeK;gRFv`8TaqxxI%I9zlIpeC|x-!P@TX&~ruEFj8c>%X~M5Uoe=VO$bZEFw0SoTAh zgcoi-unh?9f$x;Pokz4G(uh_?V5n z>dZLzyE=GsU?x-8zf+pdYNpJQ>Z^TT40=N&wEe^pk^zK2%~PSQli7Q-B$PPXl`kzBPBdYTRfpbR zVQLZ+se{_+nQmoGER}~n&uo8$88v)PL1e_oO8s%ng%)@qXU1WHRFADh;qda=Rbq(^ zTPHCI%kXRg1lI>i=F2^T}l}w}?n$GA5v!1J{#% z)m`xifEt)kBZ&f%ajnnGUQEhz_5^cM!r_nzwO%kxjz6yOxmFU+WiL!O4+HuZEp>pWi%+9GW z9rN+hJ6B#`@Bec{=mwNE`A@wTp{cBtN@B6Cs+c}+DUvle2 zgkN(TtQ>d4VB1}ck=AHaNT5jRvy~&laMZw~V)nfDV)PIw7Q$zg8sgvgar8iPzGxx; zTT47OHyBphU1&xLF@W<6jCsyASt?s#oV?X2L2wB}m-$y|@x)v6^-mIeK@V zOFAN$<@d5G*KOJz9Vl8A(EzcS>uj!uX$lQ3MH(;TvYsyl{VbK(5=)bf!0yhdH9bPz zd$D#Wg&>Nn*zD>(^YhQ)cp46}Bl-8ZJluEYGa&)Ju+gJ_D=93$lwRRARr`(SV#;yy|`l1CWxz058q!OCKO zcSa*uoD&`6dk^;x6Bpw-uU}u51KzoZRCBaxm~O^31-)a zi=C2pwGz$#uAwao2ukOM1Q@Od=~mBi z>ENy&qxUKzt=H)K(qK)7&ay3YYnJz4ne8pEX6i{ z&PPIv3p070Pg(SuX`I4EQzsb^J1?bG{sCdO*Lk}cn6YsVq#upvp`8=VQBZz7BbWJB z?TZOR#a=-+$YATI6qt{q&IV8vP~A+Y$s>JPC6)Y>_4p1etK(An_n@-s$&Q`=|)D z*1+&(fNg@%$cUb$vYWt!Kq-^H;+(82Yoy>7)pIkS;rM#=kKXU*F#$3EPTW1@U;v;X z?_pzm9ol>fdcq_fc^zJ-1(ht*`U)lS81-4xp;XeUq3#{jN9aLXj@ArjbL^E3;fyRK zqh>6a4jbqY1RXJ7wz?#Porx??UoY_8O{Xn<(-?m2qq9@dyR*7UXPw(ax%_?WXxWJ!bGyfr=Bnuij(^T%_VCyR1m8Bp%eWqpdUPA z1G3q6zKi_)AJu;NuUcZavGsZI3iYI40Hkh-=E+)b&I#H_)$MvVeSbjN$m<4LbJ zQE|qyEl^XbkVy}VND>ldKbK@k*~ zXiyQM^hn+69n}}$#zzF3Oyp}k4C&YSdLDlV8u}q}^D#kt+Mn!NkEuo`ckHZ-K-rbQ zyo}XxsCtKw$IEkHF@20Ka3;Yu^zAFl72oIB@#Q7#IMWP)Gb)?^vz%=I=wyr_xqSXs zuJn2tf6)uQ@yf5KKHpNXEGoV=)o5Tc#q3f>Uw7kouT02l1;WgL{)HDX8SU`J#L~tM zhuW%fRO>~eCsgzTch2V+Y+h7qg;&2G=*VLaez^ELm7-4x)$I*JVC*unAaV5oIz~g) z9_9!hD}XxgcP+uc^`kU$sKUgHZclHy!7D09WYv7*qBvM|F>X|o z=^(+#jDNKg@sr-7wWp;7VmstKA4e7dk@Ee=v>5^tUkRVMfA~Z&SvjY3CZ!sjt87#Y zOe|~S76#H~kd@Q8@tIArZ_&8W zcsL4mn>V%G;mYq~l&fX*jT0i@lbu@pTrhZ(VCtn9p=4RRiN6|gshV8%Rx*E;pEq1Q4c)q~}CmdOw zQPZb}c^x?2AUCPyS)?do)bXAG#)5s4l(AH~-jv~2jDTcJ{$qQ5iJT<1{A++npMrrI z1@D*TIghtE#1df2(Ksr+dm%YT_WVt81nekY->kXsu2pv|lc3H;dUc$!H>f6BDdZP<>63;x53onj z4>cII*ZvC0pAq`a)l;vgNh`WA9+4kRAK5q;7^P!S{(O7bkBuI#3P!9tGSyFj;>u0P z3lwhQQuW^tvyuq!p@s8RR5_vrtzLr7&robiGI1awDbR6ZSEh<(%r9 zOs7)@%_v`_9*S(;)E|EHOuk}{#g9ZYdhGZ^#)FhiXR1rPASdxfuJ^DtfW^wIO?GQ* zdL@f)(DFWWaJz0vD^OUkL|r^8&S zJs7B-TJo>_)}Vi=Suh2X^Z^;B*wZSu^w2oOLexdNdsi|K$i63it=sy?NOW)Xj^o#T z;c>ZiIy7G9(~``RYcjPSFdPJ_EoZEaRU{Ce|?=5T}M6dMqjfR z12=(h_Ic`=-JfA+JRRpNff(QUB>YnR5$N^$SS!b1*eK_bM$PI)c7N8O*GJPZIhPbv z|8v(4-v$+lz`o7<_3{N$>4*Y8bG|BumP|da`H)dGyQCGv-);>^f{Pzyj+lb`ba!{_ zm@7OvKzk087Pf0h`x*Iz;&Rwm-A4t3gwWSWcsWc2lbKdi-$Jla&Uf0%#P_4B^BrsG z*6<>DZZqPus`5Q(pkJu z%`q2GI`Fw1mu)-W39{9N>rOL*C%Y3B6aJXxqisJqIG|oX*RL1mRhwYE<0*-yV$1Lb zR+xl7VO=)5smt@SD3#bzRMXYS755R=(^1}iqQ|Xe4QCpOXr$^%qzNC#$T}TrFF-G@ z@ua6KP{wuiQ^KEf%*ybMhP^L2nxhoDAGvRwj}v>%x6qYv2V3L8GPqU8XrbrhU)U!R z!&5svJWR0sPOZ>y_CU#4deAX?(7NVW7lPY-lFG} zZixPFw!Vm$2S*;WnEl&4DHLfYXr|c2BhXGFBb39+p=oMdk9y?`z_!wW2u9x-t)Dz! z%HP-RTFyf?8n+}KKtxV_2AG23ULoXB1ZnTP=b|L4LDi14k2<&$P)s`?KN<|(=JO?IT~ylD$X+HdJ6Tt_E|s<{co+YE_w3uk$Vw zFkjb7$6ugH1R~hbI9VoWhy}KbZdX~9Ab~IU^(i9+L{MZ8A>tzXf<&g-wl3r^vAlIE zpqz?4jp4oTg%!V16*hYJJ5LK}xN=AqON}4Q4xa)$gRfDa7yLx?(Sl>F!->ev21o1~ zXl*G`BLIPncQ`Y-%m{C!0tbWTbRcEm+}-@;=M&rc=Ws>rE3=R#+nOa;s=+x9Bi$-* zpMXZ!{R+}iKsgDNxrtsrmnWp9tv>;9sP#}XQwt%|kdGG&&rw|Q=u&dY=wu!>ffXi+ zp`AmO@rcB)<-H6(UwhR_CX+>NeQvZ}(X&pqq1qCWCWz6Q%oO}z2o#8)>CYBOXLr^r z`u(s@YlGHh0*yB%WC0K|1aSe-*B;_<<^Ehpf{yC~`FegsbR-nd-B?J9C>6$;+97}3 z!`r5;bi+fW>`xSL&g2IvjcuHykSV-#KG!qxfU*- z>8HB)T~>6KI$;#lHMl5%Uy%byj>pp~fRcPd0t$$H^^(xKLW7~29&aw4kV$*k z;0la)vQ*#DwvS;cnh9xMwI}8*6#vOV)2Dh#PWah5efDVl-CRYqkevNOxVf8I_N16l z%+>i7cI@6FDV#q0M`LMQ(a8Z0h#{RLNABLp@eejVxC7H<64h_zt1uX~B}BUp-m(>H z8_tJmj@rxWSnE3tjTBeDubPPXA}Yk%`{S&bI!VV$A90+|ey(wVr=n0mmA}S-qe`TQ z{(0BG!(tah|7d%FOfipdiTTjoVC8C_c2wjYZ^grK-@8Gb57^*(VD)j0a}lBqlnKoSl_5z$i36^9bf`5@ODiS~_G z&FZ|UJ*QWi7P(G-PsF4yQfkEulql6b`kYrrXg|JkzCDrJ|$(6puQ_X3CY~u{D6k22&7FTdNtw3XudCHH;#e9_~bPc-oUduFzLI z&wNLA5Z+OiaC3AK$szXcCwpKcy&5#!p#3`w++6RHAJVO!(r$3f3W9vbzN;*ohGSv< zQc!TpVA_a!N0H=bHk22)%$Eh1U?uu?t#Z7EkKdDSX@XeC6%;m{_N4WeT% zHMm)?t*o>oF8c5S^EVH)fn?-je>K(t3yoI(_NPO(E>vJDfl;!Nm+)q%TNbEwvIzY+ zh*PzFxx$Q*glkr9j|}{ff2=}|Yzb`3wP;sdn=wa;=9TI`m>x{Ukpl!p`ic2+y2&wK zk`bgdh)V;UNeLXhpZb+GWt)iPOm^Omk8hHX%l7Tq!}c?%^`JcL83r&1?59Y z=;Ee+z=+`;3CuX~mxYJAS3`E1rW9GCE|d&R$=v10P^Z%33vLW=~ks{{PeJMkRU<}9F{j! zI@?}UI=L*`ll`Bv@L$%PGju3ke1^#|N>~HVQJp}uQE#4dJTV*QNxMN@1J0qFVlZdJ zSe*S*8g0Kg|2nP7E6avA3Ua96N7Y7*O4s?EFA6ljV*Fg$ijkR5=PX{7oGBY4Um4`} z0?I%BlXNRl^EkVIuscO(W%;R`1xN_MtF|==E+76=5IlBn!8`A6p6tMp9C-Rs`NW{R zZgTmw!$a=0sO{j$s3DKv(Nr%#EiU3smB_2@J6iHn0N{<%s7=5m4*tOdZ3L334J|}4 zpR3K)Htq4i8>QTzCJ7vSqXKu1H)Z(8$tcsK>e-23H;T=`&3579YI^@T%S@p0d-@y1 z(t)Xpy%3%Rasfpb0CmDH!7A-tkK~7>QS)wP+_$E+N|AQkX@Lc5CGh4g50WUlGlJcu zO!lI%DTbUOe~sbyK@M_AMDPiKGY25%**{-AK>ENL7ytx}wfcf&5P5`bD}ba{>IH9! z=z3OmzXQ{v&EL%k0~~ATixwk8wu0N~1Ds4h@stw?dqImFaVkFj%0M^x^w6*9n&=u##~JNrVCc! zeNt-9o_P!+;j02rciO@iINLSo29rGS3s?4|v)L{GvYrcizzGHTWK%=OjfQ6Hd4e&F zJpgO0bl{)Rbrq-nW{lK#9Q*T$<;sCbF1_2+0KZO&!e1|-{v%5I?i9LK(tk%B9_E8+ zPLX1~DeLGT#`KR%QuBLFcryUPAe;wJpB^p%T2QT;#s$oF>y)-nIsKiNhUOWm3!g!v zUIRW~=L&JQ0z?;@HR{fog)#D{x=me5Wixs%AP9}i5dZs=!pB2!RSGXVl&f8j@^VQpCR&R) zn6x6G7o=c%s(~k%$EFL%w0~h14LxTfLUZjz68^((g6>RE$bspM?vSP$+-wVYHJ5E` zP@Dl(NLk2=C3Q=Bm?7#idqW@{@Er>?Ubmmfwnl0%QRR2$XEg9rn7|eNVaDEXfAa$~ zd;67^P()(qf8P(@6}}a1e$+eE>=!`oZ)8A_P&!u4`tVye>S3HwnA>)Ppo0w;H$q7N_M444Z#A?jxac~#k1xXPNw%f*wDbyf=_JQu|Z`g;~=pa@#^ z-qHgvE@Iir!CS7plocm{H&8IYW(pU!d$j|}nyo}Ub!WjY(j^cpYVt(!>=4Dyt$$zR z!ZV-C*`JAmC7Ng4-gxrJF_Fq1tlR{rR&cESA|{m&YfcYanT7qQ&8h{oSuaY1m;dxz z9q6Ty>W>B(6WM#eaUON-V;;{(6BCz0IM0H2^Kg$rRv!Zx7|Lh(3c{3XqzpN+9%5)P zUmlm^$WS-duE<+Sy_{jCh?Jm+-egzWtuP^n z#FUxh&AHYLP~A641Q-Hca5kz-atMVSAbe?whP?m^$BzWH6DXY8KrIKs6i)Pw`UC2m zS-^pd{WKq`>8oiCsi^FZ0rKUPU50FD0SG<}qUHQuJ_-smACy2O#8jL;{eyAKaBBat z6&kQyDeu@Ni>M4O*v{v=Wsh5pisgbDKiwRR57~On?NCcQM^KPZ3x!8AifG1m>dJ~l z=HR43c?{{HRpKeH4Ao-PvUgG+ls>v*^uD-}!vW0xi}<7>T&V=xnJcozc3qG>h?R?P z1TJS<*t7M)ZFD}Ap~XH?_uzwŃeYc9mFNm-M;%zt^Lol)-Z5d&f1xLYMF7AM~S zPB2HX#k#hF9S-(tWp|HUgs5_lgd&A7PU-Y&j7AS3S!9h+&YOiv%fVvSRqy7aG6`X} z=~MvG3uVt*7{6@nBwhW1;WBdri2!V~FA9f>o|TwRKr{=8Mz5*GkO3+QnFMn@+(e$b zV~U9B?o*>6Fl{$g%i0y2!xFUH#j0>tBwS6@>B^Hp!F+Pz2&{jOKj7m#<3;Sm2H#jL zI;TSa_A{$4Z{<2gOlK3x#BFL%z;%nieI{Ma!`6R;4Uhh7PdZ&n+R3O}%x=UyNU9LjGiG z>GfLQrxX=9wHMr-h$q#VWTX%y4;T}i5fY@6#Yiaz>1Eye-*t52bPG}Ce>qFvf0NMT zrqIMc>P2pk{-wVdJ-!&l(atNlw`%TN*L*ZPSEe$6zAuzB33QMJ72O%yF-jcfiaZp^ zprqyjB@Pc8K6uybHY72Hc%5XxLW$2{S zpvWRr&E{{rEf^byG&113`VG4BvD0}5S!fwR;EEe5Wmhi%4nn|<0+3-8wy4gv}x1*JbAyxVzi}y4xSOVlRU9vki|+lvk#cF8#wR{Q%x8H1@aP6P*SC%{B_sQQhA3Ka zT%OFR$EgtrOgvRdbe|g(dCNNr93Td>+&m!yEHsV4DF&pNKCUs?j%mQpJb)V0F%1yB5EKG`6=eFJtRw#A&iz(i)lO1^ zGz4Iy`<{W;P~&9CBFfHRuq2u{on9?p?a%w;36EP7 zV{WN;jbdT;vP3{nQ^rhtfhG7NRf=+V!z}~8ZS4dZkQdl%;lN;1wP3`j%Uw~kYvJWY-Y6g+-Z@}{eOn(Q_VxFl03pFHa+ZU>G@ z4~PhMpKJzNxqyIAjN1V4pgWuRVH_XWyZ^QpGG8GQJDNaZf#pF;kfkxVt;>=bFAKLv ziQ=jS-(q`yD-Km90UnToeyz?k@IuRDA=y?bK;xmb-2Rir0+5LdFp8)Llf1BP#-4be z>J`6W`FYu)=uo-Q15N>|wE&i~gL#_Iqbms>-pX#G)kwhDjs|G2r{Fl~DH_fQ6TVu^ z2wh|5xc_9^Acy3g%7inJ^u>B^J$Uon^ak(F(GyTO%siQ&rm_+KDk4cYi23KJ+!NHy zgh8ld@2dtc+Cxc~c*9zCB4cJn;yfa`SNuFIGOo$B#gpMsCpQL6rvVD;w>AjTAJx5u z)HcTY64Ji^MnrUNu&+UI4R~T0B81zUqBS|?>RXDsu|4JOD!hBY4oy@WZu)1C>OX)C z>Q9v9YF3qm z0Elh>u%9Qnj#UXMc=sjw`Y1rX|@kEqKn@sOGfdbiTrl0vr|P|Pf~&GiVY75K?5qF&dJ>?Dn6^m9b zd!w;jH9)`q^e@8i7#+kY0B}xq;2g!Fj5b1}KShLn+mW>6s2jbIYyY{!$7A^OR4#9! z(bE;ZFN`x=18@_G+RGsf2$9#zO`dm~ZUzj1he9r(9Z+Xu9)Offd*t91L-#KX8EtK| z{l$5UmWX-@ZdjZ_umb<*zm$)fJ{czhcufkn-zMYJF&_9 z^47ffa9Q^coEo|OYS;8_P>~Ru;02I4&heWq=#6)kHU(&;uy)2CgzkwCi$xx5nniHw zGzh}+$U;xzLd!ATakus`LC+WA?h723404gqn+?3B@PXAH>3?CV!P?GB1JflPwg%wR@|x)lp1(Vh46VR0i5+naNn)**aFwmg5L>(7S>@g!>l|& zLGO@RZCS8LcJ!7BX0PcWhyn&o8dG{ZgcJkW1oJ26{X2G|a?xE^cQfEX$Gzw`5G%qa zi&Qux&jt$Tpf2jL^>#IHn$sWi=4%(2Mgf|Pd)%FUPDKkOU0Rq$ApGEms{!^$e|gcs z+QsM>B-35R$-QF~q7*$8&GaoIjs>5E7QFB}#GQ3*(W_9?8P*R;ocf+8T=a)g{R~(0 zvdt5M7fPu`D{LazMMEj)eB|pZoqiT2@Xp&x4i%Pz;2MUY76P664zt$hcg=2FaI)?# zZsc*Lz;OZA-vPmK{EgvnAGS6coL=s>y|-Kx9)rNNhmJ8w7pBNlOOC68%m$<=YiE=3*T{tFk-IpvYp$j0!#nu#yRv7)Ba%5dwoV$bxdP z7E|3XDf%I^ImB2O_^{@8OO?sftI-k(2c2f>vq5J=z2043sBaf%C5=pb^A|+U`#JXj zMlAfUp}PwCMuG`RIZmPGLR5N?a+>;T-Y6v3Af?_mX(s@8J^mrC3TLi=+4{GeO|Q=w z2|OiApeLc%0VE(tX3NgYX135Ib~kEKQ0UWjM2B0{ms*Zy*}<=3u0`Zh`f0!v{rld7 z2dv9Z-^gWa935&kf?$XKU^qfxd;GFu-MYI_aF924qORH!D9Y`ac>HhqGM|t$_i8{` zO>58>X5>yaS&s-EaUK_7Bla=hPqrr(E?S5MhFCavHoe@je&^nNRf3$0gt zBgV=dKEh?>p;%MBMZ(3?@V+-k4EcDUpsvC4BEZaM^HgAdyy#tL60@IL3xODfR`&Dvl4 zGe)`@Oj^;(&|lP{Y`d*LDz98SyUku(dYbM;<5L|ARIEk> zwziUy@tpC9)F_JyJ-W#e2SWNp{;vJ=H-Wb|Kqz&-G`&f}g-#s0FwrihblMspY_?N#suR+!Ir>=AFtBMWGByNk}uUFifb^P0eOk~Un!ALB-2ryJ#@p&#? z8`FbnbJdZ5Pi-pxdq-0|T`f}~006EiEc>+8fa6H{VyjrqU-9pt3F zs_VjTU*`W+_tjr*HBq}1cZx&t;9lIJEl?;f#e+*ri-h7{oZuAqQsBkif;+(}QY>h2 zw_?R^==ZJlzU!|058SNt)5+w_p4rdLJbQBH*+Cs4rj>(&5p{cXOr!;MAInw=KO6dZCbg}X)tEQ13G6rhdcj$> zI{K#tHYFo|X{)R^K>6C>cBJ8n?QLZ#?fZzj72=7tX!O%%p1csp+U>T4=-w)2nkoJz&1 z45S?RqaX&-Rc@ei39{1bXmV>jKQz)nn25A;UyiWV+GGMte#-Aog!+DdD^Hdbd=}aM zj)!U6eJ%1HZBJ^0Q#O*afaiisJVFfYE#$|t-TP-M0h$48-`6QhK90(R729ah{Y21NXM7ap$b*y)mnE_N0kKg^+y6U97%75eP2*pT9XS zSBhr%IuR)y`Wm?3wP)a5hS7Fje?8^)ED*VAIioW4{|EQoyuH4^B+YJ!y*LmttNbtK zYwd6s5xK~d8|^`H0=L1OoB$s@oDl=S9O3HQ!4E{$H`%zs`ZoE%2S4go!|yZGe3uz( zHqPcxv$-d$*Yyht_G#bR-A1WfI-EoWE+-iPMDZX0@ zCz~%Unh6UM`2;#L4|)ZZNtd_f#=bIjC{TDEmp3gSsDn6L=9y<25Elpi zSu-p_kEgz)0Hs^8lJZ$$J;hDa?s?|>2$43D()8E=oa6g7foc??`3+k2s(+<}q=T$0 zqHR>i*!OszL-MTUO&0Y0pvTJx@KqjS#41wIVLzr>i6S#DuMlz4$SvmaZ7Lw~&VdUN zcFWk0XsIvyv0sg6)O4GG-`|bv31>E&#!5c(4`uSNWkI1DPiCxK2$C2RzB7DyKJ6C7n7^sfS0yLXUH2aUce;ym%X^-r)^s*mH-LB?GkC+myCi`u( z-@#q^)9m`)h+s}uMqe}=xd-=teM&A{_C5x%BcXQTUoO9^LNa^yim0qZssrT>z&7p& z7a_^!xwLq!;wl`D%yjJQ)P9^-(ypF*9U*;|KKwHcs6%^=q>%Pit_36)h`#@!P=zgKEV z<~Ko`AkIfjarv55`@GcC1Xj*Uo`33hykdd8Rdo!(zIuD04PD0cjX=GL=l#i3I{NOd zZ)(Xe%G)?RTImIPx^b6_Z&f+afU@5>2^Yt0XN6ygZ7%JAaT!cE*51A?wZRa}1TImB zYG+s0Ni!jf=3DF}MM}gnPm)u?%M-UtE-r=!S(5M9h3%?_5-3eEuJ(^3_O9~v)FN6{ zWx&|ffPaAaF}_-xmL6Ig_dcegz|Py>-Z$xde)`d-Y7c&kjZ@uM*Ab;U(i7zH=z3a0 z$b9LWg1Wm`Jc1)0O9LOLT(`oN1?<`x$Wr*k3Pxc&> zt>Hhpo9Ia}T<4O~ge&c?ZmO|;^mu@e`<1qUfZc2N`J-*RW>?H419@p)?~esrH#*3V zP^#kVS$PM-Ui2ggf!sr??J(DVA^|3izAFUC8u207wRp}~k0(a*Xs~kAHt5!>c0&f- znbn#cio_`89Txz`4wW6<>73c?)I^Y;eC&X+8P}?8WWG#~NgociV?m`%yn zeGF@TRw}Q~iI4SW_vJ?Zta4%Q{9I2-*>k}+jaTe#5YUOS@)d`&3aBYZWJ9D)mao@8aeUJ{`@Lm0R-j<0^sM;>lBM$U1J=*>WSP7eA zSM~6I3QAX=m zz3V;BpG6&eX~J~f*sETVUjNOJ&5b&H6Dz`in@3Z&clnbGj{Zin{l~DOyNDF3JyX=aaA9Kp&;pUG`dJhW02s!bIt3WKzmOQ zYtLq-;aD4n{a#OTUfW8(3gx5e z>8UQU_KVwW3UF2+KjP@7)ANjDr*ji!joxmS>+YxEwL4X0vWt>45J6W!X9G#j*2o~< z%qzIQdzH-7G?48Al+CcO)IB);xE;(HBR-m45V7jyIWR8Y8?(r!eOhe0+aH?eDe6~> zyc<%!5SM*e+OMre-220x$9Om+*bAd-{rxkyEKb1y0bEO>^#@up!NCMF0lzz0Ya!Cw zkEaZp%j7%Y5+6aM4)z8k0;DiwVS1hgzY>o{>@c;TKcPfWMwmq;hX)xHK(} z^^cE8i=VxyzI^qMBHAWvwTsV`Sp)dn#Z|cp4Z5{W95f-FJkD}g2W-^PK~h;ZFkSNV z@h0UxHxP1UD{pQet3)av6@saJ>5N&A-sj2?KgW^vc%Ic!V0(%jN$*!LDw`u99o&5U zDT_?D>cvGzM?3)L3DQMPrWD1W=b6*|4y{wqzs*_p`i@50SPRp6zOs0bMH-OTdT$nh zWwy|*7scu+|vjlH# z9_n(FLVF;p{XMO&W`TZ&Zm`QBUP`*XHCJ1}!A@b5D8R^>o{C)iGc+gAReC85wzLrL zr>{29rO{!t;Wi{~kDqByv7+SFlO^gYMAdN}hlT|1ThDk~CRTw%rLJ3C)>p!f<2tY% ziSvui5&JXCH_B?_s|>lh2qAqZ=o%aihD|$WWJ!qPZMHQ{XVQ+bo4LnrNH|l*{lFn_h_>L{w}EZL)<%j_{1k@LRvf(zj|a#n zH-6Bm7^t$`Y3(eF`an`IFo$QT(|uWK%2TByc^&aC#Jq>EY_<~13OT-n^TJnimC(sf z=7#IFV~1n{oW>@heIWqdODN8woGsO70U>!6W_xgNbQc%U)I|e_SV?Y#%^b*hYod!t zb};qpYvXL;+SRbb`kPPnk~aEQTwlj8t@pmfCU#6Vf4`KFa;sJ8SoYJB{u$ouMB^wN z<@gCFFIOsxrVz(9hURM2x2>KBFC%6vqSF9<>}8ZA!&)}I8ZeC}AWx&38EQxvugsG} z#$Oe2)X|n*LYhyWB~24*($FE-2R?iw3LMH+$H%&dX%RGJm(N52%kH5lG)Eh)ugK^% z+C^vMZ$GSv4#BTB#l$j)9vns#`Msfs22DdkW;d;N#@)yfEng&X^e3%kGI^o6h}2mj z)ncG(mF3zfNM~P(2F_af#~uk7_C@MCUYLRTHH~NN!?=-TKhTr+&M@Eu#rj$i(l(4X zk${7}$*+7GC^qvpSRPj!2WWu+9IGQ3)j=^(eEs;y8-o2jB(z*=&MXD0KgCAZ-cYdK zA>9(I60iKgBA^YWb+E%6XXma5_+wNg?@Y!6o8#vtDw^6{8WrSYoHyS1qHu(BUU~`k zTJ73wyX^hG!^7+_12RoeMW4D|!ZJQ8j z90oSIU}%?0<}j?J^hN_+%x=h@-KmwkvQ@U!sEI(g;@jXip^VoEM&6UE{fz-+DEED7 zoo=bW(lcY3=d8~SURL?V{f3L`vVzllyZ;}m^+^8MGa8qMSc5Yhl*li zZ`0H_lTTN5e==m~v4%lC<=Ud7YaSv>aZWs>Z)2OH+jZN2fp3-kY4>OlIYobgg1GN6 zzEw53evnW4R@h`U6}(<)%VnW#f)|aOSIijrtyv9Kc4967d)+_d=E7wKnzrXg)WQ27 z+sLXBJ;4qDoz`naewJ-+=M?yQ5ZQS7G34^fa=fE?CS13`W>`L>?IHW#x;s_9itlEo zR%h-@i585qZvrH)SkyrDj>#cP4lbP4s+!mUh9|RG@RLWL_HJ&=V_G18pyTY_M-j+{V&xG9H&9T00$IUpkm|}+a z3;k%?67YYpPQO4LDV9rkjIXn~HD4@vUs%|;&Ls+kR$c0&T>?u&cvB%9V)31- z%nkx6jbLPvdWf&_>b21g0E6#|CaCcRD4VFTf+X(+@>piOY}X=aKWJsR+VTr4Y}Gi9 zETeg&)7|P@K6El8HPsn@epyml+q>zfA;&>T3IGlFV^Mq8Nkx{ytZXe+m>nc*9F?CF zi}t+SA|Zxs^tm8kF@Y!>8_Z+^b6@9gxS9RR|v)_Pr zJvnpGBfwkB$(zYBp@5tAX@OX2cGUjQ<7$Z@ry!{%l-d2dpSij92*IahtwE&dP@dH@r1tds`SQ)1@b@Rmcv~08SuO>TmCecy(-B;OCOYN!HfiH!Tx}N3jJjZ*%koWXw9L=sp zt|gZw{j~+9A2b9U>bcvB9AZ{C+E{>YH><~9KDpg4z1y8qB%*{7X*Np~2v;*aauK2a zi^q|q&c3x?w}!&ipP#%xki;M(PJ$J6#v4U9zZ8eXUAN?g&_2zW#?+&7MD3?uY7b07*3-H5nHxa@v+z}vJ6oXKrQzEV}27EOFO-->phSl%6G^Eml&ZPV$wTCm_!KTpv?S9E)WD(k? zvC=xRRzDJ5_a5>cXh|1;`+h#;bwC{QyYOC4?0Mt)eGseyw5k~c5OdhNY| ze)ClHtaM9Qeb|?u+;0TZW?u1LX7YBk-quuYy%JViEP17%*P+zcHu9EhVXIp&Md>Fq zKU7F^e7U@LOaQIOVUHHpGrM+reMea5B1dbY!79K6U799)TWWVzwslsWV&W-_X-$-s z``Vb>{IFWdPSk+CF=T?2a=LH(UJ9l5i)Ky)q;oU54>iV|vT^=DBP}%+(S;3+loZr6 zQCbj34YZP6ZdAQ?0?sQNi`ld{zvbF_K${m?`pujudReZ3fF*>K#~Y(6@`(Tj>L!!W zak&Fwno(I>hnMtoPcp|iojow?#$Y#(XMBNE`>g_ z=f*BXXi1zd4h<_w)!a@>JDO(uvx`SvaUKD|**#m_t@k#lJ~9mxc;^;bNbv7Ol8QhG zDJ5F@96f)M!D7jIC(Al<+JaLF5=`A&zRFNO6F@=q?i10jxK|!9OozaPz3={jCD|Dv~=iWHLbD?GATNWU7ISK}^`_nPy|z9A!~ z3m!d!JnhHmm<{i8kcyGH13gx!;4uwm?|B!&CeAf!hO1n+!c_mul2lf|k@NoOL~H(2 z{iuc25J~TZ>B6yR_-4+7i-7Jx5!!V7E4{5H9uY596dvoEA9eu`q)o#YwrDe(br5-p z?Ke|9ovzQDOHIPe)K|*o%w3V>iS`M$2hap@NCCVwU-&kCS`q4sRfaqz`&J$90#wyo z8p6VZ?KR2^PQNru)4BsA@JS(A0H2h}Pdeyn1ub{q$%Isw(yd$A19gG3c|HAlQ@$WC zl=J{ghBs{d`APX@zkN&;nLuQ<ia};$4dY2E4*_Xoft4<&{pGW9p)lge8m}G0AqByc~YY)^h~E$>2uJW zf3;kgkpWZJ%@^-6(g?`rp5IIa#0k@NJOpvb-Z&S!de&g}>w(MnAZ~8fVLqEu1pkz_ z$s}SFdai@q#KVq5Bnpl0ervFPIWFK7u>M*2o2)_#=Hj!z^Z@gT9z?Q$%DXV#GA}I- zyQt%9P1?9gC|{{6#}25!50jq1;>xQcWtAV&x8EarooA-GJVP_q*EjtsYpS35yZ$Oj zF*pj)@Sw;uB&URlLzWI>grqDLrHR`Ni~eqUw5Y~TGh#X$S>oU#Biwktn;ta(qU_r} z+$F%~re<@01TI{*xZgGJt@Gys&buh;-(kpURoS#xWf!Ypq8{KUv+})@dnF;@Tl(r8 zcfVcFO?J-k@r?an(~d=eFukej?F{HlA~7?oKA@sO(nAxipW5eHXJY+>?A-+PkX#059Y*cT) z`4|U~=+l0Kj&`&(il5n{S8iHMA6z+W;TCuI%maV68OPV~ozj~&lmTMRC>c85vXRTpq{h!=655dw238#-ov%Px0!vaIbg zERUU)`dhvMC<(a_L}entOvEw`)sYW^jzrPtqTED#EE?>FuOz7Hti9f?O)J^J7$AC7hI zmu3Oj)b5nzHWiNhtYXf(v1hQfL|DYZOds2OWeqGf=K8IoE0Zd3ayIvKxLnpYW3xP5 zLx}x}ZH;PIf)ZxW;*vE3;Su7fL0J?pUR0H3m{X_+KjB=xt7t%hBpyVNp6)ZiUi3!E zi%`vYyr(%R&ErEJj{lz#@FVA&t26jqYZG0EdY3svZ|Zb3n5Oh4oy~qfG?kv2sAxW7 z&_Y-mZXh&THdtNzJ*YAy4Vu$>JniTC*xnnWmm=X>vHHW_!A>eWu8eIe1>u2$YQLFMcYi4MS=LzDO1|x-Sid4%xsu~c zh_f^kR;>*O*vj~ggL4L}AK^NWgdWrLqs%Xf zIzsf%4|-qK=`H6y8#Q|oYA%WQH-JXYqaf0x48P3n5Q!QyL%P7@9^ff@8__}`GKCG& z%s*N!0NRPAhmd+pdH6?C`5RLig?yE*>M(vWY}IyGjnhG5@TW;}Sve92P_I^*jb-)| zWU(NyR@vm#)i~AY><^^mM=zD`(6wW@{qgES<#K%cIz)1|a?|tC-synCj}5104m74S9r&6fgSesn(S@hEQijhs^p}hC==Yg!cJL`{oLi>IRt2E)Bm)U%O zXYT;bs2E&xGx~^!2b166X;*i&iG_uLp>!52&(ioU_8wzBLI?1oaocgOg+|HF(oGI!KbiY zY79w;-DgMRu)Ug8j?2~#@FA>U@S%e{n zl9^K7`4I}nkG3zdC@PjiWqbvpO3yq0= za?c+J7jFSX=F#Dy;W=`ZnDlv$TD^|Po5IcZ2TpAOYu~mSUTiD>92+dLjlr!x9|vMV zgs&8yTK5uhZBagXjRJ9<<37!1h+9#fp--b0ajRdl2T|)&LCb7t2KLB8>)73-;B o>W8N>^;B>~;Q#SB?A1pU`|btPx-s(}B*f>Pyqa9a8?(Uw14iAxvj6}9 literal 0 HcmV?d00001 diff --git a/doc/BGP/img/use-case.png b/doc/BGP/img/use-case.png new file mode 100644 index 0000000000000000000000000000000000000000..33934b271440c6d1081574b3fb07b182b027e9e1 GIT binary patch literal 23827 zcmd42WmHsc8#b(hAV`U%l+q3Y64E80bjb`|(nv{zlynJ53`lnkCEZ;zNOwyN$RII8 z_qXxBpXd3$wZ6aaTJH}QYwxx9wa@Fkj^peJQ&pD1$9;l(=gu8`Ia%psUt z&x3ow9XU~3Tj1Yar*|@vcglvSw}1;wi`Po8@7$?~et7x*K5&imQC8dO&K-iT+rPVg z_JyW*?&uZDNxufW8|<`PsFS$0-rQVyDyN-m*xGvg2)KOM_9`ASUrWDYOj)cN{nKyD z&sEl7c}&6kWbcXnU8#5qluuYTia3CkT#bZ=BRljd{m+jyrgs_wOeLA&CxXtxeE7Vg ziMu{+p6St_$ljdbb^UZc$yvTao-|$DP_hIP(F=kFp9RViLX;VqDe#u;Ew9De=$T5N z#y%D_bF=pDVqieVDSn>r$S_T>FY~5RG-&jho?h3?kPdBEMM%pF-P5OXGrTVE*%UiN zBh#x+;4tBV4-?cvhIL*RsEl_*YdAzL=h!@|j>WF11d(Qf>-C>z%Y$=evM>F&$3w{Bdk=f3tSrzx-3Y%Gt9chz&aTq1TNZev(?2T*Fls=e}QHNPo5h zbDJFMJmDw(X8A>h!NFFUe9G4)_q0Eb1)0LoZB^)KENE%DC!ix@e->12Z?E4nxx`OI z?QzM2c6UDia((Kg?P9dq-{gC_eo1~2vA@}4cEY#I7Fo%4t+yIVzfA?9X~F0SZDJUe z^@{Ct!J1}nnvqF)?2ri$3(C}vG(YVe=1gPz@Q3PDjjP_T`;q&lb|?OR1i#L8MbT5l zKyg5q_*Qr6plKMxr?BDpBu2YGs1eSU)aND3X)bAeEc6o4$HhNdowo-eRuQEgpvj4&!s5xy{n@IGkC;Ne2^fy7mNpa3>R*_@FCchTNbEJi&cHsL1a%)+_j zXP1S79b{5v=cDN88%AFX4KvN*wa{*S5wW=LNKx9V?wC_1v6zBIZ*h)=ul&5cB|eUl z)czor@l3q-vz?1&&RHLqn}#_mH=*EQ-{c?Pl4H13AisVGVHdYGaJikm$;&Riv^~jL zI+;QwEZ}RgI50jU&RBOyt8qyJp=8UJd_woa&B#_?fCZBR=}~S|8wgJw>=55~+;Xdr zX=Q?6^*^^WI6cej(RFmfwQp-H%Ip^xDDIAtmZj#}M%^mK0V9l|O$5iHd;Gj@4M<}Tr9^s%mOdo~Jsht^af(alg3X0V=F;vM z3dSf3cRdSReO}_|&NVeTk=S%UI?hiz@1mEI>sIMgm9rkFWWnRoJY6CqY7KUO({fpU z*~W{qWV~Kj)3NT_@Elr*P-NCy?WL_#U;PnRF!#mquv+qW0Y`kXA=k4hDJaV;8o#oj z3~~`9Riyf=P_%VUj5`T!wDzhj9=d*vErF7(-WuvR8)%#vK+&q~Li-vPu}peYh`RQTB`Tr0qJry)ilv*o5Y5O$ zf#*{#SYBJf|0zuTq<&~|j#2XVOTgZyfW5^cqaeZ$XB&dF@FD5BUM}l%Ns@a`Vd4k% zL%(FH{{;fq*ZPV+lQ;q5fbK^a(c6r){eDi9B`J@K0X@WM`1S!j>0x%jh#qEiFp2Ja zaf=|CRcO6mUrdJ2HNw8w$@5~SAV}*0B;6S}_BSa9Fj|ptSueh5U8et`4fu#(pGkr{ zV~?AVKyauQF7WiaVN;yzdcrGGD1d}-N2iL!{Kxg`#L?@}cs;{$DHlqjSj$56`aLD| zbb_Jf4h0`4AcZxfSpS-k^|moUjeicaz>4SWS44`hzLU4ji(C#INn&7ey@k>v@5Z&{ zph$CpM+Iq)Qju_5Wwc+p>e>iT5W-SlnUqC?2RI_^OjgSqHG%uVTj z6alDm-rDKr3=v66$s}?mM@#^`rpHz~>$soq2XMUX{|_3BOKU%3cQnHXc3UbJoNB5y z^ce#gAHYPyPK|xllpPLCqgjH0ckQCPp%Msuu^Yl5H$!~{<&!6BiMY?gK7&-~uPdIN z*mW@+5X%^U6)^sFOZHJ&5!~nqP6wuNc-PhWxQLYue0DIEk?3|=z>1vUUltzKD=*A3 z{)_C;L1Inx=u~II)W>N@L#K;W5*3JTT&3QFFglos_HUnwi(YudBY>fL3@BESTQl{~ zS+LmIGAmG5^soK6x!Pk`k~eG5?n+*&SPUK z@13vaaN)mbZ@z`i7h+Avid>h?QRAt;us5m*j@Ta z1VG+srT_4;sx!#iJr??dK@pdL$ z|IJDy66gPe>lpo2`Hk{6J@@gO=tvKB&H_^0ppDWMOCMrRaD!*P?zV2_6|#g_hWMF7 zcS)Z`>_1eakME8Nm_6YcUh3i$t^^&FL0}3BQt&9^s0|heNoZ3@*dqziidVfZfVk#i z3}IgdR=;=q0w}Du-+LOxB?>E0q(r*zQQ-2kjOQD0a;%OmcFr0g?)X8x&5*SvKc|LtM_I z7_q__uxJ7MOdi?I;&E##7Z@Fwx^l#~H^5WkI;|~6AJ~~la9iqva?a%b0Q$;g7&+Q) z6iI7^{zXrQ+&}sa{Z7pP!%)*8^*CYBWob(Ad3rb~Lr6)M7kK5@*iqAbTZ?E_ch0{1 z&;R1Iz9#8~gET=jLvd@XPl5R-!#!d3vtA(;(~siMHnQN0?SP->FtB%s1MtsL-VHTP zFB0@Ui)vhF<`F@kmg1*(ZQ51*l*|qSxnU^Ok;iLb=esYI{IzN&r4ese)!Y{XyXX0T za&o#Rc0);)4hv{mljWw&p!nr4PDa)lbqXD~P_OFR>eqW~;CBDnKHjbCpJBXqvfTgO z$kUcozOg4LrOsF;UGA9HUA`xLkm5~!J8uB70CImMuh#@Zpn0XPELS&HaD5Q~NcR6!B2|4k_(3S$aWzD^C_7wipI zW$FL5cG*RFtRPcsGof8y zFZs8TI-TsO<49OXh8WI73x*PQaP$fxOE`$gI0c3dpBms{T7=00yW${3}XP(Owh0`DqyEbY+cC@kIkI5k-lT+FKKadu=u3f;aKvT4SE zir3Y`9C`bvCu@{k^L}xO?G)uPj9e8KsH8pZCIZ&xuC!}y{u!X8pM+72FFbQ!L}8sWLW}P^@E3W>Or>2~L*V{A%x(3qOUds5N||TE z5o_+sh`j`NWOFI<5c6z$*XwM@Hmw8tU*p50Ad+gBj^|yhekQGECb_KAb@gsynNFER zE4T}!iTfgLyr@~-<97pV&v1cZQVVtvcQRd*tt$^pl_XRFE2@44RSCy(Ij0=w`^lp4 zpWp~J%UEU_4$n}e&?_qY!?sMJA*i#e+HhRGxj&BD0*j#{&g)_Y^gXLKwAJ1o94uX$ zY7{!2WP*RL#sBM$txHP>s;+}#3Q0+&EhH?e4or`>i*dP!m$qr}khI|-vlF3dbCS1{ zKbBn3Q&ig`>YR}Zp=7WH2(^fQ<%uFx;MS)4b-y{NkDVG?v3J@w)K3q|Xbkze*=5W0 zhG|qkF*WBbjIz+N4z(DBKD4(djJ(VR!h^X+OGetDzBuXCxy!=?PRkU!pRMI7_RN-} zZPkZ!7rHsEMjF~?U4UOuunrMY`ax3Yi1R=xQ~BGk)k&BK8;2=5KVjsTL8OB)Q`8SO z2ir_Eg-t6%G1_LS)Q#U`$}Hin?#tOvg*I}Za0XtOG8a0Iss-P>Qgzo%;d)(t+I{y=xSANpNncGl<=sdZ^pz}(J8gcY zcA@|{jnsN8y;49j(?Z9r*Cy2L0@Z8rSrW`0hU{~(*k**{v`z0uTT$($(Y5;Ol{>Ob zfp)iUb(cGgjgt22OeI6_6t!vSd|&~OonXTkYpeVWKxZGlbqOo9iE0}=l3-%I2jSoa z94X*^!(4wX2QMZ*9GIt9vi~Qvj6a?DBcgq!pQXv~H1|i$Wi!NIZj&8~KA9X7a!LMr znhEbg>^JCgavoHlV<4*qTd%SXRb3;Mm*tKB8;U8iSm_9d7#c$M2jb5fQ&|gz-5X6m zU|fcDHnr)l7L{5^r7+%s>8Zx=h|FOiS7KtOHS{QaMDpYQntqi0!Yf)$bjL=vr;>ML z(j2vb=GA1(Y7*}Aiy^SF{o>@{!_u$rS@1WQGRxxxTdB}XYqzoJ#dPiempgVx%j)%S zNY!yBR3ptnTQqprJV)@NhX`b4Qj{tyW=hpP<<$@nIf2d0zI(O41__zn&$O%nI3th& zr4sXJSa~u@t^yA3rg2p=V3=42JD}r@Rr=gHnM*RjaEXN6L~1cNg|yIMAk4f-!Ci~_ zPY}wxS!oHO{Y~@cq4z;Y%2xq`=p^@n@YRp0I1(cD1*{rpDX&FqongX z#J_(uIYn#Xu#n)EEOtp84j|&5`iw6V)iHSiM_lw8F z?)1(1o}PyEiC^2iTg9R+=#J#^^=@rrxciFBMCZy+GvPqvV>b!70mB)yaRnt>U`#n? z^%DpJaOLM&9Tal(wp6cr+4CFA!dyNYW5E{eKK{DwZQS>(A~m@ud{2L`Y&c`pLH%#q zHLg9DpWA3Mscfyl7TA3UV#g1hF&nph3Dta>>+kcaoXOEoKx(kNf9e)w5LXxWQr3EN zKb!sRZ+^y3TY4>#J(T^pArroc;YwI=Q0!CJeUM1{hOkM7)xj38?5#C7{^%LkspLK)ZX3NiC8`M>!tz1(Yu2cgl+rHjUX@R!L?t<$3 z(3d*@xL4Vdor+x3dVSyMDgjoc5RI{0dS#BOzWH-DiGC~yhiFXi=>~>b7eh2Osg+sW z#$auGvOzeupIAnZ&0)PD!}siJ)#h?ykUuGheAj-#TRX53msix<;xz0?eK@z zJPNICb2Fm{T3qhly1wr{=JNyiZw}>Lk$%~EThsm~T(w$xKQ52X23lWyt6N?=!UFRB=UFso*v?3l?_)&kR} zY@1tQaN*|ha5Dtbw%L8uR`x^9E*5}32P0{NTxUyTuW!DAUz_t=1@LVq_bq}7RdA}@ z*SnaA;u_5z6fz8^nFsJL{>_{2fa($h?)2*felGK)uHlHzhU2np)#=m+C=xziHavaU zZE%1vqouwRHX@F>_g*=CvlIFRBW2n{7e&1^B<_(3XP2fNqny_lvdy1dMy+i4OYB*8 zf5uUqZ5eX$v`3oHJ^mwUpzHA7(o*&+^|3*#?)C+c3;FbMD9zSg2_i0miolPsRBkU) zq&iE06kT;tg-I_hiIRy+nba?$_fu>sdqN&@{E-T@g^tC)A2_!)!egJMk&Rvzx^)CM zKjcq#;Zu}d()JY`!>r|z;am_;*N6FZO79mI*dO?DKM-Zu&ab+Es9kOoM=@;nv|C+ElQvI10@s+8~v{WU^|ic zE=Kp7BJWUr_6pJT?OKb>os#w*TSEYR+2JL1~k0BYU{;2+p^c{?xLCziRNe-khm zL}x$P1c5}8cTZO`mX2Vh?A?0od%DB#jASn0=#S*&wJ~7~stEsB&zcQ{nJa&+&&GbMat8|AGZ9b#P@bKnIt@Qg zZ8STuqqrF|BImXzl{sjlgy{(w{j6p{oE9ky<6}C^s$Z;-)BfICR9dk^qY|Ty4t#BX z3ahCWI5~)Pw5|jUCfbxm4@(0<9uQW-3;hFAUvUL+J6q-tZOBxy86sNF79T>=?_@3p zlJo`9li+s9$oJgA74X{CFjm=KIU-Ewkdy6`;q6s1_@GtIB+f0Sj=6u=K`*GBK)G%Q zR>i`h(Nfv$JU6uAE^cqDx;t1t^vup56irU8I>4RF1N!A4ORamo^C9P1kb`4emIr|E zMGKgTFgZ;2vWAqLAYD!|F)J{@?S9t0ww0Ptib~UgYVr>C;GTRTx?VNdIeQDHEJLVs zXt^J*u3V7;1V^cz^S40(U}4R!7}82Cv~#sv2Gb4ZA1TpYT&YbVc7+Uy^&2p+L{!BO6|^aJ(q%_0nDSljYzaI!&RyqxA| ztD*b2Z+Q{n)Z1~Y@Ds3GI;+i@jr~z^u_z}8s`<1(@}!Q;kfYRPzc>QFggsGos@3+z z_(^Txba4^9_a?2b*GZ%xHs?$q3qs2?||(cA&K|C$MsF!e&<6v!}GO*>3y*n@p4v!t4Nkj_#z!{@&0`f zPhxIj>QH&CSs;PNw>MS^8IO9#0xQR(;#FQ9M~0sRx%n}HYoaqL*yBa!`x=fa5+(G^ zkSi$C;SUz|mNy=PM$Bz{67n7=o6A6Bmgqn&C{IEvyeP}2n%-gLWew9qQsa=os}ZX0 z&f)Hm&LO#QSD_~x?!ym!)GU~u0M#R)O0kSk9=~JQV13;5=n>#WP3|W6y)q`M7^4Vr z9tp2D%yjRXx%$;+!fAZO)#|Y?Dj}qfEw{S!zV*x=CxhTZ_Kz`26ZxW|!{?F3W$3+qUk6oA$!GVI-?C9H zo94>;m%s6TG5X7Qg*CUnS@|bHj=Lui3a?VND;MTLHMtnMB%v}&+GbKVJ7SllbDg!X zdY{?7o?FB{{dI~2_;lZz?Yx&uj4>i1Coi_=2J)$oO&ITwIA*gVF2QrYKltSFTAj=l zuc1x&yp8YK@?7y;U{3e%)OPAh&OesH`dZ{-ahZQ#vd`hpF|~XcPKXSUJB)(KKiHE? z^+yri2kJKxa+YW|e1jR>M(5o0JFlh2fj}eE=6zGVk$uRXycKNi$5W3+#7h0hhA3+V zK5PL5L~z|a&OaFCCnYP!iplWG`7w?u-3d>p;byr+f7Ae(napi<9=5g=*%LcqdTfsD z2DjwzcH73r(=g5SAce||`pmaJ8yFLU&;GE&-mxft#;RP^pn7|d;}Yg;dbEF@6bg1E z3g8HEtZ>wHpAA!m^rdfDx}JVvJ&wVTv_&ome6HCn9W@NTpW&b!J}N31Mxu(S`ke5n z76}*{dr_$ zhpoRTid@j@=uDoQ_1ezj_L|`$XYXbV1o7X~ef1<|zcLBeB$vzXO@>1rNmP!Bucs2B;S{! zmW`FO;&$1mcFlL2xe4}ntiG(Zv7SdcQ#w!Stp)@|UDc&QTwcT+Xm%#Pj&>^982tm+ zOi28&So&qdqDt>wA$rU2hk=3laYB;04=f0hOX~ZC~ z>@;KYDeg$8rR|IDL_0RaXxViMg9wEfH2L~0UUyxNMd~DIn+Ae1uxuh{xOc1`sNClc zGJPzjv#zE>=PGEq#kQp8N{VyCDfjR&auN#^G6LNr3ny=83=-wnS%uU@A`eebfpXz@ z5BoOiCVHU0HXV>4!1rdpOf-V8K98z-mE1P5WBd$a8S7LS>S9FtZisY#>Pb6@mxXB| zQGAFe`|Cya#QyeoeGVFM11)bj=djnJ8U+!)ZgukHmey!OXAZKOmVN&=xCIP>MPXH_ zlo6uCzujjdoQLSKg>=`O^u33>QBhGaryOkT5AGpI%ouE#2=2(7vaX4;b5v&PqqMqwlH$VJ@@H~b zg^uc(qCq*zHh!m+QdakT-{~0?=#7x3P*zh}#60nhA`h%*(8q#yB#2l&7i)wGB|M!9 z!Q`6^k)fp&-cvsz5y9qfAxuVI<5x0zJ(M0Be2KmcaoIx>X1eU zzTP#=8wXhas9mR@pzbmUhSE_HyK%`Khuz$YYcKEY&dY|8oY^|(w85|!s)&_mX_lMH zJ8%8*aj@IeUiG5r_g_5G=E~>?H{ux&XH`#(9By#gk5vL(i!UdIxL{PL4U&7WG&rx( zTJOI5`Ue!kYca<3L>fqG&7Z8fGhb3s8bnWjL7FmOd+ts?u}yqUO_DKG-pSE@Ep?Zk zfDAGj$>!+JY9@2qD3z15@>qm=kn^EjDVi>ua!io@RM*VnGdq3?^#t9HdGO}ia0Nzt z`W?yW275dQ{yCVCe+^Tu--W=(z_bd@LRcEA8bAGDDdD51vCRvSV9Pge=_5|N>0>n! zmJUX7p$)|Ty)k2<2C)YIiOD({qkd!wV`fCYg+ug=pW^d{G;+TQN1Ny`x^R18J&3Zi z+ELw9-WaW7OR00i%=B@!CCOlf2zrG>w1smYm(mu_Uha#OY@a%<;hOh3>thYxJe2$y zu+~r-VwYSFGeInTuOr!JjACX;d_U?>9@yPBP}BIPb`5BtufQ~cInbH&yID;|qrM9+mDybZ{CKxf+X))af0A}uI z--k9w&Dj|(cV6)?!>$cKZkKD6dHbgJ9D~m15!r4vG+`!qdM)Xc?XtRb`tGww-^bF2 z%5~6x#ELEDGK=iIt}oxdxzeTT)a|Gqbow0pBb6ivE##1D3KpFEVOe^o1O;y_D|XJg zYbVl78x>px!Q0m3l$JN8XFux?zsqeg-k~{;A8kVkc>p7m}`ii`En(I+IsSVAZE&8W(a=d^tH>}xi zmAh7*YVGCGz&L(+-#CeP&fO>(!I{_}1kED$O776Nardzk>k6pE{U)z3p>iz8S=ZVT zd)?1t)eu@TE|pOu;x9v8&nKqB%NrIityCm)B{U&@C{`Vw7${vzzo%~J0bx3XGobZZ z6$yT31=pOObgU$z&j~aduff7RHM)v!B+LLl+!!Gt=0JW*1?&SNt4`Vs{{;BaD zrNPZ)g<~~**6b@0&mUo@?W3A>!rGv^%sT!+-(3-nw@D9!QXoFNV~fjI_U<)ZDsf%g zIM399GG#Yc@mK>}s!FH$Nh&OJu;*C`8ZHb$*sY&DPbW`|Z*CkrK*YRrNnm4r&Ym&1 z{5iCLtuI<(06xLQBCc7UxFZZXM;(s|X1Eh++G-Bh``)xR?&s2%pvNl1Roukpm$^F& z3hQk?eqFR$zoURVpiqZN`M&B*!V2P213c&tN(A~;(y9o@AdtnMhY${+>aDDeSIuS^y3#XAem1nijY-1DonL48E7A(~RcsfKUhQt7G3W4H>O$NRSkg~VO9^FO}MJUw`PWmBCnL8R$bq;k(M zh<4ga|m2KfA((PtQeTlCgOu+#-WhztGW-JD{?w6dh~Q znDj0flo-0M>QN7giX5N;m5W!@Y#GTj>Ze>+*&mhDZ*5sZ$V-hFBj8*IR!S{)`MJQN z^pxPJq_&BPVF#ts%Y2rv=dl{gG1cBYH~n#68!hiMJO|Sr*Xs^>&;tmoH$c3PZbGE8 zWRqy!vsH?ox@j6(`_Y9BRY@0716Hm#M9hEa)gPzT1dtcl1Zmybt+$?Alc%q7^xfHM zoot)>)NHby>Y$dg-o68xK2iO%nD6mdiq65mmrBs(t-0~^rDg_i>x}v4qwO{vUH!Wp z;D+s6Jr96)&n+PY$vvS7vEoo;R2vH2d7_e^_mPM{x;{1k}g0S zy_`u$@qu?++@F#>Ik4dPv$qANdyEIuyCnmle9y#kX=ffvnQV1IiYLGK=FqEaSw5H7 z-?Qf29LG=VL-C#dc@*FcvX)hrbdMS62G#d#s=Qcy&f_@+9;}M_o9nfr3KCNs2

n$h}spUXcVh zc>T)*b1(s>$eG-Kr7?boh(z39YL=3m)cU)&LLeoS-0S$*LZNLGAXbWEw_ys z90{=M)*O6OYp_5p0NmRGG}oOA_Y-$|uTo%`Mg;UF{$n`~KqnhCkC3r1{`ch#$dXrG zHJ7kds-qx;<6j0hLOGk$V@p%|1&9wo~>?)7ltr-gk;)TE*$Z%T?4L_>dHNwxbF)d{BP61GEH;3 z$tN6S>%kfSEMu$zZA*{UZ$sC`c+T()F@vB$I7pSBDYo?iyIvzW=%d@<&)G#JM|36{qa&di6%QJ4^^oyj69UU4I z)DdVv%usDetJ|5pG+-q`qcFYHZH9OMx$pDgEZa824zQJ}fWX+^);G}-AQe%)Mk9r= z0ral<6}8(s(gmjXB64hazvPgG%xYV;}y|{7O!1M=`8EGR^q0;#D z-lCYB1i$*ak##5E(I;e7^s(KjsOi_Wpo^-Ap)K?9q>YV@!t<~?Y6Du!GPh=H4=se9 znZCJ|tv8TFE{1=J1^Oe(o0_7?Ep<8uX}n|t&WF|RlQ@1IwNA(Hjy!ecD_-ey6d8DI z?C7&Z3~m>fY`|ly+HNi{Fa6O@i_wh}4P~D}b<~9m(+g#nOX4rT0foITPLUsGD(uYA zvp@j_>ka>O0zbQjG7MeZ+Qc_jc{3k5^cv>p$y==mnw!~JOooeeH;!JtoVghW*4>;X zhuREZt@oxWNJ&W{ySUdx6Gyc?7Ex-F{n_XTmU`5~MbZp3(b*kQYlcU6B~vp+y?0d& z%Y~ld&2W5oF{0{7@ZZTRZj$jf@)9Az_4e`UfL6i^!34FN9Fc`0v5WV&?H1V{t-lok z3JkhK)u8ClAp{vsM@Z0ZuV{W!Od}(V;n3#tWMt->&Jo|7Jz>)H7}d5Pwd#|>VJe3+fTv%XnJ#*nq+u7Er`18y6hhe# zORs7kXzsTsM<3DY-mSfF^9Z@{oK*=O6O%;&ru~f^nbP{2w|T7vF??3j2UL*-nIUDL z8RSdSQV8SU+rIV8Wf&XpJNsbRYKh192-6JDsFZZF8EHbi*#&wsyUGj&yn!_@Y}UUB~3xLgm)3; zKD_f?*;Gs6v5UOnJR>$&vDXI$S=W?@j&ImuA|K)jbY~Imgs53;eS6sp#H?a{d6O2*hn+s>}riaR?M(LsqN05T) z>iv&%8FijKj5?J+m5`L{?u2p)+;4)9d}TEh>~W8R z*28O7y;zU*>B$HDa$E_;v2SUk&V4z`-QmaCBtf)Qi)z{QqHaaC?fArHj^EopT6JQ{f+lxBFe%RC&82OMD4_+3^$J*jnV_nqT?Wap=#nupsIQ@lra z2WFskl}GO*giv^FV1oTEZnOq6>P~kEsE!O9LYviChGX6A=s#g3!YikIuZT19TdPne z!}hz(Vg^ySt%KvF zLkyXptujyquFKo~1uexou2x6 zQVyycBn+lq8SPFpTw3hbeJ?9z%Ks%V*sMS}iLy&o#gl@P(jeoi7S;du0>XioP<>?A zr~)0pwhwwHz#L(suWRriDtyL%?&tmP9ZkV@a<&Z#RoH^g0u242nKan)o46T1+iS#- z-Q~D{@80GnIi3}Vk{Ekb;q4K3ZDixqWW!5)jzF62wd7kP!F)#CUZ$h%Zc+APu&CVe zC_-XCtMUK=(gHBSB~7U(F5iP!&Y_^X1V%LvOx{Ca@NvJWhtho2uDYvfe^3Up_33xh zvb%HMXHWNegsgOcPYdJos4AF=Ex{LM`%9}Uj$)NCN!KCb? zGM@V1;RF^Cf_N$_9AhUy{#XrwQZ^F?vCm=V45L}phm6Q~mXVsu=Vcbfk2{U7qxz$Y zsr6o@p@_Noq@cPFA+)a>RFTESo#gVbw0|<;!iyew*~5!MrS&ss-1tgs@+H%Vu+|=> z>qx5#vRbp2*cN_B@*gY(=7@9((M5o3c+MhQL4($y=#{oR`fUGVWf^jC-J@@7?0s_` zHKp&trW6%TE{)*fGCdEb#p%#3q&87QOi4h8F?Ev7nn`={W}J#c3U}#D39%BZ({LeW zJ-W_Qy3WaGbz?cn&x-sm+S;LZ??fvq6ta6>yHW@*#QQqA~wPf;aaf< zjshRM#yM{EilZ9_z zhDVRlFgdPH7fp7#iO%Z^tm)JI!4|HqnYEXBZUQ2;fqYGN!rr`21BtR=|m(a z&!dobqwJ%^fn4@5yf_?ER&03KyaayuZ zdvV9@Qb%`nGvTE{pt^fI9+K~Y?YfaROa?r#aJa8|X0d4yajX+yTqLr<9+o!4%u}q35}!hU|cGFxlg;VyzjGLmcNATcrC5r zQMFuG@KFjpN%}MvKJPeu*{8$XkJ{#eYED7v+RkgL22FusdAVk1#a7bbD;8N49m@Aj zDAsNrt7c{c?XT%x=I;6EVq3bdwHx{W{#P;SpLaZI8I{gfd&F zrh9NUaue~zubWOV0Tt-ui%^($+Wy;?07V#7vz$a4k)`^(%Bj3($`~q(_ysBv#&^|p@Z2v-P!(Y{3zI-Vh9{D)zxhaj}Pl%+#orY^0RbayO zOlg()wZJng$qw`YQ=Z3Tbm$^@-rJ5+D;%4+^v?mmz_Pht*=n;&%<(N+21PCMc$jy{dJcU4MQipyx@Uet%z?BudlaraP4g22 z^)xmd_2JIv0y7|S2lmE!tgWw_h|H36d1WgN$?%O8g3uW6DkoZOV0aa?r&-=D9y6*` zvk~-GsO3kc*L|?Z2gO8#u%|Wfjup18lV*s<32)tFSSlm&Jn|WrrVaILUZ_cQ{oz8! zs#OBK3Q45__HQYRy6jp3EZ&$?R%M z?6VhD5sa?!tpRM@fxm?*wiZKA9&atO610wq*e^YM9rc^ddO(?UPtFmF;fZxPj{#B6 z#tPX4rjvn2;qtfoNvUG-OtGg}?&9v6?)L5>?ipdA@jkinex++{2r_V7?}hkQgQp;E zP=gdgi~pUIc+?rFKBmc@u-fwx!tK|ORzDDnS|*JJ=k?5|W~pYC*JYR^HHe^Jmf13q zeejOX(j%oBq_kqa6rz23w2{r;EESEgO$IEC3W>ZhxpF*0gHUJR4wcR@z3I44kzG)g z^UBFDKy=dCMyxH9vz3aw{w?CyJCEkNsGM876dflc>J2n7=_f7ZLN=w_FQUQ6sD?i! zK$Vdd`!=8!Xjx;1npn_n65YDwLiEh<=>_DolMj*Kyd_ufW<1DHIQx8>%jYGLg^PeI z{aUpOyvJWe9mr{-`u)AvWswjKkpBYCK`o0{2bGsK3c4tzRrLRB?;})f7vEz)@xFEc zk=m{*m8@Y0NH$y!b-~aA{)Z4dz^hO?=dG___U;#z3R4nUqWEYO9~Gz?==)v(_Blf& z56<#vIP=-9SdJ#Z+~>>_XmN7=9}PmT9OuyvH-CGqIgYt=0kVOwdG7Asd@TIHHjC?srmjDE6wIwGe`nRE@fzZTqN)K zu!`y!s4GHM2CjhvV6PeV5nOVjjD$`XITmNaYoxKHI7SzceiS))7CrpId+fCpUN(v22vYk!iF-fY8j;q3b{Ly=cb*-X@i}oUo?+PUaWry|WMk)7PSL~nx`uB@3 z33VsJW^X@22` zkUH#Lz$U}AgDiO$&=2 zkA8M}>d1yyLPZuyTv*tGYT0}Vu}jBRI*BrnyP8HXSIkEFqs2Sqi00>8NKZGd{>8 zOKjiPLq}bVa&vPxpLKRtP}+^m{ae)Re0Dq87JY;}Ep0#>jR({XYIdW3$`>S_mWxGe zc%1x1+~Z+Pf&0BKmjLvTBq|yhO=444d_;*7FGQQ!I>V@>z$}v(`ZQY&3TuJ~Nr512 zpwO_`sLAc}2*`Dei{L1@CBF8A{+A!2NsrPsa(0m6N8?6UhM#s-vQ+=Px>^UHRlAXQ z#;8swkQX^VtdK;^y`~0+QkkPI!nU`z{#Xv`{rXdbsx}(zJ`bw_Q+}ShlJ)gN3vj}q zsi3}eN64{tqt6Q|j4PV0ay0~5tORskq&itPgn#dAIj>3@5l%_5mWjALzT2l!12Xy{ zpbuQ+_%*$a_vP7#Jz#FLyWBC)s!Lat$mDxEz~0I-LSxDHx#AWrT^0M`^FMVayZY(X zpyB6f_5P6?p4%U+ZG`Io;6UtXxQg|j3*^&0*Tju#p2Xn7+Ht6cPvWeoab$gQ@th5N zvsrGCWHl4PC0C3tnl{OTv$J49W$XyP#<;`*tYWrO$q=3(-&^F~+59=_R2=OaRYSAw zRc-A#2Ju z>IY|YgFd$iZfx|7-&9oKF4l9~sMMrDW7I`JCew(Db&p5uHDvD%2nhzj?Jp-&OY>IkxIb&vhoQreGCJ%AxDlWn=n z8P`$LZ2^a)6y*6lvfK1~WNN)u(9V5brT$XvIW?00634}FD)vK5`Qh|^Y&N@^4X&3| z8e(yiPLe-~ZuFU{Sw z(eZ&F-BBB>;^w|lCYh4GuDGk@Qg+G9+bt#=&G=O!qHfCwUNXq_g43lx3?4JEWl=`+ zjs>s7HLw(Ceqyyc_EnR@M8mG6=ROy^5H#ZB!)HVq9#+~tD|-XA)T}PrOY^H|tTm=< z23KCQPF?CzE#;M$p#k3-1~3yy5)$oj!V?CuBi<>uHEbFA*XSQryb8$q_HsmLID|)rh_eLOJ>~~0Y!qTL^Z?%oAgv$Cljd2==Lr>;}2Dpe^%4ZZltNo z1<5SIwGPfTQ(SoT=}e$(bp~U#9p2h0mnUpWWq1jH`$`MIqbDiNAel_GX;p(CT;^@? z_<=YLEv;$+3z4Z85zwW02-$tH@em>L0m{&C?t}r|?%fVhWu2%(@zWnlWByf5TAua5 z!;Vok7h^tCO~HH6HfV+U5X&@htVv1Wr5O$1?;p3v3x@YFu+Kdk=UtoSpgdf`AUC1| zQ8=kyrsH}|_0GfS)-*hm*98!dmlTS_H_k??zQb)H_>--1Dbl}N|5c`}wNlXDAvrP- zXCcdRJr62=xwbECzq{~jcjF8U?OJPk$Y*I!Me;f|J@oV6CP;v^%g|#W(v-ipxZrc+ zA9exJQBtP&uStAmgR=d_huc@nz4a|_D8sH@x-~@XB>}a@Wu!MbfAAKOd`a7i_&`zJ1)Ni@U*gDQ3C*!>{k@)XQ|mc9P&^F0$W~r*WjE`#=bKjd!RX_$uAKWp zRN!!Os2VHwKc`~6`tAHa%xmuFVx#rSXdOAV&OU(Juf<=YT1J~WUA;d3Ir63UvABvc zQ5R-Vj(7@$aR1=vc`(-b{P^ZO1eS-{BFgQA-QaMM4)xgQ5|G|mg$U%kUmMLA)bhH^E`R6IZ{GsE% zD8!_Nfx>**AN!^Gn~vfXo|--)W(dcS85|gz1~}`63lj;JNV}=>o);%*3+UyXI0I1p zUPZVVDZt%cac?Og+y0LZpvC~v!D?@NMc9HTM@|L0Q6R|iU-^Mx0Q;R`Hr!;u*IQ?Q zGZKKe+p*WZjO&9x5&IX*uABj;Svri@K9QCeYcBLRW*u4gPdS0LfxmBZ%X2_ql$8WG z_nParVHff5Iffy1blRCfDo1rza*L}UDBEvEj8%6y{=Sc z_h;t$83Cx*#f-oi@A)uEH40kb?X^J&r!!vu+%G#%r!#%kr)}~Up>G?hgq;}nQCtPc zGJr-a4I_+g;a*}5oSZDGmQ3SKNz2aeDd(%BqE~B7ea32MKmdp!lozHEnrhS~uWp4F zFn#a(LJ5s_KunE|Ac=~wS{A{=Xn-B@Ohb0Bm!gp>{7L!e9TfC*iLmRWcTsTm0 zd1r~kNnYcRx)-z5Aqk{NRvn1$tLeWVpZMGowr9Xn=T@^$eHUS{zNf| z=qS5Bsa)0!=qwxWyF?T?{{vFyVwaFq`A;+#7%F0Qkk&$k(W$tj*69(Re4bO1r`CHh z)RFE8>5#)PvASCZ(mmHp_PzfZrqZD-y;i7BO%U5GEV*hGI&(A_>udXxBarr1Us564 zxS4*WK;~)0HY(X!g}kl;u3b*mV|2S3cc>X&xh=LUg@q=JB3T>A&3Nl?761-INBOqd zEX*@Nb7&$o$+64FKC)WEHpU6a0!Z-mgtyW@m@kM+c{YG^J({Zl(CjGM zF(^KnjfEO-=9LB^PE65N5*SJOWoFFv%- zFh{;Sd<#v4Wgy!5l&I^kWO3d!d;ibWsow6?`|Qcy{Vu)>?_7ezzAU4rwA>X2ZmT8uxDoybo%@>5Y59{){>jTm_`JoWF)TDjj7@xZxR7n!)H>STotj=@#8#d9)J z6Ou~Dzpp2ziUXVi)&m07(jnhcM*jPBgjfR_^lyVY{U5PbU4`qjOP0or@zvtI*+lX7 zw;ir$96j6eosFIYthr0#47;GE^~9!QA3WC_kJoE2oS&X{to2zc{io1+Bj-SI@sDq7 zMdexR(Wedkr-$o{#bx9ct@1yJ5AAS2i)$%!2rxLJCX^LDezc2>E%`H>4j|^9a48JsV=m#i5K0Wg5#J3x5 zYxlp-UK$CDql5%|M(i*Dr5bFx8+;@i5H z&n^X@s9D{{Hy=Fi(yJB41`2(39UCOEeSoGSdZ^BSF;G+++SXFrvJQQ_l`4uk*SH>@ z;qipPhNSuvp)@!cV{mnp@HBXQK1j8S3*npTz;u3$#?@YIn>u`dhByW9RW%HJE+~qQ_qVLf_n;%<;`;dpR_l-1T3%d z(Y*cvCsx?1H?eh@GR7Eopg1N%h*yn?uiN-*@vl85-z8qJv1@&Qz4w4_m5nx;E||b> zreN63Z};(`9FDU%dBLRP1do`-UJC7I>w__5 zw}1n8=p&K1j1p}w@70Afi$yCDTp=g|kv8Es_N)wTtB$s%t;EsxYI0jGIBZK#= z4p#gCL=F|{LcpMHGO87mXkf^ii6ygU9Y zvZW>8xi!75RWPXG%5|@lDU?({mR{azFB-@2wSNJ7_*8-BD=;lM;WjTG12o=HRXX^j z%!7&OmZ!~FWP&xV8n_AUXxuCFasPG7XqwpWAnn2fGXgh$E>ty>ohj>ZFE@@2c^qei zn3FJSL6s))+LG8i%$zZg`~h;8lE6r9$R30=R1+T(6ZY#X+w;bkGL*Qc5!F!wfh{s& zaej+H!+Rait9&nTuoroEd{0M^xWad@Ttr&=(3X29F-jc>I8VF)S&xT8dAdFzB)C2LVUK&`Ga%Wre1po#t2QJOP`R?3nR-NTF9T>IKwBzCx=%}J~m@A*wdS( zslJ;c#L|5~71Y?B5n<_Ho#A^*hX6!G4!+To#C{?I`ojO$>M!2`r?6Ddn zg@3inH(jFGD$)4U@R$1_JNw1*D~s|gQtvCqUK}winA%id>_Ya&T~%bc(Ze z(weQ7n<_a74g)e(^jzZb+Kyi8XP$3wWYJeD4>`=av3n&Rv*Ycxz21)rpCo{dZp%tp z{j35Z67WSWfxS?VCm6ALkrlKnoL;n(-(*x-&#NymG-mUOcV~Tw%QC%{pb{BT>P{an z%GYsfFO5~h(MI%AteE=#pK}psl#^LCokVQz=Ra{w1qGC{GiQLL8-=%9b_h}bQOknI zyO}n{ILGJmxY7VM8wXmt?xjiH>(rtmQP9eMRA>~lq>y*59V#p9aVgPsh{Wdw;hLBf zK|;JuCuo#JzDvs4;&0*GFBmVzHTd))z}~*H9TGz=0*#XoLKQ#3AqXb=xzNyeMx_)T zNWBb*vbt~izh+S#cNSnjQHsQQHdKbp&<`gF5bkZHIywzyq(;gdzsKk_(T{e#GgcWE z#e`rn2u6Ss+di=I_Q{08Tx8|fW4?Q;Aos}S8!>C>D=RU>oVp_%_B3I`$c-u;v#x>T*1^W$2sL&4V7Ut%)?o3 z1op2Sl8(b0?8h>RR>yC$8IpKR7asCn$UNtx0{$O-DFX_~FOI2A7b-em`?b7iju4)`hMu+sA~-?$?>r|T&KI5SXxW{=a=|4kuEmftf0BU&w%nV!v-1#+ zmGdh)3HK1j%IV~ut!N8}E!ojfl_kUmCFSXF?yfS{Ni4(jCS#D>cPv54rKNQ|ZL7y@ z2R82Y%J8`LGZ!Q$fKD@@aTbUcbB>L&{P^O2D>@qEVnv{QMZQ5`Q-evCG!e1a*t9Db z829n^zLW#{1ZlZW7z+2hah{sW#YMu@{MgJKKLqt)`3I4(5q8M=RU1%R`E{W?S4(=o zRt!7b`v{uIjzZ%ut1KT0k-^Vg>z=vw1eA~Q_~ShkRlW!k@H>VdoNDj4@eMs1csA-= z2<6zgYN&c$$i{WcE!9j?EFw|md9)<jJV8c>2OIekU65?5cOc5=?IbqjT2sT?( z3a{`9%T4Ns=xi0YrOBom`Rk>a@hvISqePno=TEfeDjR#+&d}z$Si?j{L0A%ayYxHCTA9v<$GOKR ziX(Ccv(}mq45V68O%LZ}GRdrHKv;%w~IQ#237liH0B0JK<7_jtVC32ic#Mc>QC zrTxhP%jx!k1F}+n-U8&pJHgiHU76IlwQQ9|*l>4wOXI@o)}qcCsPuTyTqr1ih^^e2 z%~)~BUT4$eLlnW-(xZbmpxOK!NI|8 z^8)Ul#9sdxhxq-gKp_G-+c-#)wnE(Av>Mt>$65CD^h}cC==I2H)8Z>1_c946M?~a1 zTif%P?GQ=6;tA$?DCcWN$z7U+sRBf_mr>k=u%*g4haF%a5jVYNDW3Pv%ze*}8 z|Ec-WDbMZTj+k$D`r~lQC_}tD^O+(TW?Lmfnw^zD`#KpvRy8DxQ|c1LqSZrew0 zsy{10#Q|sk_XD`}5_SQktO%ln|J%O+$^_WGHhG$nQl~R0`hI`sBGu=Wq0yUOrMk{3 zK@b?C1~?!*StCy^@5u#N7B7{$jyAc7X^TRhb6*&x)M*;-m4Vt2#Ao%OAMz&srmKtq zzuz!|$o|K0+ea+}(Z&BFC{j|~2MAJWCV)}9RZ)~{Q#}S@vy6Xv240`dFDrlD6ey|K zsz>sO1^>ebQm%Ax{%%!XO&_GfK-@?WfZy`#WmHAju0gE%$gwQV4lvKRWc{YAXQ&G{ z5PMjXCqn-8-a^ex9y5pH6PV5If;}_5yaA_09|-Hn)ywoANF0-T*tJOvfh{C`B0ATh zW$K^m({ei`FM1W!FREFT;f}0+bni$ue^xJ>6cc3^JzXrwpeLz~Mn_+ZH!;nm<*pAi z+5F~_2!%zYw~7k=*^z=ZO&vl*0{a4vA5~9*Rl+UFx ze}DDm*@0_=Z@~#5e9D-S4?&7>MqEv8Qyneplk}nwd1?GgW?Rf~@Mt-H_ug^cD#gwM zu6l^xx6XHkV%S+veN6JmgFx8uq$%HP9|h8k#+pK1fM=8=UE6_2ZF4pHYFd!V=S zy5OL_`F%&|Sqs>Jlqxug$ZV~s|GzS` z>xpdv>(WmbrzT_E@$|cbCm+(~q$qU95g_EawE%%T5oGmW&T;lQ`$vEOqsse0A(S17 z2D*ciBAB&|8@B~g@65Nv@QSs^&@$fDwKb|jVyWZvPagX$P3NPuFeD5U0sCEvXLWTo zo^UQJy{5+4#93+A$XT~mHDyC`2s1V-6a%+}x9;cN-25v+MX_X9s<;p?GT)}Atf^Fl Hy%O|ac~xj( literal 0 HcmV?d00001 From 8fd23f814815e8d02c0133eb6562a1645862d6d8 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Sun, 9 Oct 2022 23:30:03 +0300 Subject: [PATCH 13/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 0665487705..4fdc065b1f 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -589,7 +589,7 @@ A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipelin ResponsePublisher::ResponsePublisher(RedisPipeline *pipeline, bool buffered = false); ``` -The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when pending tasks. +The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when there are no pending tasks to perform. ### 8. SAI API @@ -604,7 +604,11 @@ Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keep #### 9.2. Fast Reboot -On fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the BGP makes active the previous bgp routes. From now SONiC devices restores its work. BGP sessions set up and bgp graceful restart mode ends +During fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the peer's BGP re-learns advertised routes. + +Considering this, the introduction of a slight delay in advertisement could lead to some increase in fast reboot downtime. + +TBD ### 10. Restrictions/Limitations From eb83fdfda3242eaf201c33788c9027a9a802dc58 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 11 Oct 2022 23:09:08 +0300 Subject: [PATCH 14/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 4fdc065b1f..375380feb5 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -101,15 +101,12 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a - A configuration knob ```bgp-suppress-fib-pending``` in ```DEVICE_METADATA``` table in CONFIG DB to control the enablement of the feature is required. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload```. This knob can only be enabled if the corresponding response channel is enabled in ```APP_STATE_LOGGING``` - ```fpmsyncd``` must consume the responses from ```RouteOrch``` when the feature is enabled and communicate the status of a route back to ```zebra``` using ```FPM``` channel - ```FRR``` must support ```bgp suppress-fib-pending``` as well as response channel via ```FPM```. Available as part of ```FRR``` 8.4 or requires a patched 8.2 release -- Link-local2ME, IP2Me routes responses aren't published - MPLS, VNET routes are out of scope of this document ### 5. Architecture Design -TBD - -##### Figure 2. Use case scenario +##### Figure 2. Architecture

Figure 2. Architecture diagram @@ -354,12 +351,21 @@ Example usage in ```RouteOrch```: m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); ``` +Example data: +``` +127.0.0.1:6379[14]> hgetall ROUTE_TABLE:10.0.0.206/32 +1) "nexthop" +2) "192.168.1.158,192.168.1.159" +3) "ifname" +4) "Ethernet0,Ethernet0" +``` + #### RouteOrch Route Set Flow ```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. -*A note on a temporary route*: +*Temporary route*: A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. @@ -545,6 +551,9 @@ sequenceDiagram participant zebra APPL_STATE_DB ->> fpmsyncd: ROUTE_TABLE: fvs + alt Operation is SET + fpmsyncd ->> zebra: FPM message with RTM_NEWLINK + end ``` #### 7.4. Response Channel Performance considerations From e667886235033e517761c83aea4ba961fe8b1132 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 13 Oct 2022 15:39:55 +0300 Subject: [PATCH 15/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 194 ++++++++++++++++++++++++++--- 1 file changed, 180 insertions(+), 14 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 375380feb5..bb4f4ad72c 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -270,7 +270,38 @@ No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be imp #### 7.1. BGP Docker container startup -BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. The startup flow of BGP container is described below: +BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. + +This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. Configuration template snippet *bgpd.main.conf.j2*: + +```jinja2 +router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} +{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} + bgp suppress-fib-pending +{% endif %} +``` + +FPM response channel is only supported with new ```dplane_fpm_nl``` zebra plugin and SONiC needs start using the new and supported plugin. Additional configuration is required in *zebra.conf.j2*: + +``` +! Fallback to nexthops infromation as part of RTM_NEWROUTE +no fpm use-next-hop-groups + +! Configure FPM server address +fpm address 127.0.0.1 +``` + +Zebra shall be started with an option ```--asic-offload notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: + +```jinja2 +[program:zebra] +{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} +{% set zebra_extra_arguments = "--asic-offload notify_on_offload" } +{% endif %} +command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl {{ zebra_extra_arguments }} +``` + +The startup flow of BGP container is shown below: ##### Figure 3. BGP Configuration Flow Diagram @@ -290,10 +321,26 @@ sequenceDiagram participant A as BGP /usr/bin/docker_init.sh participant S as sonic-cfggen participant T as bgpd.main.j2 + participant TS as supervisor.conf.j2 + participant TZ as zebra.conf.j2 participant E as /etc/frr/bgpd.conf + participant ES as supervisord.conf + participant EZ as /etc/frr/zebra.conf activate A + A -->> TS: /usr/share/sonic/templates/supervisor/supervisor.conf.j2 + activate TS + + CONFIG_DB -->> TS: DEVICE_METADATA|localhost + + alt "bgp-suppress-fib-pending" == "enabled" + TS --> ES: configure zebra startup parameters + end + + + deactivate TS + A -->> S: /usr/share/sonic/templates/bgpd/gen_bgpd.conf.j2 activate S @@ -311,6 +358,12 @@ sequenceDiagram deactivate T + A -->> TZ: /usr/share/sonic/templates/supervisor/supervisor.conf.j2 + + activate TZ + + deactivate TZ + deactivate S A -->> A: start supervisord @@ -318,15 +371,61 @@ sequenceDiagram deactivate A ``` -This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. Configuration template snippet *bgpd.main.conf.j2*: - -```jinja2 -router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} -{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} - bgp suppress-fib-pending -{% endif %} +``` +admin@sonic:~$ show ip route +Codes: K - kernel route, C - connected, S - static, R - RIP, + O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, + T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + +B>q 0.0.0.0/0 [20/0] via 10.0.0.1, PortChannel102, weight 1, 00:04:46 + q via 10.0.0.5, PortChannel105, weight 1, 00:04:46 + q via 10.0.0.9, PortChannel108, weight 1, 00:04:46 + q via 10.0.0.13, PortChannel1011, weight 1, 00:04:46 ``` +``` +admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" +{ + "100.1.0.25/32": [ + { + "prefix": "100.1.0.25/32", + "prefixLen": 32, + "protocol": "bgp", + "vrfId": 0, + "vrfName": "default", + "selected": true, + "destSelected": true, + "distance": 20, + "metric": 0, + "installed": true, + "offloaded": true, + "table": 254, + "internalStatus": 80, + "internalFlags": 264, + "internalNextHopNum": 1, + "internalNextHopActiveNum": 1, + "nexthopGroupId": 4890, + "installedNexthopGroupId": 4890, + "uptime": "00:22:20", + "nexthops": [ + { + "flags": 3, + "fib": true, + "ip": "10.0.0.49", + "afi": "ipv4", + "interfaceIndex": 788, + "interfaceName": "PortChannel1013", + "active": true, + "weight": 1 + } + ] + } + ] +} +``` #### 7.2. RouteOrch @@ -532,6 +631,19 @@ sequenceDiagram #### 7.3. FPMsyncd + +The dplane_fpm_nl has the ability to read route netlink messages +from the underlying fpm implementation that can tell zebra +whether or not the route has been Offloaded/Failed or Trapped. +The end developer must send the data up the same socket that has +been created to listen for FPM messages from Zebra. The data sent +must have a Frame Header with Version set to 1, Message Type set to 1 +and an appropriate message Length. The message data must contain +a RTM_NEWROUTE netlink message that sends the prefix and nexthops +associated with the route. Finally rtm_flags must contain +RTM_F_OFFLOAD, RTM_F_TRAP and or RTM_F_OFFLOAD_FAILED to signify +what has happened to the route in the ASIC. + ##### Figure 6. FPMsyncd response processing @@ -550,9 +662,14 @@ sequenceDiagram participant fpmsyncd participant zebra - APPL_STATE_DB ->> fpmsyncd: ROUTE_TABLE: fvs + APPL_STATE_DB ->> fpmsyncd: ROUTE_TABLE: alt Operation is SET - fpmsyncd ->> zebra: FPM message with RTM_NEWLINK + fpmsyncd ->> fpmsyncd: Find cached route for + fpmsyncd ->> fpmsyncd: Updated nexthop according to + fpmsyncd ->> fpmsyncd: rtnl_route_set_flags(routeObject, RTM_F_OFFLOAD) + fpmsyncd ->> zebra: Send FPM message + zebra ->> fpmsyncd:
+ fpmsyncd ->> fpmsyncd: Remove from the cache end ``` @@ -629,16 +746,65 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati #### 11.1. Unit Test cases -TODO +- FPMSyncd: test case regular route + 1. Mock FPM interface + 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 + 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format + 4. Verify a correct FPM message is being sent to zebra through FPM interface mock + 5. Verify RTM_F_OFFLOAD set in route message sent to zebra +- FPMSyncd: test only one nexthop is attached to the route + 1. Mock FPM interface + 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 + 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB containing only one nexthop 2.2.2.2 format + 4. Verify a correct FPM message is being sent to zebra through FPM interface mock with one nexthop + 5. Verify RTM_F_OFFLOAD set in route message sent to zebra +- FPMSyncd: test case VRF route + 1. Mock FPM interface + 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 + 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format + 4. Verify a correct FPM message is being sent to zebra through FPM interface mock + 5. Verify RTM_F_OFFLOAD set in route message sent to zebra +- FPMSyncd: send RTM_DELROUTE + 1. Mock FPM interface + 2. Call RouteSync::onRouteMsg() with RTM_DELROUTE message containing route 1.1.1.0/24 + 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format with delete operation response + 4. Verify no message is sent to zebra + + +- RouteOrch: program ASIC route + 1. Create FVS and call RouteOrch::doTask() + 2. Verify route is created in SAI + 3. Verify the content of APPL_STATE_DB + + +- RouteOrch: program ASIC route with temporary nexthop + 1. Mock SAI_SWITCH_ATTR_NUMBER_OF_ECMP_GROUPS to 0 + 2. Create FVS and call RouteOrch::doTask() + 3. Verify route is created in SAI with 1 nexthop + 4. Verify the content of APPL_STATE_DB contains only 1 nexthop #### 11.2. VS Test cases -TODO +Code coverage is satisfied by UT and the E2E flow is not possible to test with current VS infrastructure, so no VS tests are planned. #### 11.3. System Test cases -TODO +In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test is based on T0 topology: + +1. Enable ```bgp-suppress-fib-pending``` +2. Stop syncd process to simulate a delay: ```kill --SIGSTOP $(pidof syncd)``` +3. Setup BGP speaker +4. Announces routes to DUT through exabgp +5. Verify BGP session is established +6. Make sure announced BGP route is queued in ```show ip route``` output +7. Verify the route is not announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer +8. Restore syncd process: ```kill --SIGCONT $(pidof syncd)``` +8. Make sure route is programmed in ASIC_DB +9. Make sure announced BGP route is FIB synced in ```show ip route``` output +7. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer + ### 12. Open/Action items - if any -TODO +- Default state of the feature based off minigraph device type +- Non dataplane BGP sessions and whether we have to enable feature per neighbor From 88315cafb6d45f9e9fe419cd21388c279b477176 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 13 Oct 2022 16:41:32 +0300 Subject: [PATCH 16/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index bb4f4ad72c..d1802abf74 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -96,18 +96,22 @@ To avoid that, the route programming has to be synchronous down to the ASIC to a ### 4. Requirements -- ```RouteOrch``` must use the ```ResponsePublisher``` API to create a feedback channel and write each route entry programming status for the ```fpmsyncd``` to consume -- The response channel can be turned off based on user configuration in a new ```APP_STATE_LOGGING``` table in CONFIG DB. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload``` -- A configuration knob ```bgp-suppress-fib-pending``` in ```DEVICE_METADATA``` table in CONFIG DB to control the enablement of the feature is required. This configuration is applied at startup and can't be changed while the system is running, requiring a ```config reload```. This knob can only be enabled if the corresponding response channel is enabled in ```APP_STATE_LOGGING``` -- ```fpmsyncd``` must consume the responses from ```RouteOrch``` when the feature is enabled and communicate the status of a route back to ```zebra``` using ```FPM``` channel -- ```FRR``` must support ```bgp suppress-fib-pending``` as well as response channel via ```FPM```. Available as part of ```FRR``` 8.4 or requires a patched 8.2 release +High level requirements: + +- Provide an end to end ASIC hardware to BGP route programming feedback mechanism +- BGP must not advertise routes which aren't yet offloaded +- Connected and statically configured routes do not wait route offload status - MPLS, VNET routes are out of scope of this document +- Both BGP suppression as well as feedback channel are optional and by disabled default +- The above configuration can be applied only after reload ### 5. Architecture Design ##### Figure 2. Architecture +The below diagram shows the high level architecture including sending an RTM_NEWROUTE response message with RTM_F_OFFLOAD flag set to zebra through FPM: +

Figure 2. Architecture diagram

@@ -371,6 +375,8 @@ sequenceDiagram deactivate A ``` +Before the route is offloaded ```show ip route``` shows routes as queued: + ``` admin@sonic:~$ show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, @@ -386,6 +392,8 @@ B>q 0.0.0.0/0 [20/0] via 10.0.0.1, PortChannel102, weight 1, 00:04:46 q via 10.0.0.13, PortChannel1011, weight 1, 00:04:46 ``` +Once zebra is notified about successfull route offload status the *offload* flag is set: + ``` admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" { From 708ea6812d0221eb6d54cf4ec119910c09c7b135 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 17 Oct 2022 18:46:11 +0300 Subject: [PATCH 17/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 318 ++++++++++++++++------------- 1 file changed, 171 insertions(+), 147 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index d1802abf74..2b1e9ef2e2 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -18,11 +18,12 @@ - [6.3.1. APP_STATE_LOGGING](#631-app_state_logging) - [6.3.2. DEVICE_METADATA](#632--device_metadata) - [7. High-Level Design](#7-high-level-design) - - [7.1. BGP Docker container startup](#71-bgp-docker-container-startup) - - [7.2. RouteOrch](#72-routeorch) - - [7.3. FPMsyncd](#73-fpmsyncd) - - [7.4. Response Channel Performance considerations](#74-response-channel-performance-considerations) - - [7.4.1. Table 1. Publishing 1k ROUTE_TABLE responses](#741-table-1-publishing-1k-route_table-responses) + - [7.1. FPM plugin migration](#71-fpm-plugin-migration) + - [7.2. BGP Docker container startup](#72-bgp-docker-container-startup) + - [7.3. RouteOrch](#73-routeorch) + - [7.4. FPMsyncd](#74-fpmsyncd) + - [7.5. Response Channel Performance considerations](#75-response-channel-performance-considerations) + - [7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses](#751-table-1-publishing-1k-route_table-responses) - [8. SAI API](#8-sai-api) - [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - [9.1. Warm Reboot](#91-warm-reboot) @@ -43,39 +44,26 @@ ### 1. Scope -This document describes a feedback mechanism that allows BGP not to adveritise routes that haven't been programmed yet or failed to be programmed to ASIC. +This document describes a feedback mechanism that allows BGP not to advertise routes that haven't been programmed yet. ### 2. Definitions/Abbreviations | Definitions/Abbreviation | Description | | ------------------------ | ---------------------------- | +| ASIC | Application specific integrated circuit | +| HW | Hardware | +| SW | Software | | BGP | Border Gateway Protocol | | FRR | Free Range Routing | | SWSS | Switch state service | -| SYNCD | ASIC syncrhonization service | +| SYNCD | ASIC synchronization service | | FPM | Forwarding Plane Manager | | SAI | Switch Abstraction Interface | | OID | Object Identifier | ### 3. Overview -The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. - -The solution is to provide a configurable option to check for the FIB install status of the prefixes and advertise to peers if the prefixes are successfully installed in the FIB. The advertisement of the prefixes are suppressed if it is not installed in FIB. - -The following conditions apply will apply when checking for route installation status in FIB: - -- The advertisement or suppression of routes based on FIB install status applies only for newly learnt routes from peer (routes which are not in BGP local RIB). -- If the route received from peer already exists in BGP local RIB and route attributes have changed (best path changed), the old path is deleted and new path is installed in FIB. The FIB install status will not have any effect. Therefore only when the route is received first time the checks apply. -- The feature will not apply for routes learnt through other means like redistribution to bgp from other protocols. This is applicable only to peer learnt routes. -- If a route is installed in FIB and then gets deleted from the dataplane, then routes will not be withdrawn from peers. This will be considered as dataplane issue. -- The feature will slightly increase the time required to advertise the routes to peers since the route install status needs to be received from the FIB -- If routes are received by the peer before the configuration is applied, then the bgp sessions need to be reset for the configuration to take effect. -- If the route which is already installed in dataplane is removed for some reason, sending withdraw message to peers is not currently supported. - -[FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst) - -Considering the following scenario: +As of today, SONiC BGP advertises learnt prefixes regardless of whether these prefixes were successfully programmed into ASIC. While route programming failure is followed by orchagent crash and all services restart, even for successfully created routes there is a short period of time when the peer will be black holing traffic. Also, in the following scenario, a credit loop occurs: ##### Figure 1. Use case scenario @@ -90,20 +78,39 @@ The problem with BGP programming occurs after the T1 switch is rebooted: 3. FRR advertises the prefixes to T2 without waiting for them to be programmed in the ASIC 4. T2 starts forwarding traffic for prefixes not yet programmed, according to T1’s routing table, T1 sends it back to a default route – same T2 - When the traffic is bounced back on lossless queue, buffers on both sides are overflown, credit loop happens, with PFC storm and watchdog triggered shutting down the port. To avoid that, the route programming has to be synchronous down to the ASIC to avoid credit loops. +FRR supports this through a feature called *BGP suppress FIB pending*. [FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst): + +> The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. +> +> The solution is to provide a configurable option to check for the FIB install status of the prefixes and advertise to peers if the prefixes are successfully installed in the FIB. The advertisement of the prefixes are suppressed if it is not installed in FIB. +> +> The following conditions apply will apply when checking for route installation status in FIB: +> +> - The advertisement or suppression of routes based on FIB install status applies only for newly learnt routes from peer (routes which are not in BGP local RIB). +> - If the route received from peer already exists in BGP local RIB and route attributes have changed (best path changed), the old path is deleted and new path is installed in FIB. The FIB install status will not have any effect. Therefore only when the route is received first time the checks apply. +> - The feature will not apply for routes learnt through other means like redistribution to bgp from other protocols. This is applicable only to peer learnt routes. +> - If a route is installed in FIB and then gets deleted from the dataplane, then routes will not be withdrawn from peers. This will be considered as dataplane issue. +> - The feature will slightly increase the time required to advertise the routes to peers since the route install status needs to be received from the FIB +> - If routes are received by the peer before the configuration is applied, then the bgp sessions need to be reset for the configuration to take effect. +> - If the route which is already installed in dataplane is removed for some reason, sending withdraw message to peers is not currently supported. + +BGP will queue routes which aren't installed in the FIB. To make zebra install routes in FIB an RTM_NEWROUTE message with *RTM_F_OFFLOAD* flag set must be sent by SONiC to zebra through FPM socket. + +**NOTE**: This feature is implemented as part of new *dplane_fpm_nl* zebra plugin. SONiC is still using old *fpm* plugin which isn't developed anymore and thus SONiC must migrate to the new implementation as part of this change. + ### 4. Requirements High level requirements: - Provide an end to end ASIC hardware to BGP route programming feedback mechanism -- BGP must not advertise routes which aren't yet offloaded -- Connected and statically configured routes do not wait route offload status +- BGP must not advertise routes which aren't yet offloaded if this feature is enabled +- Connected and statically configured routes do not respect offload status - MPLS, VNET routes are out of scope of this document - Both BGP suppression as well as feedback channel are optional and by disabled default -- The above configuration can be applied only after reload +- The above configuration can be applied only at startup and cannot be changed while system is running ### 5. Architecture Design @@ -116,6 +123,9 @@ The below diagram shows the high level architecture including sending an RTM_NEW Figure 2. Architecture diagram

+- *2*-*8* - existing flows in SONiC +- *1*, *9*, *10* - new flows to be added + ### 6. Configuration and management #### 6.1. Config DB Enhancements @@ -272,7 +282,25 @@ No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be imp ### 7. High-Level Design -#### 7.1. BGP Docker container startup +#### 7.1. FPM plugin migration + +FPM response channel is only supported with new ```dplane_fpm_nl``` zebra plugin and SONiC needs start using the new and supported plugin. Additional configuration is required in *zebra.conf.j2*: + +``` +! Fallback to nexthops information as part of RTM_NEWROUTE message +no fpm use-next-hop-groups + +! Configure FPM server address +fpm address 127.0.0.1 +``` + +Zebra shall be started with an ```-M dplane_fpm_nl``` to use the new plugin. + +New FPM plugin shall be patched the same way as old one to pass VRF if_index in ```rtmsg->rtm_table``` instead of table ID in FPM message. [SONiC FRR patch](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-frr/patch/0003-Use-vrf_id-for-vrf-not-tabled_id.patch). + +Since now SONiC will start communicating RTM_NEWROUTE messages back to zebra, similar change has to be made to accept VRF if_index instead of table ID. + +#### 7.2. BGP Docker container startup BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. @@ -285,16 +313,6 @@ router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} {% endif %} ``` -FPM response channel is only supported with new ```dplane_fpm_nl``` zebra plugin and SONiC needs start using the new and supported plugin. Additional configuration is required in *zebra.conf.j2*: - -``` -! Fallback to nexthops infromation as part of RTM_NEWROUTE -no fpm use-next-hop-groups - -! Configure FPM server address -fpm address 127.0.0.1 -``` - Zebra shall be started with an option ```--asic-offload notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: ```jinja2 @@ -435,7 +453,7 @@ admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" } ``` -#### 7.2. RouteOrch +#### 7.3. RouteOrch #### ResponsePublisher in RouteOrch @@ -448,23 +466,22 @@ A snippet of ```ResponsePublisher```'s API that is going to be used is given bel void ResponsePublisher::publish(const std::string &table, const std::string &key, const std::vector &intent_attrs, const ReturnCode &status, - const std::vector &state_attrs, bool replace = false) override; ``` Example usage in ```RouteOrch```: ```c++ -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), true); ``` Example data: ``` -127.0.0.1:6379[14]> hgetall ROUTE_TABLE:10.0.0.206/32 +127.0.0.1:6379[14]> hgetall "ROUTE_TABLE:193.5.96.0/25" 1) "nexthop" -2) "192.168.1.158,192.168.1.159" +2) "10.0.0.1,10.0.0.5,10.0.0.9,10.0.0.13" 3) "ifname" -4) "Ethernet0,Ethernet0" +4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" ``` @@ -474,8 +491,7 @@ Example data: *Temporary route*: -A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. - +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well because the switch is till able to forward traffic matching that prefix. ##### Figure 4. RouteOrch Route Set Flow @@ -505,64 +521,58 @@ sequenceDiagram RouteOrch ->> RouteOrch: doTask() loop For each entry in m_toSync - alt NHG exist or can be added - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - else NH can't be added - Note right of RouteOrch: 1 NH is selected and addRoute()
is called with temporary NH - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - end + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker end - Note left of RouteOrch: On any error/pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
+ Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
- RouteOrch ->> RouteBulker: RouteBulker.flush() - activate RouteBulker - RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - - RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB + alt New route + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + else Existing route + RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + end RouteBulker -->> RouteOrch:
deactivate RouteBulker - loop For each status in RouteBulker - alt SAI_STATUS_SUCCESS - RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() - activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB:
- activate APPL_STATE_DB - APPL_STATE_DB -->> ResponsePublisher:
- deactivate APPL_STATE_DB - Note right of RouteOrch: Publish actual FVs
(in case of temporary route only 1 NH is added) - ResponsePublisher -->> RouteOrch:
- deactivate ResponsePublisher - else - RouteOrch ->> RouteOrch: abort - end + Note left of RouteOrch: On any route programming error
RouteOrch crashes orchagent.
No response is written
+ + loop For each entry status in RouteBulker + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB:
+ activate APPL_STATE_DB + APPL_STATE_DB -->> ResponsePublisher:
+ deactivate APPL_STATE_DB + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher end deactivate RouteOrch ``` +```RouteOrch``` reads *APP_STATE_LOGGING* table flag from CONFIG_DB and completely disables response publishing if the feature is disabled. *15*-*18* are not invoked and the flow remains the same as today. + + #### RouteOrch Route delete Flow @@ -602,7 +612,7 @@ sequenceDiagram deactivate RouteBulker end - Note left of RouteOrch: On any error/pre-condition failed
RouteOrch is retrying to remove
the route on next iteration
In this case no response is written
+ Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to remove
the route on next iteration
In this case no response is written
RouteOrch ->> RouteBulker: RouteBulker.flush() activate RouteBulker @@ -618,42 +628,42 @@ sequenceDiagram RouteBulker -->> RouteOrch:
deactivate RouteBulker + Note left of RouteOrch: On any route deletion error
RouteOrch crashes orchagent.
No response is written
+ loop For each status in RouteBulker - alt SAI_STATUS_SUCCESS - RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() - activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB:
- activate APPL_STATE_DB - APPL_STATE_DB -->> ResponsePublisher:
- deactivate APPL_STATE_DB - Note right of RouteOrch: Publish the status and remove entry from APPL_STATE_DB - ResponsePublisher -->> RouteOrch:
- deactivate ResponsePublisher - else - RouteOrch ->> RouteOrch: abort - end + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB:
+ activate APPL_STATE_DB + APPL_STATE_DB -->> ResponsePublisher:
+ deactivate APPL_STATE_DB + Note right of RouteOrch: Publish the status and remove entry from APPL_STATE_DB + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher end deactivate RouteOrch ``` -#### 7.3. FPMsyncd +#### 7.4. FPMsyncd -The dplane_fpm_nl has the ability to read route netlink messages -from the underlying fpm implementation that can tell zebra -whether or not the route has been Offloaded/Failed or Trapped. -The end developer must send the data up the same socket that has -been created to listen for FPM messages from Zebra. The data sent -must have a Frame Header with Version set to 1, Message Type set to 1 -and an appropriate message Length. The message data must contain -a RTM_NEWROUTE netlink message that sends the prefix and nexthops -associated with the route. Finally rtm_flags must contain -RTM_F_OFFLOAD, RTM_F_TRAP and or RTM_F_OFFLOAD_FAILED to signify -what has happened to the route in the ASIC. +> The dplane_fpm_nl has the ability to read route netlink messages +> from the underlying fpm implementation that can tell zebra +> whether or not the route has been Offloaded/Failed or Trapped. +> The end developer must send the data up the same socket that has +> been created to listen for FPM messages from Zebra. The data sent +> must have a Frame Header with Version set to 1, Message Type set to 1 +> and an appropriate message Length. The message data must contain +> a RTM_NEWROUTE netlink message that sends the prefix and nexthops +> associated with the route. Finally rtm_flags must contain +> RTM_F_OFFLOAD, RTM_F_TRAP and or RTM_F_OFFLOAD_FAILED to signify +> what has happened to the route in the ASIC. + +A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -##### Figure 6. FPMsyncd response processing +##### Figure 6. FPMsyncd flow ```mermaid %%{ @@ -666,28 +676,50 @@ what has happened to the route in the ASIC. } }%% sequenceDiagram - participant APPL_STATE_DB - participant fpmsyncd participant zebra + participant RouteSync + participant RouteFeedbackChannel + participant APPL_DB + participant APPL_STATE_DB + - APPL_STATE_DB ->> fpmsyncd: ROUTE_TABLE: - alt Operation is SET - fpmsyncd ->> fpmsyncd: Find cached route for - fpmsyncd ->> fpmsyncd: Updated nexthop according to - fpmsyncd ->> fpmsyncd: rtnl_route_set_flags(routeObject, RTM_F_OFFLOAD) - fpmsyncd ->> zebra: Send FPM message - zebra ->> fpmsyncd:
- fpmsyncd ->> fpmsyncd: Remove from the cache + zebra ->> RouteSync: RTM_NEWROUTE FPM message + activate RouteSync + RouteSync ->> APPL_DB: Write entry to ROUTE_TABLE + activate APPL_DB + APPL_DB ->> RouteSync:
+ deactivate APPL_DB + + RouteSync ->> RouteFeedbackChannel: onRouteMsg() + activate RouteFeedbackChannel + Note right of RouteFeedbackChannel: rtnl_route object is saved
into pending routes container
of RouteFeedbackChannel + RouteFeedbackChannel ->> RouteSync:
+ deactivate RouteFeedbackChannel + + deactivate RouteSync + + APPL_STATE_DB ->> RouteFeedbackChannel: ROUTE_TABLE publish event/onRouteResponse() + activate RouteFeedbackChannel + alt Successful SET operation + Note right of RouteFeedbackChannel: Pop rtnl_route object
and set RTM_F_OFFLOAD flag
Send updated rtnl_route object
through FPM + RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE + activate zebra + zebra ->> RouteFeedbackChannel:
+ deactivate zebra end + deactivate RouteFeedbackChannel + ``` -#### 7.4. Response Channel Performance considerations +fpmsyncd reads *bgp-suppress-fib-pending* flag from CONFIG_DB and completely disables *RouteFeedbackChannel* if the feature is disabled. *4*-*7* are not invoked and the flow remains the same as today. + +#### 7.5. Response Channel Performance considerations Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. The following table shows the results for publishing 10k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: -##### 7.4.1. Table 1. Publishing 1k ROUTE_TABLE responses +##### 7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses | Scenario | Time (sec) | Ratio | | ---------------------- | ---------- | ----- | @@ -742,10 +774,11 @@ During fast reboot BGP session is closed by SONiC device without the notificatio Considering this, the introduction of a slight delay in advertisement could lead to some increase in fast reboot downtime. -TBD - ### 10. Restrictions/Limitations +- Connected and statically configured routes do not respect offload status +- MPLS, VNET routes are out of scope of this document + ### 11. Testing Requirements/Design Regression test, VS test and unit testing are required for this functionality. @@ -760,12 +793,6 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format 4. Verify a correct FPM message is being sent to zebra through FPM interface mock 5. Verify RTM_F_OFFLOAD set in route message sent to zebra -- FPMSyncd: test only one nexthop is attached to the route - 1. Mock FPM interface - 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 - 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB containing only one nexthop 2.2.2.2 format - 4. Verify a correct FPM message is being sent to zebra through FPM interface mock with one nexthop - 5. Verify RTM_F_OFFLOAD set in route message sent to zebra - FPMSyncd: test case VRF route 1. Mock FPM interface 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 @@ -778,18 +805,15 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format with delete operation response 4. Verify no message is sent to zebra - - RouteOrch: program ASIC route 1. Create FVS and call RouteOrch::doTask() 2. Verify route is created in SAI 3. Verify the content of APPL_STATE_DB - -- RouteOrch: program ASIC route with temporary nexthop - 1. Mock SAI_SWITCH_ATTR_NUMBER_OF_ECMP_GROUPS to 0 - 2. Create FVS and call RouteOrch::doTask() - 3. Verify route is created in SAI with 1 nexthop - 4. Verify the content of APPL_STATE_DB contains only 1 nexthop +- RouteOrch: delete ASIC route + 1. Create delete entry and call RouteOrch::doTask() + 2. Verify route is delete in SAI + 3. Verify the content of APPL_STATE_DB #### 11.2. VS Test cases From e43bf41f435f19956a8eeb3b4341a12484f753bf Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 17 Oct 2022 18:47:32 +0300 Subject: [PATCH 18/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 2b1e9ef2e2..3b0901df1e 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -11,19 +11,14 @@ - [5. Architecture Design](#5-architecture-design) - [6. Configuration and management](#6-configuration-and-management) - [6.1. Config DB Enhancements](#61-config-db-enhancements) - - [6.1.1. APP_STATE_LOGGING](#611-app_state_logging) - - [6.1.2. DEVICE_METADATA](#612-device_metadata) - [6.2. Manifest (if the feature is an Application Extension)](#62-manifest-if-the-feature-is-an-application-extension) - [6.3. CLI/YANG model Enhancements](#63-cliyang-model-enhancements) - - [6.3.1. APP_STATE_LOGGING](#631-app_state_logging) - - [6.3.2. DEVICE_METADATA](#632--device_metadata) - [7. High-Level Design](#7-high-level-design) - [7.1. FPM plugin migration](#71-fpm-plugin-migration) - [7.2. BGP Docker container startup](#72-bgp-docker-container-startup) - [7.3. RouteOrch](#73-routeorch) - [7.4. FPMsyncd](#74-fpmsyncd) - [7.5. Response Channel Performance considerations](#75-response-channel-performance-considerations) - - [7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses](#751-table-1-publishing-1k-route_table-responses) - [8. SAI API](#8-sai-api) - [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - [9.1. Warm Reboot](#91-warm-reboot) @@ -130,6 +125,7 @@ The below diagram shows the high level architecture including sending an RTM_NEW #### 6.1. Config DB Enhancements + ##### 6.1.1. APP_STATE_LOGGING Configuration schema in ABNF format: @@ -152,6 +148,7 @@ Sample of CONFIG DB snippet given below: } ``` + ##### 6.1.2. DEVICE_METADATA Configuration schema in ABNF format: @@ -183,6 +180,7 @@ This feature is implemented as part of existing BGP and SWSS containers, no mani #### 6.3. CLI/YANG model Enhancements + ##### 6.3.1. APP_STATE_LOGGING A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: @@ -227,6 +225,7 @@ module sonic-app-state-logging { Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. + ##### 6.3.2. DEVICE_METADATA A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. @@ -719,6 +718,7 @@ Route programming performance is one of crucial characteristics of a network swi The following table shows the results for publishing 10k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: + ##### 7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses | Scenario | Time (sec) | Ratio | From a529a554a47a944de3fe6bb175ce471fb2767539 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 14:54:13 +0300 Subject: [PATCH 19/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 84 +++--------------------------- 1 file changed, 8 insertions(+), 76 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 3b0901df1e..6332e7db98 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -312,86 +312,16 @@ router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} {% endif %} ``` -Zebra shall be started with an option ```--asic-offload notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: +Zebra shall be started with an option ```--asic-offload=notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: ```jinja2 [program:zebra] {% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} -{% set zebra_extra_arguments = "--asic-offload notify_on_offload" } +{% set zebra_extra_arguments = "--asic-offload=notify_on_offload" } {% endif %} command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl {{ zebra_extra_arguments }} ``` -The startup flow of BGP container is shown below: - - -##### Figure 3. BGP Configuration Flow Diagram - -```mermaid -%%{ - init: { - "theme": "forest", - "sequence": { - "rightAngles": true, - "showSequenceNumbers": true - } - } -}%% -sequenceDiagram - participant CONFIG_DB - participant A as BGP /usr/bin/docker_init.sh - participant S as sonic-cfggen - participant T as bgpd.main.j2 - participant TS as supervisor.conf.j2 - participant TZ as zebra.conf.j2 - participant E as /etc/frr/bgpd.conf - participant ES as supervisord.conf - participant EZ as /etc/frr/zebra.conf - - activate A - - A -->> TS: /usr/share/sonic/templates/supervisor/supervisor.conf.j2 - activate TS - - CONFIG_DB -->> TS: DEVICE_METADATA|localhost - - alt "bgp-suppress-fib-pending" == "enabled" - TS --> ES: configure zebra startup parameters - end - - - deactivate TS - - A -->> S: /usr/share/sonic/templates/bgpd/gen_bgpd.conf.j2 - - activate S - - S -->> T:
- - activate T - - CONFIG_DB -->> T: DEVICE_METADATA|localhost - - alt "bgp-suppress-fib-pending" == "enabled" - T --> E: configure "bgp suppress-fib-pending" for router - end - - - deactivate T - - A -->> TZ: /usr/share/sonic/templates/supervisor/supervisor.conf.j2 - - activate TZ - - deactivate TZ - - deactivate S - - A -->> A: start supervisord - - deactivate A -``` - Before the route is offloaded ```show ip route``` shows routes as queued: ``` @@ -493,7 +423,7 @@ Example data: A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well because the switch is till able to forward traffic matching that prefix. -##### Figure 4. RouteOrch Route Set Flow +##### Figure 3. RouteOrch Route Set Flow ```mermaid %%{ @@ -576,7 +506,7 @@ sequenceDiagram #### RouteOrch Route delete Flow -##### Figure 5. RouteOrch Route Delete Flow +##### Figure 4. RouteOrch Route Delete Flow Similar processing happens on ```DEL``` operation for a prefix from ```ROUTE_TABLE```. The removed prefixes are filled in ```RouteBulker``` which are then flushed forming a bulk remove API call to SAIRedis. ```RouteOrch::removeRoutePost()``` then collects the object statuses and if the status is not SAI_STATUS_SUCCESS orchagent ```abort()```s, otherwise it should publish the result via ```ResponsePublisher::publish()``` leaving ```intent_attrs``` vector empty which will remove the corresponding state key from ```APPL_STATE_DB```. @@ -662,7 +592,7 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -##### Figure 6. FPMsyncd flow +##### Figure 5. FPMsyncd flow ```mermaid %%{ @@ -734,6 +664,8 @@ On the other side, ```fpmsyncd``` does not wait for each individual route status The following schema represents communication methods between components: + +##### Figure 6. zebra/orchagent feedback channel ```mermaid %%{ init: { @@ -821,7 +753,7 @@ Code coverage is satisfied by UT and the E2E flow is not possible to test with c #### 11.3. System Test cases -In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test is based on T0 topology: +In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: 1. Enable ```bgp-suppress-fib-pending``` 2. Stop syncd process to simulate a delay: ```kill --SIGSTOP $(pidof syncd)``` From b97ba7a8131dad9f0f9596f3843e0b1a380e4537 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 16:34:25 +0300 Subject: [PATCH 20/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 6332e7db98..04d142897e 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -591,6 +591,8 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. RTM_NEWROUTE message must contain same route attributes as the original message. Since we do not store anywhere protocol, scope, metric and other route kernel attributes, *RouteFeedbackChannel* will cache the original message in a *multimap* where they are associated with APPL_DB key. + ##### Figure 5. FPMsyncd flow From ab3d6feb818796b6177a3838613e2bdf35c5471f Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 16:37:33 +0300 Subject: [PATCH 21/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 04d142897e..ab5f8a980b 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -102,7 +102,6 @@ High level requirements: - Provide an end to end ASIC hardware to BGP route programming feedback mechanism - BGP must not advertise routes which aren't yet offloaded if this feature is enabled -- Connected and statically configured routes do not respect offload status - MPLS, VNET routes are out of scope of this document - Both BGP suppression as well as feedback channel are optional and by disabled default - The above configuration can be applied only at startup and cannot be changed while system is running @@ -710,7 +709,6 @@ Considering this, the introduction of a slight delay in advertisement could lead ### 10. Restrictions/Limitations -- Connected and statically configured routes do not respect offload status - MPLS, VNET routes are out of scope of this document ### 11. Testing Requirements/Design From 6f38f572d63d2ff6ceacbca288f8fe6fdaed0391 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 17:35:32 +0300 Subject: [PATCH 22/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index ab5f8a980b..70b0d9ae1b 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -590,7 +590,7 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. RTM_NEWROUTE message must contain same route attributes as the original message. Since we do not store anywhere protocol, scope, metric and other route kernel attributes, *RouteFeedbackChannel* will cache the original message in a *multimap* where they are associated with APPL_DB key. +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. In order to reconstruct the same message additional data from ```rtmsg``` struct has to be passed to APPL_DB (tos, scope, etc.). ##### Figure 5. FPMsyncd flow @@ -620,24 +620,21 @@ sequenceDiagram APPL_DB ->> RouteSync:
deactivate APPL_DB - RouteSync ->> RouteFeedbackChannel: onRouteMsg() - activate RouteFeedbackChannel - Note right of RouteFeedbackChannel: rtnl_route object is saved
into pending routes container
of RouteFeedbackChannel - RouteFeedbackChannel ->> RouteSync:
- deactivate RouteFeedbackChannel - deactivate RouteSync - APPL_STATE_DB ->> RouteFeedbackChannel: ROUTE_TABLE publish event/onRouteResponse() + APPL_STATE_DB ->> RouteFeedbackChannel: ROUTE_TABLE publish event + activate RouteSync + RouteSync ->> RouteFeedbackChannel: onRouteResponse() activate RouteFeedbackChannel alt Successful SET operation - Note right of RouteFeedbackChannel: Pop rtnl_route object
and set RTM_F_OFFLOAD flag
Send updated rtnl_route object
through FPM + Note right of RouteFeedbackChannel: Construct RTM_NEWROUTE based of
response field value tuples
and set RTM_F_OFFLOAD flag
RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE activate zebra zebra ->> RouteFeedbackChannel:
deactivate zebra end deactivate RouteFeedbackChannel + deactivate RouteSync ``` From 1d07b68d65a697d4e94ac7f29d3ffd112cf9fa1e Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 18:13:33 +0300 Subject: [PATCH 23/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 70b0d9ae1b..11211bfb43 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -590,7 +590,7 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. In order to reconstruct the same message additional data from ```rtmsg``` struct has to be passed to APPL_DB (tos, scope, etc.). +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. ##### Figure 5. FPMsyncd flow From a52c4093bf6d6d55d6aa20450c83bba509839e96 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 18 Oct 2022 19:14:49 +0300 Subject: [PATCH 24/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 64 +++++++++++++++++------------- 1 file changed, 37 insertions(+), 27 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 11211bfb43..4d299c62ae 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -419,7 +419,7 @@ Example data: *Temporary route*: -A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well because the switch is till able to forward traffic matching that prefix. +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. ##### Figure 3. RouteOrch Route Set Flow @@ -449,34 +449,44 @@ sequenceDiagram RouteOrch ->> RouteOrch: doTask() loop For each entry in m_toSync - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker + alt NHG exist or can be added + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + else NH can't be added + Note right of RouteOrch: 1 NH is selected and addRoute()
is called with temporary NH + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + end end Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
- alt New route - RouteOrch ->> RouteBulker: RouteBulker.flush() - activate RouteBulker - RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - else Existing route - RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB + loop For each route + alt New route + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + else Existing route + RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + end end RouteBulker -->> RouteOrch:
@@ -491,6 +501,7 @@ sequenceDiagram activate APPL_STATE_DB APPL_STATE_DB -->> ResponsePublisher:
deactivate APPL_STATE_DB + Note right of RouteOrch: Publish actual FVs
(in case of temporary route only 1 NH is added) ResponsePublisher -->> RouteOrch:
deactivate ResponsePublisher end @@ -590,7 +601,7 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. ##### Figure 5. FPMsyncd flow @@ -697,7 +708,6 @@ No new SAI API or changes to SAI design and behaviour needed for this functional Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. - #### 9.2. Fast Reboot During fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the peer's BGP re-learns advertised routes. From 5f33539dffce23cfd49b1b392d01fd5cf72c729d Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Wed, 19 Oct 2022 12:55:32 +0300 Subject: [PATCH 25/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 4d299c62ae..fe396a5441 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -743,16 +743,26 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati 2. Call RouteSync::onRouteMsg() with RTM_DELROUTE message containing route 1.1.1.0/24 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format with delete operation response 4. Verify no message is sent to zebra +- FPMSyncd: test only one nexthop is attached to the route + 1. Mock FPM interface + 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 + 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB containing only one nexthop 2.2.2.2 format + 4. Verify a correct FPM message is being sent to zebra through FPM interface mock with one nexthop + 5. Verify RTM_F_OFFLOAD set in route message sent to zebra - RouteOrch: program ASIC route 1. Create FVS and call RouteOrch::doTask() 2. Verify route is created in SAI 3. Verify the content of APPL_STATE_DB - - RouteOrch: delete ASIC route 1. Create delete entry and call RouteOrch::doTask() 2. Verify route is delete in SAI 3. Verify the content of APPL_STATE_DB +- RouteOrch: program ASIC route with temporary nexthop + 1. Mock SAI_SWITCH_ATTR_NUMBER_OF_ECMP_GROUPS to 0 + 2. Create FVS and call RouteOrch::doTask() + 3. Verify route is created in SAI with 1 nexthop + 4. Verify the content of APPL_STATE_DB contains only 1 nexthop #### 11.2. VS Test cases From 61d159cebc182bb14cf8be13b8f0495d7ced7e36 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Wed, 19 Oct 2022 14:17:34 +0300 Subject: [PATCH 26/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 43 ++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index fe396a5441..dd224b712e 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -394,15 +394,17 @@ A snippet of ```ResponsePublisher```'s API that is going to be used is given bel void ResponsePublisher::publish(const std::string &table, const std::string &key, const std::vector &intent_attrs, const ReturnCode &status, + const std::vector &state_attrs, bool replace = false) override; ``` Example usage in ```RouteOrch```: ```c++ -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), true); +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); ``` + Example data: ``` 127.0.0.1:6379[14]> hgetall "ROUTE_TABLE:193.5.96.0/25" @@ -601,7 +603,7 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. The offload route status is relevant only for BGP received routes. In order to send back RTM_NEWROUTE only for BGP originated routes it is required to encode "proto" (rtmsg->rtm_protocol field) in APPL_DB which will be sent back as part of response, so that if the "proto" is received for non-BGP route the response to zebra isn't sent. ##### Figure 5. FPMsyncd flow @@ -649,6 +651,43 @@ sequenceDiagram ``` +*FpmLink* class has to be extended with new API to send netlink message through FPM socket: + +```c++ +ssize_t FpmLink::send(nl_msg* msg) +``` + +Example pseudo-code snippet of RTM_NEWROUTE construction: + +```c++ +auto route = rtnl_route_alloc(); + +rtnl_route_set_dst(routeObject, dstAddr); +rtnl_route_set_protocol(routeObject, proto); +rtnl_route_set_family(routeObject.get(), ipFamily); + +// in SONiC FPM uses vrf if_index instead of table +rtnl_route_set_table(routeObject.get(), vrfIfIndex); + +// Mark route as OFFLOAD +rtnl_route_set_flags(routeObject.get(), RTM_F_OFFLOAD); + +for (const auto& nexthop: nexthops) +{ + auto* nlNhop = rtnl_route_nh_alloc(); + + rtnl_route_nh_set_ifindex(nlNhop, nhIfIndex); + rtnl_route_nh_set_gateway(nlNhop, nhAddr); + + rtnl_route_add_nexthop(routeObject, nlNhop); +} + +nl_msg* msg; +rtnl_route_build_add_request(routeObject, NLM_F_CREATE, &msg); + +fpm.send(msg); +``` + fpmsyncd reads *bgp-suppress-fib-pending* flag from CONFIG_DB and completely disables *RouteFeedbackChannel* if the feature is disabled. *4*-*7* are not invoked and the flow remains the same as today. #### 7.5. Response Channel Performance considerations From d7e890067e56592b3b56ff2ad54b880bc5345751 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Wed, 19 Oct 2022 14:37:57 +0300 Subject: [PATCH 27/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index dd224b712e..b37b886ad0 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -746,6 +746,7 @@ No new SAI API or changes to SAI design and behaviour needed for this functional #### 9.1. Warm Reboot Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. +When fpmsyncd reaches *reconciled* state it must send RTM_NEWROUTE for all prefixes that were restored previously from APPL_DB since they are already in the ASIC. #### 9.2. Fast Reboot From 2aaf0bbde90eb4edaae992755d74376005e2d298 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Wed, 19 Oct 2022 18:57:25 +0300 Subject: [PATCH 28/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index b37b886ad0..2f1295e9bd 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -294,9 +294,12 @@ fpm address 127.0.0.1 Zebra shall be started with an ```-M dplane_fpm_nl``` to use the new plugin. -New FPM plugin shall be patched the same way as old one to pass VRF if_index in ```rtmsg->rtm_table``` instead of table ID in FPM message. [SONiC FRR patch](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-frr/patch/0003-Use-vrf_id-for-vrf-not-tabled_id.patch). +Some patches that change FPM behavior to the desired SONiC behavior need to be ported to the new plugin: + +- New FPM plugin shall be patched the same way as old one to pass VRF if_index in ```rtmsg->rtm_table``` instead of table ID in FPM message. [SONiC FRR patch](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-frr/patch/0003-Use-vrf_id-for-vrf-not-tabled_id.patch).Since now SONiC will start communicating RTM_NEWROUTE messages back to zebra, similar change has to be made to accept VRF if_index instead of table ID. +- [[PATCH] ignore route from default table](https://github.com/yxieca/sonic-buildimage/blob/26c5bf81f0a8b2f5d6809048bb54a872403726c2/src/sonic-frr/patch/0009-ignore-route-from-default-table.patch) +- [0007-Add-support-of-bgp-l3vni-evpn.patch](https://github.com/sonic-net/sonic-buildimage/blob/a477dbb1751b741ac2762b4328da08b37eb400db/src/sonic-frr/patch/0007-Add-support-of-bgp-l3vni-evpn.patch) -Since now SONiC will start communicating RTM_NEWROUTE messages back to zebra, similar change has to be made to accept VRF if_index instead of table ID. #### 7.2. BGP Docker container startup @@ -675,7 +678,7 @@ rtnl_route_set_flags(routeObject.get(), RTM_F_OFFLOAD); for (const auto& nexthop: nexthops) { auto* nlNhop = rtnl_route_nh_alloc(); - + rtnl_route_nh_set_ifindex(nlNhop, nhIfIndex); rtnl_route_nh_set_gateway(nlNhop, nhAddr); From f4aa91f32de261185e607b01ddd9835b1e6a91e3 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Fri, 21 Oct 2022 16:50:49 +0300 Subject: [PATCH 29/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 41 +++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 2f1295e9bd..b7cf1ad60b 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -94,7 +94,7 @@ FRR supports this through a feature called *BGP suppress FIB pending*. [FRR docu BGP will queue routes which aren't installed in the FIB. To make zebra install routes in FIB an RTM_NEWROUTE message with *RTM_F_OFFLOAD* flag set must be sent by SONiC to zebra through FPM socket. -**NOTE**: This feature is implemented as part of new *dplane_fpm_nl* zebra plugin. SONiC is still using old *fpm* plugin which isn't developed anymore and thus SONiC must migrate to the new implementation as part of this change. +**NOTE**: This feature is implemented as part of new *dplane_fpm_nl* zebra plugin in FRR 8.4 and a backport patch must be created for current FRR 8.2 SONiC is using. SONiC is still using old *fpm* plugin which isn't developed anymore and thus SONiC must migrate to the new implementation as part of this change. ### 4. Requirements @@ -102,10 +102,14 @@ High level requirements: - Provide an end to end ASIC hardware to BGP route programming feedback mechanism - BGP must not advertise routes which aren't yet offloaded if this feature is enabled -- MPLS, VNET routes are out of scope of this document - Both BGP suppression as well as feedback channel are optional and by disabled default - The above configuration can be applied only at startup and cannot be changed while system is running +Restrictions/Limitations: + +- MPLS, VNET routes are out of scope of this document +- Directly connected and static routes are announced by BGP regardless of their offload status + ### 5. Architecture Design @@ -426,6 +430,10 @@ Example data: A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. +*Full mask prefix subnet routes*: + +When orchagent receives a /32 IPv4 or /128 IPv6 prefix which is an IP of a local interface the route programming is skipped as we already have this route programmed with IP2ME action as part of interface configuration. In this case a successful route programming response status must be sent to notify *fpmsyncd* that this prefix is offloaded. + ##### Figure 3. RouteOrch Route Set Flow @@ -606,7 +614,21 @@ sequenceDiagram A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. The offload route status is relevant only for BGP received routes. In order to send back RTM_NEWROUTE only for BGP originated routes it is required to encode "proto" (rtmsg->rtm_protocol field) in APPL_DB which will be sent back as part of response, so that if the "proto" is received for non-BGP route the response to zebra isn't sent. +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. + +The offload route status is relevant only for BGP received routes. In order to send back RTM_NEWROUTE it is required to encode "protocol" (rtmsg->rtm_protocol field) in APPL_DB which will be sent back as part of response: + +``` +127.0.0.1:6379> hgetall "ROUTE_TABLE:193.5.96.0/25" +1) "nexthop" +2) "10.0.0.1,10.0.0.5,10.0.0.9,10.0.0.13" +3) "ifname" +4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" +5) "protocol" +6) "186" +``` + +The protocol number is then set in RTM_NEWROUTE when preparing message to be sent to zebra. ##### Figure 5. FPMsyncd flow @@ -666,7 +688,7 @@ Example pseudo-code snippet of RTM_NEWROUTE construction: auto route = rtnl_route_alloc(); rtnl_route_set_dst(routeObject, dstAddr); -rtnl_route_set_protocol(routeObject, proto); +rtnl_route_set_protocol(routeObject, protocol); rtnl_route_set_family(routeObject.get(), ipFamily); // in SONiC FPM uses vrf if_index instead of table @@ -697,7 +719,7 @@ fpmsyncd reads *bgp-suppress-fib-pending* flag from CONFIG_DB and completely dis Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. -The following table shows the results for publishing 10k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: +The following table shows the results for publishing 1k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: ##### 7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses @@ -735,7 +757,13 @@ flowchart LR A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipeline``` and a flag ```buffered``` to make it use the pipelining. The constructor is similar to one in use with ```ProducerStateTable```: ```c++ -ResponsePublisher::ResponsePublisher(RedisPipeline *pipeline, bool buffered = false); +ResponsePublisher::ResponsePublisher(bool buffered = false); +``` + +And an API to set buffered mode: + +```c++ +void ResponsePublisher::setBuffered(bool buffered); ``` The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when there are no pending tasks to perform. @@ -760,6 +788,7 @@ Considering this, the introduction of a slight delay in advertisement could lead ### 10. Restrictions/Limitations - MPLS, VNET routes are out of scope of this document +- Directly connected and static routes are announced by BGP regardless of their offload status ### 11. Testing Requirements/Design From 43df059a46c52bf7824a900cf08d0ad7d095846d Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 24 Oct 2022 11:47:23 +0300 Subject: [PATCH 30/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index b7cf1ad60b..28b4f129d5 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -625,7 +625,7 @@ The offload route status is relevant only for BGP received routes. In order to s 3) "ifname" 4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" 5) "protocol" -6) "186" +6) "bgp" ``` The protocol number is then set in RTM_NEWROUTE when preparing message to be sent to zebra. From ddfcfe65f4878f3f5dd5df7cbd5b41c101aaf02d Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 24 Oct 2022 13:02:27 +0300 Subject: [PATCH 31/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 28b4f129d5..165fcc166a 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -860,4 +860,3 @@ In order to test this feature end to end it is required to simulate a delay in A ### 12. Open/Action items - if any - Default state of the feature based off minigraph device type -- Non dataplane BGP sessions and whether we have to enable feature per neighbor From 31afc8b5302eb33fb3157e8feaa062a025b035d0 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 24 Oct 2022 13:10:03 +0300 Subject: [PATCH 32/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 165fcc166a..f0a2665a8d 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -783,7 +783,7 @@ When fpmsyncd reaches *reconciled* state it must send RTM_NEWROUTE for all prefi During fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the peer's BGP re-learns advertised routes. -Considering this, the introduction of a slight delay in advertisement could lead to some increase in fast reboot downtime. +Due to additional response publishing in orchagent there might be a slight delay in fast reboot reconfiguration. ### 10. Restrictions/Limitations From cef489cd0fdc6e47530d41af4d3f1f7e1dd38c0d Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 24 Oct 2022 13:12:15 +0300 Subject: [PATCH 33/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index f0a2665a8d..a0138ce409 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -845,13 +845,13 @@ Code coverage is satisfied by UT and the E2E flow is not possible to test with c In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: 1. Enable ```bgp-suppress-fib-pending``` -2. Stop syncd process to simulate a delay: ```kill --SIGSTOP $(pidof syncd)``` +2. Stop orchagent process to simulate a delay: ```kill --SIGSTOP $(pidof orchagent)``` 3. Setup BGP speaker 4. Announces routes to DUT through exabgp 5. Verify BGP session is established 6. Make sure announced BGP route is queued in ```show ip route``` output 7. Verify the route is not announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer -8. Restore syncd process: ```kill --SIGCONT $(pidof syncd)``` +8. Restore orchagent process: ```kill --SIGCONT $(pidof orchagent)``` 8. Make sure route is programmed in ASIC_DB 9. Make sure announced BGP route is FIB synced in ```show ip route``` output 7. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer From 9c6b5d63beabdb1b70a16425285efba33a6def68 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 25 Oct 2022 19:21:55 +0300 Subject: [PATCH 34/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index a0138ce409..521e363f93 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -304,6 +304,7 @@ Some patches that change FPM behavior to the desired SONiC behavior need to be p - [[PATCH] ignore route from default table](https://github.com/yxieca/sonic-buildimage/blob/26c5bf81f0a8b2f5d6809048bb54a872403726c2/src/sonic-frr/patch/0009-ignore-route-from-default-table.patch) - [0007-Add-support-of-bgp-l3vni-evpn.patch](https://github.com/sonic-net/sonic-buildimage/blob/a477dbb1751b741ac2762b4328da08b37eb400db/src/sonic-frr/patch/0007-Add-support-of-bgp-l3vni-evpn.patch) +**Q**: Is there any plan for aligning SONiC to upstream vanilla version FPM implementation? #### 7.2. BGP Docker container startup @@ -660,7 +661,7 @@ sequenceDiagram deactivate RouteSync - APPL_STATE_DB ->> RouteFeedbackChannel: ROUTE_TABLE publish event + APPL_STATE_DB ->> RouteSync: ROUTE_TABLE publish event activate RouteSync RouteSync ->> RouteFeedbackChannel: onRouteResponse() activate RouteFeedbackChannel @@ -846,7 +847,6 @@ In order to test this feature end to end it is required to simulate a delay in A 1. Enable ```bgp-suppress-fib-pending``` 2. Stop orchagent process to simulate a delay: ```kill --SIGSTOP $(pidof orchagent)``` -3. Setup BGP speaker 4. Announces routes to DUT through exabgp 5. Verify BGP session is established 6. Make sure announced BGP route is queued in ```show ip route``` output From 9a201d09a0491ac4aa5edbbd7f81623643400f47 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Tue, 25 Oct 2022 19:28:45 +0300 Subject: [PATCH 35/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 521e363f93..a1ed3026c5 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -777,8 +777,8 @@ No new SAI API or changes to SAI design and behaviour needed for this functional #### 9.1. Warm Reboot -Warm reboot process remains unchanged. With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. -When fpmsyncd reaches *reconciled* state it must send RTM_NEWROUTE for all prefixes that were restored previously from APPL_DB since they are already in the ASIC. +With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. +When fpmsyncd is about to set *reconciled* state it must send RTM_NEWROUTE for all prefixes that were restored previously from APPL_DB since they are already in the ASIC. Once all existing routes are pushed to zebra a comparison logic will finish the warm reboot process and fpmsyncd will continue in a normal operational mode. #### 9.2. Fast Reboot From c3343642bb6962911494f43e829cf5badb646bf2 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Fri, 28 Oct 2022 14:30:37 +0300 Subject: [PATCH 36/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 355 ++++++++++++----------------- 1 file changed, 146 insertions(+), 209 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index a1ed3026c5..9631ef890e 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -1,5 +1,5 @@ -# BGP Supress FIB Pending # +# BGP Suppress Announcements of Routes Not Installed in HW # ## Table of Content @@ -19,6 +19,7 @@ - [7.3. RouteOrch](#73-routeorch) - [7.4. FPMsyncd](#74-fpmsyncd) - [7.5. Response Channel Performance considerations](#75-response-channel-performance-considerations) + - [7.6. Consistency monitoring script and mitigation action](#76-consistency-monitoring-script-and-mitigation-action) - [8. SAI API](#8-sai-api) - [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - [9.1. Warm Reboot](#91-warm-reboot) @@ -43,18 +44,18 @@ This document describes a feedback mechanism that allows BGP not to advertise ro ### 2. Definitions/Abbreviations -| Definitions/Abbreviation | Description | -| ------------------------ | ---------------------------- | +| Definitions/Abbreviation | Description | +| ------------------------ | --------------------------------------- | | ASIC | Application specific integrated circuit | -| HW | Hardware | -| SW | Software | -| BGP | Border Gateway Protocol | -| FRR | Free Range Routing | -| SWSS | Switch state service | -| SYNCD | ASIC synchronization service | -| FPM | Forwarding Plane Manager | -| SAI | Switch Abstraction Interface | -| OID | Object Identifier | +| HW | Hardware | +| SW | Software | +| BGP | Border Gateway Protocol | +| FRR | Free Range Routing | +| SWSS | Switch state service | +| SYNCD | ASIC synchronization service | +| FPM | Forwarding Plane Manager | +| SAI | Switch Abstraction Interface | +| OID | Object Identifier | ### 3. Overview @@ -102,8 +103,9 @@ High level requirements: - Provide an end to end ASIC hardware to BGP route programming feedback mechanism - BGP must not advertise routes which aren't yet offloaded if this feature is enabled -- Both BGP suppression as well as feedback channel are optional and by disabled default -- The above configuration can be applied only at startup and cannot be changed while system is running +- BGP route suppression is optional and disabled by default +- Route suppression can be turned on or off at runtime +- Offloaded routes consistency monitoring and automatic mitigation Restrictions/Limitations: @@ -129,37 +131,14 @@ The below diagram shows the high level architecture including sending an RTM_NEW #### 6.1. Config DB Enhancements -##### 6.1.1. APP_STATE_LOGGING - -Configuration schema in ABNF format: - -```abnf -; APP_STATE_LOGGING table -key = APP_STATE_LOGGING|ROUTE_TABLE ; Configuration for ROUTE_TABLE -state = "enabled"/"disabled" ; Enable/disable response logging for ROUTE_TABLE -``` - -Sample of CONFIG DB snippet given below: - -```json -{ - "APP_STATE_LOGGING": { - "ROUTE_TABLE": { - "state": "enabled" - } - } -} -``` - - -##### 6.1.2. DEVICE_METADATA +##### 6.1.1.DEVICE_METADATA Configuration schema in ABNF format: ```abnf ; DEVICE_METADATA table key = DEVICE_METADATA|localhost ; Device metadata configuration table -bgp-suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable BGP suppress-fib-pending feature, +suppress-pending-fib = "enabled"/"disabled" ; Globally enable/disable routes announcement suppression feature, ; by default this flag is disabled ``` @@ -169,69 +148,35 @@ Sample of CONFIG DB snippet given below: { "DEVICE_METADATA": { "localhost": { - "bgp-suppress-fib-pending": "enabled" + "suppress-pending-fib": "enabled" } } } ``` -This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behaviour as this flag is set to be disabled by default. +This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behavior as this flag is set to be disabled by default. -#### 6.2. Manifest (if the feature is an Application Extension) +This setting can be turned on or off at runtime. When this feature is turned off *fpmsyncd* sends an offload status reply immediately when a route is received from zebra. *fpmsyncd* listens to the changes +in *DEVICE_METADATA* and the new added field and when it gets enabled starts replying to zebra only after a reply from orchagent for all newly added prefixes. This design will make it possible to control +this feature at runtime. When user turns the feature off, newly received routes from zebra will get immediate answer back. -This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. - -#### 6.3. CLI/YANG model Enhancements +In case user detects an issue with a feature (i.e routes aren't marked as offloaded in zebra and thus BGP does not announce them), it can be turned off. -##### 6.3.1. APP_STATE_LOGGING - -A new table ```APP_STATE_LOGGING``` and a corresponding YANG model is added: - -```yang -module sonic-app-state-logging { - yang-version 1.1; +##### 6.1.2. Minigraph - namespace "http://github.com/Azure/sonic-app-state-logging"; - prefix app-state-logging; +*suppress-pending-fib* is enabled when generating CONFIG_DB out of minigraph for device role *LeafRouter*. - description "APP_STATE_LOGGING YANG module for SONiC OS"; - - revision 2022-09-15 { - description "Initial revision"; - } - - container sonic-app-state-logging { - container APP_STATE_LOGGING { - description "Controls the enablement of a response channel per APPL_DB table"; - - container ROUTE_TABLE { - description "Configure response channel for ASIC route configuration"; +#### 6.2. Manifest (if the feature is an Application Extension) - leaf state { - description "Enablement state of response channel for the given table"; - type enumeration { - enum enabled; - enum disabled; - } - default disabled; - } - } - /* end of container ROUTE_TABLE */ - } - /* end of container APP_STATE_LOGGING */ - } - /* end of container of top level container */ -} -/* end of module sonic-app-state-logging */ -``` +This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. -Note that response channel for ROUTE_TABLE can be enabled regardless of ```synchronous_mode``` as we might still get a response from ```RouteOrch``` validation logic as well as ```SAIRedis``` validation. +#### 6.3. CLI/YANG model Enhancements ##### 6.3.2. DEVICE_METADATA -A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```bgp-suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```suppress-pending-fib``` which can be set to ```"enable"``` or ```"disable"```. Snippet of ```sonic-device_metatadata.yang```: @@ -250,22 +195,17 @@ module sonic-device_metadata { description "DEVICE_METADATA part of config_db.json"; container localhost{ - leaf bgp-suppress-fib-pending { - description "Enable BGP suppress FIB pending feature. BGP will wait for route - FIB intallation before announcing routes. This configuration requires - restarting BGP sessions."; + leaf suppress-pending-fib { + description "Enable suppress pending FIB feature. BGP will wait for route + FIB installation before announcing routes."; type enumeration { enum enabled; enum disabled; } default disabled; - must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable' - and /app-state-logging:sonic-app-state-logging/app-state-logging: - APP_STATE_LOGGING/app-state-logging: - ROUTE_TABLE/app-state-logging:state = 'enabled'))" { - error-message "ASIC synchronous mode and APP_STATE_LOGGIN for ROUTE_TABLE must - be enabled in order to enable BGP suppress FIB pending feature"; + must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable'))" { + error-message "ASIC synchronous mode must be enabled in order to enable routes suppression"; } } } @@ -280,7 +220,20 @@ module sonic-device_metadata { This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. -No python ```click```-based CLI command nor ```KLISH``` CLI is planned to be implemented for this functionality. + +#### 6.4. CLI + +A new CLI command is added to control this feature: + +Command to enable feature: +``` +admin@sonic:~$ sudo config suppress-pending-fib enabled +``` + +Command to disable feature: +``` +admin@sonic:~$ sudo config suppress-pending-fib disabled +``` ### 7. High-Level Design @@ -304,29 +257,24 @@ Some patches that change FPM behavior to the desired SONiC behavior need to be p - [[PATCH] ignore route from default table](https://github.com/yxieca/sonic-buildimage/blob/26c5bf81f0a8b2f5d6809048bb54a872403726c2/src/sonic-frr/patch/0009-ignore-route-from-default-table.patch) - [0007-Add-support-of-bgp-l3vni-evpn.patch](https://github.com/sonic-net/sonic-buildimage/blob/a477dbb1751b741ac2762b4328da08b37eb400db/src/sonic-frr/patch/0007-Add-support-of-bgp-l3vni-evpn.patch) -**Q**: Is there any plan for aligning SONiC to upstream vanilla version FPM implementation? +An additional set of patches is required to be ported to FRR 8.2 to support this feature: [Add ability for dplane_fpm_nl to receive RTM_NEWROUTE netlink messages that signal route handling in the asic](https://github.com/stepanblyschak/frr/pull/1). #### 7.2. BGP Docker container startup -BGP configuration template ```bgpd.main.j2``` requires update to support the new field ```bgp-suppress-fib-pending``` and configure FRR accordingly. +BGP configuration template ```bgpd.main.j2``` requires update to configure FRR accordingly. -This configuration is supported in both ```unified``` and ```separated``` FRR configuration modes. Configuration template snippet *bgpd.main.conf.j2*: +Configuration template snippet *bgpd.main.conf.j2*: ```jinja2 router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} -{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} bgp suppress-fib-pending -{% endif %} ``` Zebra shall be started with an option ```--asic-offload=notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: ```jinja2 [program:zebra] -{% if DEVICE_METADATA['localhost']['bgp-suppress-fib-pending'] == 'enabled' %} -{% set zebra_extra_arguments = "--asic-offload=notify_on_offload" } -{% endif %} -command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl {{ zebra_extra_arguments }} +command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl --asic-offload=notify_on_offload ``` Before the route is offloaded ```show ip route``` shows routes as queued: @@ -394,34 +342,34 @@ admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" #### ResponsePublisher in RouteOrch -```RouteOrch``` need to use existing ```ResponsePublisher``` API to be able to publish responses into ```APPL_STATE_DB```. +*RouteOrch* need to use existing *ResponsePublisher* API to be able to publish responses into *APPL_STATE_DB*. -A snippet of ```ResponsePublisher```'s API that is going to be used is given below. The ```state_attrs``` argument is used when a ```SET``` operation for a route entry in ```ROUTE_TABLE``` was performed successfully but the actual set of next hops differs from the intent. When the ```intent_attrs``` vector is empty it implies a ```DEL``` operations. +A snippet of *ResponsePublisher* API that is going to be used is given below. ```c++ void ResponsePublisher::publish(const std::string &table, const std::string &key, const std::vector &intent_attrs, const ReturnCode &status, - const std::vector &state_attrs, bool replace = false) override; ``` -Example usage in ```RouteOrch```: +*RouteOrch* will not publish all fields and values to *APPL_STATE_DB* to reduce memory usage and reduce load on redis database. Instead, the key is published as well as the *protocol* field as this one is required for fpmsyncd to construct RTM_NEWROUTE message. + +Example usage in *RouteOrch*: ```c++ -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(saiStatus), actualFvs, true); +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(SAI_STATUS_SUCCESS), true); ``` - Example data: ``` 127.0.0.1:6379[14]> hgetall "ROUTE_TABLE:193.5.96.0/25" -1) "nexthop" -2) "10.0.0.1,10.0.0.5,10.0.0.9,10.0.0.13" -3) "ifname" -4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" +1) "protocol" +2) "bgp" ``` +In order to delete an entry from *APPL_STATE_DB* it is required to pass an empty *intent_attrs* vector which will imply this is a *DEL* operation. + #### RouteOrch Route Set Flow @@ -429,7 +377,7 @@ Example data: *Temporary route*: -A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status as well as the actual ```nexthop``` field to ```APPL_STATE_DB```. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status regardless as this route is considered as successfully programmed. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. *Full mask prefix subnet routes*: @@ -463,44 +411,25 @@ sequenceDiagram RouteOrch ->> RouteOrch: doTask() loop For each entry in m_toSync - alt NHG exist or can be added - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - else NH can't be added - Note right of RouteOrch: 1 NH is selected and addRoute()
is called with temporary NH - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - end + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker end Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
loop For each route - alt New route - RouteOrch ->> RouteBulker: RouteBulker.flush() - activate RouteBulker - RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - else Existing route - RouteBulker ->> ASIC_DB: sai_route_api->set_route_entries_attribute() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->set_route_entries_attribute() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - end + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() /
sai_route_api->set_route_entry_attribute() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB end RouteBulker -->> RouteOrch:
@@ -509,15 +438,17 @@ sequenceDiagram Note left of RouteOrch: On any route programming error
RouteOrch crashes orchagent.
No response is written
loop For each entry status in RouteBulker + alt Is newly created route RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB:
+ ResponsePublisher ->> APPL_STATE_DB: Set to ROUTE_TABLE activate APPL_STATE_DB APPL_STATE_DB -->> ResponsePublisher:
deactivate APPL_STATE_DB - Note right of RouteOrch: Publish actual FVs
(in case of temporary route only 1 NH is added) + Note right of RouteOrch: Publish status to APPL_STATE_DB if it is a new route ResponsePublisher -->> RouteOrch:
deactivate ResponsePublisher + end end deactivate RouteOrch @@ -586,7 +517,7 @@ sequenceDiagram loop For each status in RouteBulker RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB:
+ ResponsePublisher ->> APPL_STATE_DB: Delete from ROUTE_TABLE activate APPL_STATE_DB APPL_STATE_DB -->> ResponsePublisher:
deactivate APPL_STATE_DB @@ -659,22 +590,29 @@ sequenceDiagram APPL_DB ->> RouteSync:
deactivate APPL_DB + alt Suppression is disabled + Note left of RouteSync: Reply back immidiatelly
when this feature is disabled + RouteSync ->> zebra: Send back RTM_NEWROUTE
with RTM_F_OFFLOAD set + zebra ->> RouteSync:
+ end + deactivate RouteSync APPL_STATE_DB ->> RouteSync: ROUTE_TABLE publish event - activate RouteSync - RouteSync ->> RouteFeedbackChannel: onRouteResponse() - activate RouteFeedbackChannel - alt Successful SET operation - Note right of RouteFeedbackChannel: Construct RTM_NEWROUTE based of
response field value tuples
and set RTM_F_OFFLOAD flag
- RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE - activate zebra - zebra ->> RouteFeedbackChannel:
- deactivate zebra + alt Suppression is enabled + activate RouteSync + RouteSync ->> RouteFeedbackChannel: onRouteResponse() + activate RouteFeedbackChannel + alt Successful SET operation + Note right of RouteFeedbackChannel: Construct RTM_NEWROUTE based of
response field values
and set RTM_F_OFFLOAD flag
+ RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE + activate zebra + zebra ->> RouteFeedbackChannel:
+ deactivate zebra + end + deactivate RouteFeedbackChannel + deactivate RouteSync end - deactivate RouteFeedbackChannel - deactivate RouteSync - ``` *FpmLink* class has to be extended with new API to send netlink message through FPM socket: @@ -698,24 +636,12 @@ rtnl_route_set_table(routeObject.get(), vrfIfIndex); // Mark route as OFFLOAD rtnl_route_set_flags(routeObject.get(), RTM_F_OFFLOAD); -for (const auto& nexthop: nexthops) -{ - auto* nlNhop = rtnl_route_nh_alloc(); - - rtnl_route_nh_set_ifindex(nlNhop, nhIfIndex); - rtnl_route_nh_set_gateway(nlNhop, nhAddr); - - rtnl_route_add_nexthop(routeObject, nlNhop); -} - nl_msg* msg; rtnl_route_build_add_request(routeObject, NLM_F_CREATE, &msg); fpm.send(msg); ``` -fpmsyncd reads *bgp-suppress-fib-pending* flag from CONFIG_DB and completely disables *RouteFeedbackChannel* if the feature is disabled. *4*-*7* are not invoked and the flow remains the same as today. - #### 7.5. Response Channel Performance considerations Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. @@ -769,9 +695,19 @@ void ResponsePublisher::setBuffered(bool buffered); The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when there are no pending tasks to perform. +#### 7.6. Consistency monitoring script and mitigation action + +Routing is a crucial part of a network switch. This feature adds additional flows in existing route processing pipeline and so along the way there might be unexpected failures leading to routes being not marked as offloaded in zebra - missed notification, race conditions leading to in-consistency, etc. It is required to monitor the consistency of routes state periodically and mitigate problems. + +At the moment *route_check.py* verifies routes between APPL_DB & ASIC_DB and makes sure they are in sync. If this script is failed for 3 periodic cycles of monit an error alert is written to the syslog. In addition to that it is required to extend this script to check for consistency APPL_DB, APPL_STATE_DB and zebra by calling ```show ip route json``` and verify every route installed in ASIC_DB is marked as offloaded. If this check fails a specific status code is returned to monit. + +Once monit detects this specific status code has been returned from this script for 3 cycles in a row a mitigation script - *mitigate_routes_offload.py* is invoked. This script publishes required notifications to trigger fpmsyncd flows to send RTM_NEWROUTE message to zebra and log alerts in the syslog. + +This way the problem is mitigated automatically and a network operator is notified via syslog alert about a problem. + ### 8. SAI API -No new SAI API or changes to SAI design and behaviour needed for this functionality. +No new SAI API or changes to SAI design and behavior needed for this functionality. ### 9. Warmboot and Fastboot Design Impact @@ -801,27 +737,18 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati - FPMSyncd: test case regular route 1. Mock FPM interface - 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 - 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format + 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 4. Verify a correct FPM message is being sent to zebra through FPM interface mock 5. Verify RTM_F_OFFLOAD set in route message sent to zebra - FPMSyncd: test case VRF route 1. Mock FPM interface - 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 - 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format - 4. Verify a correct FPM message is being sent to zebra through FPM interface mock - 5. Verify RTM_F_OFFLOAD set in route message sent to zebra + 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 + 3. Verify a correct FPM message is being sent to zebra through FPM interface mock + 4. Verify RTM_F_OFFLOAD set in route message sent to zebra - FPMSyncd: send RTM_DELROUTE 1. Mock FPM interface - 2. Call RouteSync::onRouteMsg() with RTM_DELROUTE message containing route 1.1.1.0/24 - 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB format with delete operation response + 2. Call RouteSync::onRouteResponse() with delete response message containing route 1.1.1.0/24 4. Verify no message is sent to zebra -- FPMSyncd: test only one nexthop is attached to the route - 1. Mock FPM interface - 2. Call RouteSync::onRouteMsg() with RTM_NEWROUTE message containing route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 - 3. Call RouteSync::onRouteResponse() with APPL_STATE_DB containing only one nexthop 2.2.2.2 format - 4. Verify a correct FPM message is being sent to zebra through FPM interface mock with one nexthop - 5. Verify RTM_F_OFFLOAD set in route message sent to zebra - RouteOrch: program ASIC route 1. Create FVS and call RouteOrch::doTask() @@ -831,11 +758,6 @@ Fast/warm reboot regression test suite needs to be ran and verified no degradati 1. Create delete entry and call RouteOrch::doTask() 2. Verify route is delete in SAI 3. Verify the content of APPL_STATE_DB -- RouteOrch: program ASIC route with temporary nexthop - 1. Mock SAI_SWITCH_ATTR_NUMBER_OF_ECMP_GROUPS to 0 - 2. Create FVS and call RouteOrch::doTask() - 3. Verify route is created in SAI with 1 nexthop - 4. Verify the content of APPL_STATE_DB contains only 1 nexthop #### 11.2. VS Test cases @@ -843,20 +765,35 @@ Code coverage is satisfied by UT and the E2E flow is not possible to test with c #### 11.3. System Test cases -In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: + +##### Figure 7. T1 test topology -1. Enable ```bgp-suppress-fib-pending``` -2. Stop orchagent process to simulate a delay: ```kill --SIGSTOP $(pidof orchagent)``` -4. Announces routes to DUT through exabgp -5. Verify BGP session is established -6. Make sure announced BGP route is queued in ```show ip route``` output -7. Verify the route is not announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer -8. Restore orchagent process: ```kill --SIGCONT $(pidof orchagent)``` -8. Make sure route is programmed in ASIC_DB -9. Make sure announced BGP route is FIB synced in ```show ip route``` output -7. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer +

+Figure 7. T1 test topology +

+In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: -### 12. Open/Action items - if any +- **Functional Test** + 1. Enable ```suppress-pending-fib``` + 2. Suspend orchagent process to simulate a delay with ```kill --SIGSTOP $(pidof orchagent)``` + 3. Announces prefix *1.1.1.0/24* (*1::/64*) to DUT through *exabgp* from T0 Arista VM + 4. Make sure announced BGP route is queued in ```show ip route``` output + 5. Verify the route is not announced to Arista T2 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer + 6. Send traffic matching the prefix and verify packets are not forwarded to T0 Arista VM + 7. Restore orchagent process: ```kill --SIGCONT $(pidof orchagent)``` + 8. Make sure route is programmed in ASIC_DB + 9. Make sure announced BGP route is FIB synced in ```show ip route``` output + 10. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer + 11. Send traffic matching the prefix and verify packets are forwarded to T0 Arista VM +- **Performance Test** + 1. TBD +- **Stress Test** + 1. Enable ```suppress-pending-fib``` + 2. In a loop for 100 times: + 1. Announce 10 prefix to DUT through *exabgp* from T0 Arista VM + 2. Withdraw 10 prefix to DUT through *exabgp* from T0 Arista VM + 3. Once reached the required number of cycles the loop breaks after first step + 4. Consistency check is applied, there are no withdrawn routes and announced routes were successfully installed and correctly marked as offloaded in zebra (In case a race condition happens or a notification is missed for some reason this test will try to catch it) -- Default state of the feature based off minigraph device type +### 12. Open/Action items - if any From 8666ce56b4d19471397d8e97c6ed7a74f54a8aef Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Fri, 28 Oct 2022 14:31:23 +0300 Subject: [PATCH 37/43] Add files via upload --- doc/BGP/img/test-topology.svg | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 doc/BGP/img/test-topology.svg diff --git a/doc/BGP/img/test-topology.svg b/doc/BGP/img/test-topology.svg new file mode 100644 index 0000000000..8b8e88754c --- /dev/null +++ b/doc/BGP/img/test-topology.svg @@ -0,0 +1,4 @@ + + + +
SONiC DUT
SONiC DUT
T2 Arista VM
T2 Arista VM
T0 Arista VM
T0 Arista VM
PTF
PTFText is not SVG - cannot display \ No newline at end of file From 249ffc90e0e78ac52b3fdabdaa30fdaabf6908aa Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 17 Nov 2022 12:32:43 +0200 Subject: [PATCH 38/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 9631ef890e..0781161627 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -699,9 +699,7 @@ The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flus Routing is a crucial part of a network switch. This feature adds additional flows in existing route processing pipeline and so along the way there might be unexpected failures leading to routes being not marked as offloaded in zebra - missed notification, race conditions leading to in-consistency, etc. It is required to monitor the consistency of routes state periodically and mitigate problems. -At the moment *route_check.py* verifies routes between APPL_DB & ASIC_DB and makes sure they are in sync. If this script is failed for 3 periodic cycles of monit an error alert is written to the syslog. In addition to that it is required to extend this script to check for consistency APPL_DB, APPL_STATE_DB and zebra by calling ```show ip route json``` and verify every route installed in ASIC_DB is marked as offloaded. If this check fails a specific status code is returned to monit. - -Once monit detects this specific status code has been returned from this script for 3 cycles in a row a mitigation script - *mitigate_routes_offload.py* is invoked. This script publishes required notifications to trigger fpmsyncd flows to send RTM_NEWROUTE message to zebra and log alerts in the syslog. +At the moment *route_check.py* verifies routes between APPL_DB & ASIC_DB and makes sure they are in sync. In addition to that it is required to extend this script to check for consistency APPL_DB, APPL_STATE_DB and zebra by calling ```show ip route json``` and verify every route installed in ASIC_DB is marked as offloaded. The script will retry the check for 3 times every 15 sec and if zebra FIB is not in sync with ASIC_DB a mitigation action is performed. The mitigation action publishes required notifications to trigger fpmsyncd flows to send RTM_NEWROUTE message to zebra and log alerts in the syslog. This way the problem is mitigated automatically and a network operator is notified via syslog alert about a problem. @@ -787,7 +785,11 @@ In order to test this feature end to end it is required to simulate a delay in A 10. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer 11. Send traffic matching the prefix and verify packets are forwarded to T0 Arista VM - **Performance Test** - 1. TBD + 1. Announce 1K routes from exabpg through T0 Arista VM + 2. Send traffic towards these prefixes from PTF in another thread + 3. Measure time from announce start till all packets get received + 4. The test is parametrized for suppression state for comparison + 5. The test passes if the time is under the predefined threshold - **Stress Test** 1. Enable ```suppress-pending-fib``` 2. In a loop for 100 times: From 69b1cd12406acf98b7ac69e0b4721c8fa435aa8a Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Thu, 17 Nov 2022 14:36:41 +0200 Subject: [PATCH 39/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 0781161627..069f0fd366 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -454,8 +454,6 @@ sequenceDiagram deactivate RouteOrch ``` -```RouteOrch``` reads *APP_STATE_LOGGING* table flag from CONFIG_DB and completely disables response publishing if the feature is disabled. *15*-*18* are not invoked and the flow remains the same as today. - #### RouteOrch Route delete Flow From bb09d8b6d3ae491b3bf81a8bd178e4093fe3c551 Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 21 Nov 2022 13:06:57 +0200 Subject: [PATCH 40/43] Update BGP-supress-fib-pending.md --- doc/BGP/BGP-supress-fib-pending.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 069f0fd366..b1100f82c1 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -783,8 +783,8 @@ In order to test this feature end to end it is required to simulate a delay in A 10. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer 11. Send traffic matching the prefix and verify packets are forwarded to T0 Arista VM - **Performance Test** - 1. Announce 1K routes from exabpg through T0 Arista VM - 2. Send traffic towards these prefixes from PTF in another thread + 1. Announce 10K routes to DUT + 2. Send traffic towards these prefixes from AUX 3. Measure time from announce start till all packets get received 4. The test is parametrized for suppression state for comparison 5. The test passes if the time is under the predefined threshold From 1abee247ac1a6865fe58e44c1cb4566d8e362881 Mon Sep 17 00:00:00 2001 From: Stepan Blyschak Date: Fri, 25 Nov 2022 15:12:09 +0200 Subject: [PATCH 41/43] convert line endings Signed-off-by: Stepan Blyschak --- doc/BGP/BGP-supress-fib-pending.md | 1598 ++++++++++++++-------------- 1 file changed, 799 insertions(+), 799 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index b1100f82c1..dddb4b9169 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -1,799 +1,799 @@ - -# BGP Suppress Announcements of Routes Not Installed in HW # - - -## Table of Content - -- [1. Scope](#1-scope) -- [2. Definitions/Abbreviations](#2-definitionsabbreviations) -- [3. Overview](#3-overview) -- [4. Requirements](#4-requirements) -- [5. Architecture Design](#5-architecture-design) -- [6. Configuration and management](#6-configuration-and-management) - - [6.1. Config DB Enhancements](#61-config-db-enhancements) - - [6.2. Manifest (if the feature is an Application Extension)](#62-manifest-if-the-feature-is-an-application-extension) - - [6.3. CLI/YANG model Enhancements](#63-cliyang-model-enhancements) -- [7. High-Level Design](#7-high-level-design) - - [7.1. FPM plugin migration](#71-fpm-plugin-migration) - - [7.2. BGP Docker container startup](#72-bgp-docker-container-startup) - - [7.3. RouteOrch](#73-routeorch) - - [7.4. FPMsyncd](#74-fpmsyncd) - - [7.5. Response Channel Performance considerations](#75-response-channel-performance-considerations) - - [7.6. Consistency monitoring script and mitigation action](#76-consistency-monitoring-script-and-mitigation-action) -- [8. SAI API](#8-sai-api) -- [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) - - [9.1. Warm Reboot](#91-warm-reboot) - - [9.2. Fast Reboot](#92-fast-reboot) -- [10. Restrictions/Limitations](#10-restrictionslimitations) -- [11. Testing Requirements/Design](#11-testing-requirementsdesign) - - [11.1. Unit Test cases](#111-unit-test-cases) - - [11.2. VS Test cases](#112-vs-test-cases) - - [11.3. System Test cases](#113-system-test-cases) -- [12. Open/Action items - if any](#12-openaction-items---if-any) - - -### 1. Revision - -| Revision | Date | Author | Change Description | -| -------- | ----------- | ---------------- | ------------------ | -| 1.0 | Sep 15 2022 | Stepan Blyshchak | Initial proposal | - -### 1. Scope - -This document describes a feedback mechanism that allows BGP not to advertise routes that haven't been programmed yet. - -### 2. Definitions/Abbreviations - -| Definitions/Abbreviation | Description | -| ------------------------ | --------------------------------------- | -| ASIC | Application specific integrated circuit | -| HW | Hardware | -| SW | Software | -| BGP | Border Gateway Protocol | -| FRR | Free Range Routing | -| SWSS | Switch state service | -| SYNCD | ASIC synchronization service | -| FPM | Forwarding Plane Manager | -| SAI | Switch Abstraction Interface | -| OID | Object Identifier | - -### 3. Overview - -As of today, SONiC BGP advertises learnt prefixes regardless of whether these prefixes were successfully programmed into ASIC. While route programming failure is followed by orchagent crash and all services restart, even for successfully created routes there is a short period of time when the peer will be black holing traffic. Also, in the following scenario, a credit loop occurs: - - -##### Figure 1. Use case scenario - -

-Figure 1. Use case scenario -

- -The problem with BGP programming occurs after the T1 switch is rebooted: -1. First, the T1 FRR learns a default route from at least 1 T2 -2. The T0 advertises it’s prefixes to T1 -3. FRR advertises the prefixes to T2 without waiting for them to be programmed in the ASIC -4. T2 starts forwarding traffic for prefixes not yet programmed, according to T1’s routing table, T1 sends it back to a default route – same T2 - -When the traffic is bounced back on lossless queue, buffers on both sides are overflown, credit loop happens, with PFC storm and watchdog triggered shutting down the port. -To avoid that, the route programming has to be synchronous down to the ASIC to avoid credit loops. - -FRR supports this through a feature called *BGP suppress FIB pending*. [FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst): - -> The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. -> -> The solution is to provide a configurable option to check for the FIB install status of the prefixes and advertise to peers if the prefixes are successfully installed in the FIB. The advertisement of the prefixes are suppressed if it is not installed in FIB. -> -> The following conditions apply will apply when checking for route installation status in FIB: -> -> - The advertisement or suppression of routes based on FIB install status applies only for newly learnt routes from peer (routes which are not in BGP local RIB). -> - If the route received from peer already exists in BGP local RIB and route attributes have changed (best path changed), the old path is deleted and new path is installed in FIB. The FIB install status will not have any effect. Therefore only when the route is received first time the checks apply. -> - The feature will not apply for routes learnt through other means like redistribution to bgp from other protocols. This is applicable only to peer learnt routes. -> - If a route is installed in FIB and then gets deleted from the dataplane, then routes will not be withdrawn from peers. This will be considered as dataplane issue. -> - The feature will slightly increase the time required to advertise the routes to peers since the route install status needs to be received from the FIB -> - If routes are received by the peer before the configuration is applied, then the bgp sessions need to be reset for the configuration to take effect. -> - If the route which is already installed in dataplane is removed for some reason, sending withdraw message to peers is not currently supported. - -BGP will queue routes which aren't installed in the FIB. To make zebra install routes in FIB an RTM_NEWROUTE message with *RTM_F_OFFLOAD* flag set must be sent by SONiC to zebra through FPM socket. - -**NOTE**: This feature is implemented as part of new *dplane_fpm_nl* zebra plugin in FRR 8.4 and a backport patch must be created for current FRR 8.2 SONiC is using. SONiC is still using old *fpm* plugin which isn't developed anymore and thus SONiC must migrate to the new implementation as part of this change. - -### 4. Requirements - -High level requirements: - -- Provide an end to end ASIC hardware to BGP route programming feedback mechanism -- BGP must not advertise routes which aren't yet offloaded if this feature is enabled -- BGP route suppression is optional and disabled by default -- Route suppression can be turned on or off at runtime -- Offloaded routes consistency monitoring and automatic mitigation - -Restrictions/Limitations: - -- MPLS, VNET routes are out of scope of this document -- Directly connected and static routes are announced by BGP regardless of their offload status - -### 5. Architecture Design - - -##### Figure 2. Architecture - -The below diagram shows the high level architecture including sending an RTM_NEWROUTE response message with RTM_F_OFFLOAD flag set to zebra through FPM: - -

-Figure 2. Architecture diagram -

- -- *2*-*8* - existing flows in SONiC -- *1*, *9*, *10* - new flows to be added - -### 6. Configuration and management - -#### 6.1. Config DB Enhancements - - -##### 6.1.1.DEVICE_METADATA - -Configuration schema in ABNF format: - -```abnf -; DEVICE_METADATA table -key = DEVICE_METADATA|localhost ; Device metadata configuration table -suppress-pending-fib = "enabled"/"disabled" ; Globally enable/disable routes announcement suppression feature, - ; by default this flag is disabled -``` - -Sample of CONFIG DB snippet given below: - -```json -{ - "DEVICE_METADATA": { - "localhost": { - "suppress-pending-fib": "enabled" - } - } -} -``` - -This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behavior as this flag is set to be disabled by default. - -This setting can be turned on or off at runtime. When this feature is turned off *fpmsyncd* sends an offload status reply immediately when a route is received from zebra. *fpmsyncd* listens to the changes -in *DEVICE_METADATA* and the new added field and when it gets enabled starts replying to zebra only after a reply from orchagent for all newly added prefixes. This design will make it possible to control -this feature at runtime. When user turns the feature off, newly received routes from zebra will get immediate answer back. - -In case user detects an issue with a feature (i.e routes aren't marked as offloaded in zebra and thus BGP does not announce them), it can be turned off. - - -##### 6.1.2. Minigraph - -*suppress-pending-fib* is enabled when generating CONFIG_DB out of minigraph for device role *LeafRouter*. - -#### 6.2. Manifest (if the feature is an Application Extension) - -This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. - -#### 6.3. CLI/YANG model Enhancements - - -##### 6.3.2. DEVICE_METADATA - -A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```suppress-pending-fib``` which can be set to ```"enable"``` or ```"disable"```. - -Snippet of ```sonic-device_metatadata.yang```: - -```yang -module sonic-device_metadata { - import sonic-app-state-logging { - prefix app-state-logging; - } - - revision 2022-09-15 { - description "Add BGP suppress FIB pending configuration knob"; - } - - container sonic-device_metadata { - container DEVICE_METADATA { - description "DEVICE_METADATA part of config_db.json"; - - container localhost{ - leaf suppress-pending-fib { - description "Enable suppress pending FIB feature. BGP will wait for route - FIB installation before announcing routes."; - type enumeration { - enum enabled; - enum disabled; - } - default disabled; - - must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable'))" { - error-message "ASIC synchronous mode must be enabled in order to enable routes suppression"; - } - } - } - /* end of container localhost */ - } - /* end of container DEVICE_METADATA */ - } - /* end of top level container */ -} -/* end of module sonic-device_metadata -``` - -This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. - - -#### 6.4. CLI - -A new CLI command is added to control this feature: - -Command to enable feature: -``` -admin@sonic:~$ sudo config suppress-pending-fib enabled -``` - -Command to disable feature: -``` -admin@sonic:~$ sudo config suppress-pending-fib disabled -``` - -### 7. High-Level Design - -#### 7.1. FPM plugin migration - -FPM response channel is only supported with new ```dplane_fpm_nl``` zebra plugin and SONiC needs start using the new and supported plugin. Additional configuration is required in *zebra.conf.j2*: - -``` -! Fallback to nexthops information as part of RTM_NEWROUTE message -no fpm use-next-hop-groups - -! Configure FPM server address -fpm address 127.0.0.1 -``` - -Zebra shall be started with an ```-M dplane_fpm_nl``` to use the new plugin. - -Some patches that change FPM behavior to the desired SONiC behavior need to be ported to the new plugin: - -- New FPM plugin shall be patched the same way as old one to pass VRF if_index in ```rtmsg->rtm_table``` instead of table ID in FPM message. [SONiC FRR patch](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-frr/patch/0003-Use-vrf_id-for-vrf-not-tabled_id.patch).Since now SONiC will start communicating RTM_NEWROUTE messages back to zebra, similar change has to be made to accept VRF if_index instead of table ID. -- [[PATCH] ignore route from default table](https://github.com/yxieca/sonic-buildimage/blob/26c5bf81f0a8b2f5d6809048bb54a872403726c2/src/sonic-frr/patch/0009-ignore-route-from-default-table.patch) -- [0007-Add-support-of-bgp-l3vni-evpn.patch](https://github.com/sonic-net/sonic-buildimage/blob/a477dbb1751b741ac2762b4328da08b37eb400db/src/sonic-frr/patch/0007-Add-support-of-bgp-l3vni-evpn.patch) - -An additional set of patches is required to be ported to FRR 8.2 to support this feature: [Add ability for dplane_fpm_nl to receive RTM_NEWROUTE netlink messages that signal route handling in the asic](https://github.com/stepanblyschak/frr/pull/1). - -#### 7.2. BGP Docker container startup - -BGP configuration template ```bgpd.main.j2``` requires update to configure FRR accordingly. - -Configuration template snippet *bgpd.main.conf.j2*: - -```jinja2 -router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} - bgp suppress-fib-pending -``` - -Zebra shall be started with an option ```--asic-offload=notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: - -```jinja2 -[program:zebra] -command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl --asic-offload=notify_on_offload -``` - -Before the route is offloaded ```show ip route``` shows routes as queued: - -``` -admin@sonic:~$ show ip route -Codes: K - kernel route, C - connected, S - static, R - RIP, - O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, - T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, - f - OpenFabric, - > - selected route, * - FIB route, q - queued, r - rejected, b - backup - t - trapped, o - offload failure - -B>q 0.0.0.0/0 [20/0] via 10.0.0.1, PortChannel102, weight 1, 00:04:46 - q via 10.0.0.5, PortChannel105, weight 1, 00:04:46 - q via 10.0.0.9, PortChannel108, weight 1, 00:04:46 - q via 10.0.0.13, PortChannel1011, weight 1, 00:04:46 -``` - -Once zebra is notified about successfull route offload status the *offload* flag is set: - -``` -admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" -{ - "100.1.0.25/32": [ - { - "prefix": "100.1.0.25/32", - "prefixLen": 32, - "protocol": "bgp", - "vrfId": 0, - "vrfName": "default", - "selected": true, - "destSelected": true, - "distance": 20, - "metric": 0, - "installed": true, - "offloaded": true, - "table": 254, - "internalStatus": 80, - "internalFlags": 264, - "internalNextHopNum": 1, - "internalNextHopActiveNum": 1, - "nexthopGroupId": 4890, - "installedNexthopGroupId": 4890, - "uptime": "00:22:20", - "nexthops": [ - { - "flags": 3, - "fib": true, - "ip": "10.0.0.49", - "afi": "ipv4", - "interfaceIndex": 788, - "interfaceName": "PortChannel1013", - "active": true, - "weight": 1 - } - ] - } - ] -} -``` - -#### 7.3. RouteOrch - - -#### ResponsePublisher in RouteOrch - -*RouteOrch* need to use existing *ResponsePublisher* API to be able to publish responses into *APPL_STATE_DB*. - -A snippet of *ResponsePublisher* API that is going to be used is given below. - -```c++ -void ResponsePublisher::publish(const std::string &table, const std::string &key, - const std::vector &intent_attrs, - const ReturnCode &status, - bool replace = false) override; -``` - -*RouteOrch* will not publish all fields and values to *APPL_STATE_DB* to reduce memory usage and reduce load on redis database. Instead, the key is published as well as the *protocol* field as this one is required for fpmsyncd to construct RTM_NEWROUTE message. - -Example usage in *RouteOrch*: - -```c++ -m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(SAI_STATUS_SUCCESS), true); -``` - -Example data: -``` -127.0.0.1:6379[14]> hgetall "ROUTE_TABLE:193.5.96.0/25" -1) "protocol" -2) "bgp" -``` - -In order to delete an entry from *APPL_STATE_DB* it is required to pass an empty *intent_attrs* vector which will imply this is a *DEL* operation. - - -#### RouteOrch Route Set Flow - -```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. - -*Temporary route*: - -A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status regardless as this route is considered as successfully programmed. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. - -*Full mask prefix subnet routes*: - -When orchagent receives a /32 IPv4 or /128 IPv6 prefix which is an IP of a local interface the route programming is skipped as we already have this route programmed with IP2ME action as part of interface configuration. In this case a successful route programming response status must be sent to notify *fpmsyncd* that this prefix is offloaded. - - -##### Figure 3. RouteOrch Route Set Flow - -```mermaid -%%{ - init: { - "theme": "forest", - "sequence": { - "rightAngles": true, - "showSequenceNumbers": true - } - } -}%% -sequenceDiagram - participant APPL_DB - participant RouteOrch - participant RouteBulker - participant ResponsePublisher - participant APPL_STATE_DB - participant ASIC_DB - participant SYNCD - - APPL_DB --) RouteOrch: ROUTE_TABLE Notification - activate RouteOrch - - RouteOrch ->> RouteOrch: doTask() - - loop For each entry in m_toSync - RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - end - - Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
- - loop For each route - RouteOrch ->> RouteBulker: RouteBulker.flush() - activate RouteBulker - RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() /
sai_route_api->set_route_entry_attribute() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - end - - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - - Note left of RouteOrch: On any route programming error
RouteOrch crashes orchagent.
No response is written
- - loop For each entry status in RouteBulker - alt Is newly created route - RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() - activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB: Set to ROUTE_TABLE - activate APPL_STATE_DB - APPL_STATE_DB -->> ResponsePublisher:
- deactivate APPL_STATE_DB - Note right of RouteOrch: Publish status to APPL_STATE_DB if it is a new route - ResponsePublisher -->> RouteOrch:
- deactivate ResponsePublisher - end - end - - deactivate RouteOrch -``` - - - -#### RouteOrch Route delete Flow - - -##### Figure 4. RouteOrch Route Delete Flow - -Similar processing happens on ```DEL``` operation for a prefix from ```ROUTE_TABLE```. The removed prefixes are filled in ```RouteBulker``` which are then flushed forming a bulk remove API call to SAIRedis. ```RouteOrch::removeRoutePost()``` then collects the object statuses and if the status is not SAI_STATUS_SUCCESS orchagent ```abort()```s, otherwise it should publish the result via ```ResponsePublisher::publish()``` leaving ```intent_attrs``` vector empty which will remove the corresponding state key from ```APPL_STATE_DB```. - -```mermaid -%%{ - init: { - "theme": "forest", - "sequence": { - "rightAngles": true, - "showSequenceNumbers": true - } - } -}%% -sequenceDiagram - participant APPL_DB - participant RouteOrch - participant RouteBulker - participant ResponsePublisher - participant APPL_STATE_DB - participant ASIC_DB - participant SYNCD - - APPL_DB --) RouteOrch: ROUTE_TABLE Notification - activate RouteOrch - - RouteOrch ->> RouteOrch: doTask() - - loop For each entry in m_toSync - RouteOrch ->> RouteBulker: RouteBulker.remove_entry() - activate RouteBulker - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - end - - Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to remove
the route on next iteration
In this case no response is written
- - RouteOrch ->> RouteBulker: RouteBulker.flush() - activate RouteBulker - RouteBulker ->> ASIC_DB: sai_route_api->remove_route_entries() - activate ASIC_DB - ASIC_DB ->> SYNCD: sai_route_api->remove_route_entries() - activate SYNCD - SYNCD -->> ASIC_DB: sai_status_t *object_statuses - deactivate SYNCD - ASIC_DB -->> RouteBulker: sai_status_t *object_statuses - deactivate ASIC_DB - - RouteBulker -->> RouteOrch:
- deactivate RouteBulker - - Note left of RouteOrch: On any route deletion error
RouteOrch crashes orchagent.
No response is written
- - loop For each status in RouteBulker - RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() - activate ResponsePublisher - ResponsePublisher ->> APPL_STATE_DB: Delete from ROUTE_TABLE - activate APPL_STATE_DB - APPL_STATE_DB -->> ResponsePublisher:
- deactivate APPL_STATE_DB - Note right of RouteOrch: Publish the status and remove entry from APPL_STATE_DB - ResponsePublisher -->> RouteOrch:
- deactivate ResponsePublisher - end - - deactivate RouteOrch -``` - -#### 7.4. FPMsyncd - - -> The dplane_fpm_nl has the ability to read route netlink messages -> from the underlying fpm implementation that can tell zebra -> whether or not the route has been Offloaded/Failed or Trapped. -> The end developer must send the data up the same socket that has -> been created to listen for FPM messages from Zebra. The data sent -> must have a Frame Header with Version set to 1, Message Type set to 1 -> and an appropriate message Length. The message data must contain -> a RTM_NEWROUTE netlink message that sends the prefix and nexthops -> associated with the route. Finally rtm_flags must contain -> RTM_F_OFFLOAD, RTM_F_TRAP and or RTM_F_OFFLOAD_FAILED to signify -> what has happened to the route in the ASIC. - -A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. - -Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. - -The offload route status is relevant only for BGP received routes. In order to send back RTM_NEWROUTE it is required to encode "protocol" (rtmsg->rtm_protocol field) in APPL_DB which will be sent back as part of response: - -``` -127.0.0.1:6379> hgetall "ROUTE_TABLE:193.5.96.0/25" -1) "nexthop" -2) "10.0.0.1,10.0.0.5,10.0.0.9,10.0.0.13" -3) "ifname" -4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" -5) "protocol" -6) "bgp" -``` - -The protocol number is then set in RTM_NEWROUTE when preparing message to be sent to zebra. - - -##### Figure 5. FPMsyncd flow - -```mermaid -%%{ - init: { - "theme": "forest", - "sequence": { - "rightAngles": true, - "showSequenceNumbers": true - } - } -}%% -sequenceDiagram - participant zebra - participant RouteSync - participant RouteFeedbackChannel - participant APPL_DB - participant APPL_STATE_DB - - - zebra ->> RouteSync: RTM_NEWROUTE FPM message - activate RouteSync - RouteSync ->> APPL_DB: Write entry to ROUTE_TABLE - activate APPL_DB - APPL_DB ->> RouteSync:
- deactivate APPL_DB - - alt Suppression is disabled - Note left of RouteSync: Reply back immidiatelly
when this feature is disabled - RouteSync ->> zebra: Send back RTM_NEWROUTE
with RTM_F_OFFLOAD set - zebra ->> RouteSync:
- end - - deactivate RouteSync - - APPL_STATE_DB ->> RouteSync: ROUTE_TABLE publish event - alt Suppression is enabled - activate RouteSync - RouteSync ->> RouteFeedbackChannel: onRouteResponse() - activate RouteFeedbackChannel - alt Successful SET operation - Note right of RouteFeedbackChannel: Construct RTM_NEWROUTE based of
response field values
and set RTM_F_OFFLOAD flag
- RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE - activate zebra - zebra ->> RouteFeedbackChannel:
- deactivate zebra - end - deactivate RouteFeedbackChannel - deactivate RouteSync - end -``` - -*FpmLink* class has to be extended with new API to send netlink message through FPM socket: - -```c++ -ssize_t FpmLink::send(nl_msg* msg) -``` - -Example pseudo-code snippet of RTM_NEWROUTE construction: - -```c++ -auto route = rtnl_route_alloc(); - -rtnl_route_set_dst(routeObject, dstAddr); -rtnl_route_set_protocol(routeObject, protocol); -rtnl_route_set_family(routeObject.get(), ipFamily); - -// in SONiC FPM uses vrf if_index instead of table -rtnl_route_set_table(routeObject.get(), vrfIfIndex); - -// Mark route as OFFLOAD -rtnl_route_set_flags(routeObject.get(), RTM_F_OFFLOAD); - -nl_msg* msg; -rtnl_route_build_add_request(routeObject, NLM_F_CREATE, &msg); - -fpm.send(msg); -``` - -#### 7.5. Response Channel Performance considerations - -Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. - -The following table shows the results for publishing 1k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: - - -##### 7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses - -| Scenario | Time (sec) | Ratio | -| ---------------------- | ---------- | ----- | -| Without Redis Pipeline | 0.641 | 4.27 | -| With Redis Pipeline | 0.150 | 1 | - -Adding a feedback mechanism to the system introduces a delay as each of the component needs to wait for the reply from the lower layer counterpart in order to proceed. SONiC has already moved to synchronous SAI Redis pipeline a route programming performance degradation caused by it is leveled by the use of SAIRedis Bulk API. - -By introducing a response channel it is required to leverage Redis Pipeline, so that the route configuration producer using Redis Pipeline with ```ProducerStateTable``` also receives route programming status responses produced by pipelined ```NotificationProducer``` which is part of ```ResponsePublisher```. - -On the other side, ```fpmsyncd``` does not wait for each individual route status but rather performs an asynchronous processing. - -The following schema represents communication methods between components: - - -##### Figure 6. zebra/orchagent feedback channel -```mermaid -%%{ - init: { - "theme": "forest" - } -}%% -flowchart LR - zebra("
zebra

") -- FPM channel --> fpmsyncd("
fpmsyncd

") - fpmsyncd -- "ProducerStateTable (Pipelined)" --> RouteOrch("
RouteOrch

") - RouteOrch -. "ResponsePublisher (Pipelined)" -.-> fpmsyncd - fpmsyncd -. FPM channel -.-> zebra - RouteOrch -- "SAIRedis Bulk API" --> syncd("
syncd

") - syncd -. "SAIRedis Bulk Reply" -.-> RouteOrch -``` - -A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipeline``` and a flag ```buffered``` to make it use the pipelining. The constructor is similar to one in use with ```ProducerStateTable```: - -```c++ -ResponsePublisher::ResponsePublisher(bool buffered = false); -``` - -And an API to set buffered mode: - -```c++ -void ResponsePublisher::setBuffered(bool buffered); -``` - -The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when there are no pending tasks to perform. - -#### 7.6. Consistency monitoring script and mitigation action - -Routing is a crucial part of a network switch. This feature adds additional flows in existing route processing pipeline and so along the way there might be unexpected failures leading to routes being not marked as offloaded in zebra - missed notification, race conditions leading to in-consistency, etc. It is required to monitor the consistency of routes state periodically and mitigate problems. - -At the moment *route_check.py* verifies routes between APPL_DB & ASIC_DB and makes sure they are in sync. In addition to that it is required to extend this script to check for consistency APPL_DB, APPL_STATE_DB and zebra by calling ```show ip route json``` and verify every route installed in ASIC_DB is marked as offloaded. The script will retry the check for 3 times every 15 sec and if zebra FIB is not in sync with ASIC_DB a mitigation action is performed. The mitigation action publishes required notifications to trigger fpmsyncd flows to send RTM_NEWROUTE message to zebra and log alerts in the syslog. - -This way the problem is mitigated automatically and a network operator is notified via syslog alert about a problem. - -### 8. SAI API - -No new SAI API or changes to SAI design and behavior needed for this functionality. - -### 9. Warmboot and Fastboot Design Impact - -#### 9.1. Warm Reboot - -With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. -When fpmsyncd is about to set *reconciled* state it must send RTM_NEWROUTE for all prefixes that were restored previously from APPL_DB since they are already in the ASIC. Once all existing routes are pushed to zebra a comparison logic will finish the warm reboot process and fpmsyncd will continue in a normal operational mode. - -#### 9.2. Fast Reboot - -During fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the peer's BGP re-learns advertised routes. - -Due to additional response publishing in orchagent there might be a slight delay in fast reboot reconfiguration. - -### 10. Restrictions/Limitations - -- MPLS, VNET routes are out of scope of this document -- Directly connected and static routes are announced by BGP regardless of their offload status - -### 11. Testing Requirements/Design - -Regression test, VS test and unit testing are required for this functionality. - -Fast/warm reboot regression test suite needs to be ran and verified no degradation introduced by the feature. - -#### 11.1. Unit Test cases - -- FPMSyncd: test case regular route - 1. Mock FPM interface - 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 - 4. Verify a correct FPM message is being sent to zebra through FPM interface mock - 5. Verify RTM_F_OFFLOAD set in route message sent to zebra -- FPMSyncd: test case VRF route - 1. Mock FPM interface - 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 - 3. Verify a correct FPM message is being sent to zebra through FPM interface mock - 4. Verify RTM_F_OFFLOAD set in route message sent to zebra -- FPMSyncd: send RTM_DELROUTE - 1. Mock FPM interface - 2. Call RouteSync::onRouteResponse() with delete response message containing route 1.1.1.0/24 - 4. Verify no message is sent to zebra - -- RouteOrch: program ASIC route - 1. Create FVS and call RouteOrch::doTask() - 2. Verify route is created in SAI - 3. Verify the content of APPL_STATE_DB -- RouteOrch: delete ASIC route - 1. Create delete entry and call RouteOrch::doTask() - 2. Verify route is delete in SAI - 3. Verify the content of APPL_STATE_DB - -#### 11.2. VS Test cases - -Code coverage is satisfied by UT and the E2E flow is not possible to test with current VS infrastructure, so no VS tests are planned. - -#### 11.3. System Test cases - - -##### Figure 7. T1 test topology - -

-Figure 7. T1 test topology -

- -In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: - -- **Functional Test** - 1. Enable ```suppress-pending-fib``` - 2. Suspend orchagent process to simulate a delay with ```kill --SIGSTOP $(pidof orchagent)``` - 3. Announces prefix *1.1.1.0/24* (*1::/64*) to DUT through *exabgp* from T0 Arista VM - 4. Make sure announced BGP route is queued in ```show ip route``` output - 5. Verify the route is not announced to Arista T2 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer - 6. Send traffic matching the prefix and verify packets are not forwarded to T0 Arista VM - 7. Restore orchagent process: ```kill --SIGCONT $(pidof orchagent)``` - 8. Make sure route is programmed in ASIC_DB - 9. Make sure announced BGP route is FIB synced in ```show ip route``` output - 10. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer - 11. Send traffic matching the prefix and verify packets are forwarded to T0 Arista VM -- **Performance Test** - 1. Announce 10K routes to DUT - 2. Send traffic towards these prefixes from AUX - 3. Measure time from announce start till all packets get received - 4. The test is parametrized for suppression state for comparison - 5. The test passes if the time is under the predefined threshold -- **Stress Test** - 1. Enable ```suppress-pending-fib``` - 2. In a loop for 100 times: - 1. Announce 10 prefix to DUT through *exabgp* from T0 Arista VM - 2. Withdraw 10 prefix to DUT through *exabgp* from T0 Arista VM - 3. Once reached the required number of cycles the loop breaks after first step - 4. Consistency check is applied, there are no withdrawn routes and announced routes were successfully installed and correctly marked as offloaded in zebra (In case a race condition happens or a notification is missed for some reason this test will try to catch it) - -### 12. Open/Action items - if any + +# BGP Suppress Announcements of Routes Not Installed in HW # + + +## Table of Content + +- [1. Scope](#1-scope) +- [2. Definitions/Abbreviations](#2-definitionsabbreviations) +- [3. Overview](#3-overview) +- [4. Requirements](#4-requirements) +- [5. Architecture Design](#5-architecture-design) +- [6. Configuration and management](#6-configuration-and-management) + - [6.1. Config DB Enhancements](#61-config-db-enhancements) + - [6.2. Manifest (if the feature is an Application Extension)](#62-manifest-if-the-feature-is-an-application-extension) + - [6.3. CLI/YANG model Enhancements](#63-cliyang-model-enhancements) +- [7. High-Level Design](#7-high-level-design) + - [7.1. FPM plugin migration](#71-fpm-plugin-migration) + - [7.2. BGP Docker container startup](#72-bgp-docker-container-startup) + - [7.3. RouteOrch](#73-routeorch) + - [7.4. FPMsyncd](#74-fpmsyncd) + - [7.5. Response Channel Performance considerations](#75-response-channel-performance-considerations) + - [7.6. Consistency monitoring script and mitigation action](#76-consistency-monitoring-script-and-mitigation-action) +- [8. SAI API](#8-sai-api) +- [9. Warmboot and Fastboot Design Impact](#9-warmboot-and-fastboot-design-impact) + - [9.1. Warm Reboot](#91-warm-reboot) + - [9.2. Fast Reboot](#92-fast-reboot) +- [10. Restrictions/Limitations](#10-restrictionslimitations) +- [11. Testing Requirements/Design](#11-testing-requirementsdesign) + - [11.1. Unit Test cases](#111-unit-test-cases) + - [11.2. VS Test cases](#112-vs-test-cases) + - [11.3. System Test cases](#113-system-test-cases) +- [12. Open/Action items - if any](#12-openaction-items---if-any) + + +### 1. Revision + +| Revision | Date | Author | Change Description | +| -------- | ----------- | ---------------- | ------------------ | +| 1.0 | Sep 15 2022 | Stepan Blyshchak | Initial proposal | + +### 1. Scope + +This document describes a feedback mechanism that allows BGP not to advertise routes that haven't been programmed yet. + +### 2. Definitions/Abbreviations + +| Definitions/Abbreviation | Description | +| ------------------------ | --------------------------------------- | +| ASIC | Application specific integrated circuit | +| HW | Hardware | +| SW | Software | +| BGP | Border Gateway Protocol | +| FRR | Free Range Routing | +| SWSS | Switch state service | +| SYNCD | ASIC synchronization service | +| FPM | Forwarding Plane Manager | +| SAI | Switch Abstraction Interface | +| OID | Object Identifier | + +### 3. Overview + +As of today, SONiC BGP advertises learnt prefixes regardless of whether these prefixes were successfully programmed into ASIC. While route programming failure is followed by orchagent crash and all services restart, even for successfully created routes there is a short period of time when the peer will be black holing traffic. Also, in the following scenario, a credit loop occurs: + + +##### Figure 1. Use case scenario + +

+Figure 1. Use case scenario +

+ +The problem with BGP programming occurs after the T1 switch is rebooted: +1. First, the T1 FRR learns a default route from at least 1 T2 +2. The T0 advertises it’s prefixes to T1 +3. FRR advertises the prefixes to T2 without waiting for them to be programmed in the ASIC +4. T2 starts forwarding traffic for prefixes not yet programmed, according to T1’s routing table, T1 sends it back to a default route – same T2 + +When the traffic is bounced back on lossless queue, buffers on both sides are overflown, credit loop happens, with PFC storm and watchdog triggered shutting down the port. +To avoid that, the route programming has to be synchronous down to the ASIC to avoid credit loops. + +FRR supports this through a feature called *BGP suppress FIB pending*. [FRR documentation reference](https://github.com/FRRouting/frr/blob/master/doc/user/bgp.rst): + +> The FRR implementation of BGP advertises prefixes learnt from a peer to other peers even if the routes do not get installed in the FIB. There can be scenarios where the hardware tables in some of the routers (along the path from the source to destination) is full which will result in all routes not getting installed in the FIB. If these routes are advertised to the downstream routers then traffic will start flowing and will be dropped at the intermediate router. +> +> The solution is to provide a configurable option to check for the FIB install status of the prefixes and advertise to peers if the prefixes are successfully installed in the FIB. The advertisement of the prefixes are suppressed if it is not installed in FIB. +> +> The following conditions apply will apply when checking for route installation status in FIB: +> +> - The advertisement or suppression of routes based on FIB install status applies only for newly learnt routes from peer (routes which are not in BGP local RIB). +> - If the route received from peer already exists in BGP local RIB and route attributes have changed (best path changed), the old path is deleted and new path is installed in FIB. The FIB install status will not have any effect. Therefore only when the route is received first time the checks apply. +> - The feature will not apply for routes learnt through other means like redistribution to bgp from other protocols. This is applicable only to peer learnt routes. +> - If a route is installed in FIB and then gets deleted from the dataplane, then routes will not be withdrawn from peers. This will be considered as dataplane issue. +> - The feature will slightly increase the time required to advertise the routes to peers since the route install status needs to be received from the FIB +> - If routes are received by the peer before the configuration is applied, then the bgp sessions need to be reset for the configuration to take effect. +> - If the route which is already installed in dataplane is removed for some reason, sending withdraw message to peers is not currently supported. + +BGP will queue routes which aren't installed in the FIB. To make zebra install routes in FIB an RTM_NEWROUTE message with *RTM_F_OFFLOAD* flag set must be sent by SONiC to zebra through FPM socket. + +**NOTE**: This feature is implemented as part of new *dplane_fpm_nl* zebra plugin in FRR 8.4 and a backport patch must be created for current FRR 8.2 SONiC is using. SONiC is still using old *fpm* plugin which isn't developed anymore and thus SONiC must migrate to the new implementation as part of this change. + +### 4. Requirements + +High level requirements: + +- Provide an end to end ASIC hardware to BGP route programming feedback mechanism +- BGP must not advertise routes which aren't yet offloaded if this feature is enabled +- BGP route suppression is optional and disabled by default +- Route suppression can be turned on or off at runtime +- Offloaded routes consistency monitoring and automatic mitigation + +Restrictions/Limitations: + +- MPLS, VNET routes are out of scope of this document +- Directly connected and static routes are announced by BGP regardless of their offload status + +### 5. Architecture Design + + +##### Figure 2. Architecture + +The below diagram shows the high level architecture including sending an RTM_NEWROUTE response message with RTM_F_OFFLOAD flag set to zebra through FPM: + +

+Figure 2. Architecture diagram +

+ +- *2*-*8* - existing flows in SONiC +- *1*, *9*, *10* - new flows to be added + +### 6. Configuration and management + +#### 6.1. Config DB Enhancements + + +##### 6.1.1.DEVICE_METADATA + +Configuration schema in ABNF format: + +```abnf +; DEVICE_METADATA table +key = DEVICE_METADATA|localhost ; Device metadata configuration table +suppress-pending-fib = "enabled"/"disabled" ; Globally enable/disable routes announcement suppression feature, + ; by default this flag is disabled +``` + +Sample of CONFIG DB snippet given below: + +```json +{ + "DEVICE_METADATA": { + "localhost": { + "suppress-pending-fib": "enabled" + } + } +} +``` + +This configuration is backward compatible. Upgrade from a SONiC version that does not support this feature does not change the user's expected behavior as this flag is set to be disabled by default. + +This setting can be turned on or off at runtime. When this feature is turned off *fpmsyncd* sends an offload status reply immediately when a route is received from zebra. *fpmsyncd* listens to the changes +in *DEVICE_METADATA* and the new added field and when it gets enabled starts replying to zebra only after a reply from orchagent for all newly added prefixes. This design will make it possible to control +this feature at runtime. When user turns the feature off, newly received routes from zebra will get immediate answer back. + +In case user detects an issue with a feature (i.e routes aren't marked as offloaded in zebra and thus BGP does not announce them), it can be turned off. + + +##### 6.1.2. Minigraph + +*suppress-pending-fib* is enabled when generating CONFIG_DB out of minigraph for device role *LeafRouter*. + +#### 6.2. Manifest (if the feature is an Application Extension) + +This feature is implemented as part of existing BGP and SWSS containers, no manifest changes are required. + +#### 6.3. CLI/YANG model Enhancements + + +##### 6.3.2. DEVICE_METADATA + +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```suppress-pending-fib``` which can be set to ```"enable"``` or ```"disable"```. + +Snippet of ```sonic-device_metatadata.yang```: + +```yang +module sonic-device_metadata { + import sonic-app-state-logging { + prefix app-state-logging; + } + + revision 2022-09-15 { + description "Add BGP suppress FIB pending configuration knob"; + } + + container sonic-device_metadata { + container DEVICE_METADATA { + description "DEVICE_METADATA part of config_db.json"; + + container localhost{ + leaf suppress-pending-fib { + description "Enable suppress pending FIB feature. BGP will wait for route + FIB installation before announcing routes."; + type enumeration { + enum enabled; + enum disabled; + } + default disabled; + + must "((current() = 'disabled') or (current() = 'enabled' and ../synchronous_mode = 'enable'))" { + error-message "ASIC synchronous mode must be enabled in order to enable routes suppression"; + } + } + } + /* end of container localhost */ + } + /* end of container DEVICE_METADATA */ + } + /* end of top level container */ +} +/* end of module sonic-device_metadata +``` + +This knob can only be set to ```"enable"``` when syncrhonous SAI configuration mode is on. This constraint is guaranteed by the ```must``` expression for this leaf. + + +#### 6.4. CLI + +A new CLI command is added to control this feature: + +Command to enable feature: +``` +admin@sonic:~$ sudo config suppress-pending-fib enabled +``` + +Command to disable feature: +``` +admin@sonic:~$ sudo config suppress-pending-fib disabled +``` + +### 7. High-Level Design + +#### 7.1. FPM plugin migration + +FPM response channel is only supported with new ```dplane_fpm_nl``` zebra plugin and SONiC needs start using the new and supported plugin. Additional configuration is required in *zebra.conf.j2*: + +``` +! Fallback to nexthops information as part of RTM_NEWROUTE message +no fpm use-next-hop-groups + +! Configure FPM server address +fpm address 127.0.0.1 +``` + +Zebra shall be started with an ```-M dplane_fpm_nl``` to use the new plugin. + +Some patches that change FPM behavior to the desired SONiC behavior need to be ported to the new plugin: + +- New FPM plugin shall be patched the same way as old one to pass VRF if_index in ```rtmsg->rtm_table``` instead of table ID in FPM message. [SONiC FRR patch](https://github.com/sonic-net/sonic-buildimage/blob/master/src/sonic-frr/patch/0003-Use-vrf_id-for-vrf-not-tabled_id.patch).Since now SONiC will start communicating RTM_NEWROUTE messages back to zebra, similar change has to be made to accept VRF if_index instead of table ID. +- [[PATCH] ignore route from default table](https://github.com/yxieca/sonic-buildimage/blob/26c5bf81f0a8b2f5d6809048bb54a872403726c2/src/sonic-frr/patch/0009-ignore-route-from-default-table.patch) +- [0007-Add-support-of-bgp-l3vni-evpn.patch](https://github.com/sonic-net/sonic-buildimage/blob/a477dbb1751b741ac2762b4328da08b37eb400db/src/sonic-frr/patch/0007-Add-support-of-bgp-l3vni-evpn.patch) + +An additional set of patches is required to be ported to FRR 8.2 to support this feature: [Add ability for dplane_fpm_nl to receive RTM_NEWROUTE netlink messages that signal route handling in the asic](https://github.com/stepanblyschak/frr/pull/1). + +#### 7.2. BGP Docker container startup + +BGP configuration template ```bgpd.main.j2``` requires update to configure FRR accordingly. + +Configuration template snippet *bgpd.main.conf.j2*: + +```jinja2 +router bgp {{ DEVICE_METADATA['localhost']['bgp_asn'] }} + bgp suppress-fib-pending +``` + +Zebra shall be started with an option ```--asic-offload=notify_on_offload```. This option will make zebra wait for ```RTM_F_OFFLOAD``` before notifying about installation status to bgpd. The template *supervisor.conf.j2* file is updated: + +```jinja2 +[program:zebra] +command=/usr/lib/frr/zebra -A 127.0.0.1 -s 90000000 -M dplane_fpm_nl --asic-offload=notify_on_offload +``` + +Before the route is offloaded ```show ip route``` shows routes as queued: + +``` +admin@sonic:~$ show ip route +Codes: K - kernel route, C - connected, S - static, R - RIP, + O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, + T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, + f - OpenFabric, + > - selected route, * - FIB route, q - queued, r - rejected, b - backup + t - trapped, o - offload failure + +B>q 0.0.0.0/0 [20/0] via 10.0.0.1, PortChannel102, weight 1, 00:04:46 + q via 10.0.0.5, PortChannel105, weight 1, 00:04:46 + q via 10.0.0.9, PortChannel108, weight 1, 00:04:46 + q via 10.0.0.13, PortChannel1011, weight 1, 00:04:46 +``` + +Once zebra is notified about successfull route offload status the *offload* flag is set: + +``` +admin@sonic:~$ vtysh -c "show ip route 100.1.0.25/32 json" +{ + "100.1.0.25/32": [ + { + "prefix": "100.1.0.25/32", + "prefixLen": 32, + "protocol": "bgp", + "vrfId": 0, + "vrfName": "default", + "selected": true, + "destSelected": true, + "distance": 20, + "metric": 0, + "installed": true, + "offloaded": true, + "table": 254, + "internalStatus": 80, + "internalFlags": 264, + "internalNextHopNum": 1, + "internalNextHopActiveNum": 1, + "nexthopGroupId": 4890, + "installedNexthopGroupId": 4890, + "uptime": "00:22:20", + "nexthops": [ + { + "flags": 3, + "fib": true, + "ip": "10.0.0.49", + "afi": "ipv4", + "interfaceIndex": 788, + "interfaceName": "PortChannel1013", + "active": true, + "weight": 1 + } + ] + } + ] +} +``` + +#### 7.3. RouteOrch + + +#### ResponsePublisher in RouteOrch + +*RouteOrch* need to use existing *ResponsePublisher* API to be able to publish responses into *APPL_STATE_DB*. + +A snippet of *ResponsePublisher* API that is going to be used is given below. + +```c++ +void ResponsePublisher::publish(const std::string &table, const std::string &key, + const std::vector &intent_attrs, + const ReturnCode &status, + bool replace = false) override; +``` + +*RouteOrch* will not publish all fields and values to *APPL_STATE_DB* to reduce memory usage and reduce load on redis database. Instead, the key is published as well as the *protocol* field as this one is required for fpmsyncd to construct RTM_NEWROUTE message. + +Example usage in *RouteOrch*: + +```c++ +m_publisher.publish(APP_ROUTE_TABLE_NAME, kfvKey(kofvs), kfvFieldsValues(kofvs), ReturnCode(SAI_STATUS_SUCCESS), true); +``` + +Example data: +``` +127.0.0.1:6379[14]> hgetall "ROUTE_TABLE:193.5.96.0/25" +1) "protocol" +2) "bgp" +``` + +In order to delete an entry from *APPL_STATE_DB* it is required to pass an empty *intent_attrs* vector which will imply this is a *DEL* operation. + + +#### RouteOrch Route Set Flow + +```RouteOrch``` handles both local routes, pointing to local router interface, as well as next hop routes. On ```SET``` operation received in ```RouteOrch``` the ```RouteBulker``` is filled with create/set SAI operations depending on whether the route entry for the corresponding prefix was already created. In normal circumstances when next hop group is successfully created or found to be already existing the route entry is created or set with the corresponding next hop group SAI OID. The ```RouteOrch``` then calls ```RouteBulker::flush()``` that will form a bulk API call to SAIRedis. In ```RouteOrch::addRoutePost()``` the resulting object statuses are then collected and if some CREATE/SET operation failed orchagent calls ```abort()```, otherwise, on success, ```RouteOrch::addRoutePost()``` should publish the route programming status. Since there is no graceful handling of failed SAI operation the ```ResponsePublisher::publish()``` will be only called for a successfully programmed routes. In case a pre-condition for route programming is unmet, i.e unresolved neighbors in next hop group, the route entry programming is retried later and in such case there is no publishing of the result of the operation to ```APPL_STATE_DB``` until a pre-condition check is passed. + +*Temporary route*: + +A special handling exists in ```RouteOrch``` for a case when there can't be more next hop groups created. In this case ```RouteOrch``` creates, a so called, "temporary" route, using only 1 next hop from the group and using it's ```SAI_OBJECT_TYPE_NEXT_HOP``` OID as ```SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID```. In this case, ```RouteOrch::addRoutePost()``` has to publish the route entry status regardless as this route is considered as successfully programmed. A *temporary* route is kept in the ```m_toSync``` queue to be later on reprogrammed when sufficient resources are available for full next hop group creation. + +*Full mask prefix subnet routes*: + +When orchagent receives a /32 IPv4 or /128 IPv6 prefix which is an IP of a local interface the route programming is skipped as we already have this route programmed with IP2ME action as part of interface configuration. In this case a successful route programming response status must be sent to notify *fpmsyncd* that this prefix is offloaded. + + +##### Figure 3. RouteOrch Route Set Flow + +```mermaid +%%{ + init: { + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } + } +}%% +sequenceDiagram + participant APPL_DB + participant RouteOrch + participant RouteBulker + participant ResponsePublisher + participant APPL_STATE_DB + participant ASIC_DB + participant SYNCD + + APPL_DB --) RouteOrch: ROUTE_TABLE Notification + activate RouteOrch + + RouteOrch ->> RouteOrch: doTask() + + loop For each entry in m_toSync + RouteOrch ->> RouteBulker: RouteBulker.create_entry() / RouteBulker.set_entry_attribute() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + end + + Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to program
the route on next iteration
In this case no response is written
+ + loop For each route + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->create_route_entries() /
sai_route_api->set_route_entry_attribute() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->create_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + end + + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + + Note left of RouteOrch: On any route programming error
RouteOrch crashes orchagent.
No response is written
+ + loop For each entry status in RouteBulker + alt Is newly created route + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB: Set to ROUTE_TABLE + activate APPL_STATE_DB + APPL_STATE_DB -->> ResponsePublisher:
+ deactivate APPL_STATE_DB + Note right of RouteOrch: Publish status to APPL_STATE_DB if it is a new route + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher + end + end + + deactivate RouteOrch +``` + + + +#### RouteOrch Route delete Flow + + +##### Figure 4. RouteOrch Route Delete Flow + +Similar processing happens on ```DEL``` operation for a prefix from ```ROUTE_TABLE```. The removed prefixes are filled in ```RouteBulker``` which are then flushed forming a bulk remove API call to SAIRedis. ```RouteOrch::removeRoutePost()``` then collects the object statuses and if the status is not SAI_STATUS_SUCCESS orchagent ```abort()```s, otherwise it should publish the result via ```ResponsePublisher::publish()``` leaving ```intent_attrs``` vector empty which will remove the corresponding state key from ```APPL_STATE_DB```. + +```mermaid +%%{ + init: { + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } + } +}%% +sequenceDiagram + participant APPL_DB + participant RouteOrch + participant RouteBulker + participant ResponsePublisher + participant APPL_STATE_DB + participant ASIC_DB + participant SYNCD + + APPL_DB --) RouteOrch: ROUTE_TABLE Notification + activate RouteOrch + + RouteOrch ->> RouteOrch: doTask() + + loop For each entry in m_toSync + RouteOrch ->> RouteBulker: RouteBulker.remove_entry() + activate RouteBulker + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + end + + Note left of RouteOrch: On any pre-condition failed
RouteOrch is retrying to remove
the route on next iteration
In this case no response is written
+ + RouteOrch ->> RouteBulker: RouteBulker.flush() + activate RouteBulker + RouteBulker ->> ASIC_DB: sai_route_api->remove_route_entries() + activate ASIC_DB + ASIC_DB ->> SYNCD: sai_route_api->remove_route_entries() + activate SYNCD + SYNCD -->> ASIC_DB: sai_status_t *object_statuses + deactivate SYNCD + ASIC_DB -->> RouteBulker: sai_status_t *object_statuses + deactivate ASIC_DB + + RouteBulker -->> RouteOrch:
+ deactivate RouteBulker + + Note left of RouteOrch: On any route deletion error
RouteOrch crashes orchagent.
No response is written
+ + loop For each status in RouteBulker + RouteOrch ->> ResponsePublisher: ResponseBulisher.publish() + activate ResponsePublisher + ResponsePublisher ->> APPL_STATE_DB: Delete from ROUTE_TABLE + activate APPL_STATE_DB + APPL_STATE_DB -->> ResponsePublisher:
+ deactivate APPL_STATE_DB + Note right of RouteOrch: Publish the status and remove entry from APPL_STATE_DB + ResponsePublisher -->> RouteOrch:
+ deactivate ResponsePublisher + end + + deactivate RouteOrch +``` + +#### 7.4. FPMsyncd + + +> The dplane_fpm_nl has the ability to read route netlink messages +> from the underlying fpm implementation that can tell zebra +> whether or not the route has been Offloaded/Failed or Trapped. +> The end developer must send the data up the same socket that has +> been created to listen for FPM messages from Zebra. The data sent +> must have a Frame Header with Version set to 1, Message Type set to 1 +> and an appropriate message Length. The message data must contain +> a RTM_NEWROUTE netlink message that sends the prefix and nexthops +> associated with the route. Finally rtm_flags must contain +> RTM_F_OFFLOAD, RTM_F_TRAP and or RTM_F_OFFLOAD_FAILED to signify +> what has happened to the route in the ASIC. + +A new class *RouteFeedbackChannel* in fpmsyncd is introduced to manage orchagent responses and send the corresponding FPM message to zebra. + +Zebra requires to send RTM_NEWROUTE back with RTM_F_OFFLOAD flag set once route is programmed in HW. FpmSyncd create an RTM_NEWROUTE message based off VRF, prefix and nexthops that where actually installed which are part of publish notification. + +The offload route status is relevant only for BGP received routes. In order to send back RTM_NEWROUTE it is required to encode "protocol" (rtmsg->rtm_protocol field) in APPL_DB which will be sent back as part of response: + +``` +127.0.0.1:6379> hgetall "ROUTE_TABLE:193.5.96.0/25" +1) "nexthop" +2) "10.0.0.1,10.0.0.5,10.0.0.9,10.0.0.13" +3) "ifname" +4) "PortChannel102,PortChannel105,PortChannel108,PortChannel1011" +5) "protocol" +6) "bgp" +``` + +The protocol number is then set in RTM_NEWROUTE when preparing message to be sent to zebra. + + +##### Figure 5. FPMsyncd flow + +```mermaid +%%{ + init: { + "theme": "forest", + "sequence": { + "rightAngles": true, + "showSequenceNumbers": true + } + } +}%% +sequenceDiagram + participant zebra + participant RouteSync + participant RouteFeedbackChannel + participant APPL_DB + participant APPL_STATE_DB + + + zebra ->> RouteSync: RTM_NEWROUTE FPM message + activate RouteSync + RouteSync ->> APPL_DB: Write entry to ROUTE_TABLE + activate APPL_DB + APPL_DB ->> RouteSync:
+ deactivate APPL_DB + + alt Suppression is disabled + Note left of RouteSync: Reply back immidiatelly
when this feature is disabled + RouteSync ->> zebra: Send back RTM_NEWROUTE
with RTM_F_OFFLOAD set + zebra ->> RouteSync:
+ end + + deactivate RouteSync + + APPL_STATE_DB ->> RouteSync: ROUTE_TABLE publish event + alt Suppression is enabled + activate RouteSync + RouteSync ->> RouteFeedbackChannel: onRouteResponse() + activate RouteFeedbackChannel + alt Successful SET operation + Note right of RouteFeedbackChannel: Construct RTM_NEWROUTE based of
response field values
and set RTM_F_OFFLOAD flag
+ RouteFeedbackChannel ->> zebra: Send RTM_NEWROUTE + activate zebra + zebra ->> RouteFeedbackChannel:
+ deactivate zebra + end + deactivate RouteFeedbackChannel + deactivate RouteSync + end +``` + +*FpmLink* class has to be extended with new API to send netlink message through FPM socket: + +```c++ +ssize_t FpmLink::send(nl_msg* msg) +``` + +Example pseudo-code snippet of RTM_NEWROUTE construction: + +```c++ +auto route = rtnl_route_alloc(); + +rtnl_route_set_dst(routeObject, dstAddr); +rtnl_route_set_protocol(routeObject, protocol); +rtnl_route_set_family(routeObject.get(), ipFamily); + +// in SONiC FPM uses vrf if_index instead of table +rtnl_route_set_table(routeObject.get(), vrfIfIndex); + +// Mark route as OFFLOAD +rtnl_route_set_flags(routeObject.get(), RTM_F_OFFLOAD); + +nl_msg* msg; +rtnl_route_build_add_request(routeObject, NLM_F_CREATE, &msg); + +fpm.send(msg); +``` + +#### 7.5. Response Channel Performance considerations + +Route programming performance is one of crucial characteristics of a network switch. It is desired to program a lot of route entries as quick as possible. SONiC has optimized route programming pipeline levaraging Redis Pipeline in ```ProducerStateTable``` as well as SAIRedis bulk APIs. Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Such an optimization gives around ~4-5x times faster processing for ```Publisher/Subscriber``` pattern using a simple python script as a test. + +The following table shows the results for publishing 1k messages with and without Redis Pipeline proving the need for pipeline support for ```ResponseChannel```: + + +##### 7.5.1. Table 1. Publishing 1k ROUTE_TABLE responses + +| Scenario | Time (sec) | Ratio | +| ---------------------- | ---------- | ----- | +| Without Redis Pipeline | 0.641 | 4.27 | +| With Redis Pipeline | 0.150 | 1 | + +Adding a feedback mechanism to the system introduces a delay as each of the component needs to wait for the reply from the lower layer counterpart in order to proceed. SONiC has already moved to synchronous SAI Redis pipeline a route programming performance degradation caused by it is leveled by the use of SAIRedis Bulk API. + +By introducing a response channel it is required to leverage Redis Pipeline, so that the route configuration producer using Redis Pipeline with ```ProducerStateTable``` also receives route programming status responses produced by pipelined ```NotificationProducer``` which is part of ```ResponsePublisher```. + +On the other side, ```fpmsyncd``` does not wait for each individual route status but rather performs an asynchronous processing. + +The following schema represents communication methods between components: + + +##### Figure 6. zebra/orchagent feedback channel +```mermaid +%%{ + init: { + "theme": "forest" + } +}%% +flowchart LR + zebra("
zebra

") -- FPM channel --> fpmsyncd("
fpmsyncd

") + fpmsyncd -- "ProducerStateTable (Pipelined)" --> RouteOrch("
RouteOrch

") + RouteOrch -. "ResponsePublisher (Pipelined)" -.-> fpmsyncd + fpmsyncd -. FPM channel -.-> zebra + RouteOrch -- "SAIRedis Bulk API" --> syncd("
syncd

") + syncd -. "SAIRedis Bulk Reply" -.-> RouteOrch +``` + +A ```ResponsePublisher``` must have a constructor that accepts a ```RedisPipeline``` and a flag ```buffered``` to make it use the pipelining. The constructor is similar to one in use with ```ProducerStateTable```: + +```c++ +ResponsePublisher::ResponsePublisher(bool buffered = false); +``` + +And an API to set buffered mode: + +```c++ +void ResponsePublisher::setBuffered(bool buffered); +``` + +The ```OrchDaemon``` has to flush the ```RedisPipeline``` in ```OrchDaemon::flush``` when there are no pending tasks to perform. + +#### 7.6. Consistency monitoring script and mitigation action + +Routing is a crucial part of a network switch. This feature adds additional flows in existing route processing pipeline and so along the way there might be unexpected failures leading to routes being not marked as offloaded in zebra - missed notification, race conditions leading to in-consistency, etc. It is required to monitor the consistency of routes state periodically and mitigate problems. + +At the moment *route_check.py* verifies routes between APPL_DB & ASIC_DB and makes sure they are in sync. In addition to that it is required to extend this script to check for consistency APPL_DB, APPL_STATE_DB and zebra by calling ```show ip route json``` and verify every route installed in ASIC_DB is marked as offloaded. The script will retry the check for 3 times every 15 sec and if zebra FIB is not in sync with ASIC_DB a mitigation action is performed. The mitigation action publishes required notifications to trigger fpmsyncd flows to send RTM_NEWROUTE message to zebra and log alerts in the syslog. + +This way the problem is mitigated automatically and a network operator is notified via syslog alert about a problem. + +### 8. SAI API + +No new SAI API or changes to SAI design and behavior needed for this functionality. + +### 9. Warmboot and Fastboot Design Impact + +#### 9.1. Warm Reboot + +With BGP Graceful Restart, peers are keeping advertised routes in the FIB while the switch restarts. +When fpmsyncd is about to set *reconciled* state it must send RTM_NEWROUTE for all prefixes that were restored previously from APPL_DB since they are already in the ASIC. Once all existing routes are pushed to zebra a comparison logic will finish the warm reboot process and fpmsyncd will continue in a normal operational mode. + +#### 9.2. Fast Reboot + +During fast reboot BGP session is closed by SONiC device without the notification. BGP session is preserved in graceful restart mode. BGP routes on the peer are still active, because nexthop interfaces are up. Once interfaces go down, received BGP routes on the peer are removed from the routing table. Nothing is sent to SONiC device since then. After interfaces go up and BGP sessions re-establish the peer's BGP re-learns advertised routes. + +Due to additional response publishing in orchagent there might be a slight delay in fast reboot reconfiguration. + +### 10. Restrictions/Limitations + +- MPLS, VNET routes are out of scope of this document +- Directly connected and static routes are announced by BGP regardless of their offload status + +### 11. Testing Requirements/Design + +Regression test, VS test and unit testing are required for this functionality. + +Fast/warm reboot regression test suite needs to be ran and verified no degradation introduced by the feature. + +#### 11.1. Unit Test cases + +- FPMSyncd: test case regular route + 1. Mock FPM interface + 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 nexthop 2.2.2.2, 3.3.3.3 + 4. Verify a correct FPM message is being sent to zebra through FPM interface mock + 5. Verify RTM_F_OFFLOAD set in route message sent to zebra +- FPMSyncd: test case VRF route + 1. Mock FPM interface + 2. Call RouteSync::onRouteResponse() with route 1.1.1.0/24 in VRF nexthop 2.2.2.2, 3.3.3.3 + 3. Verify a correct FPM message is being sent to zebra through FPM interface mock + 4. Verify RTM_F_OFFLOAD set in route message sent to zebra +- FPMSyncd: send RTM_DELROUTE + 1. Mock FPM interface + 2. Call RouteSync::onRouteResponse() with delete response message containing route 1.1.1.0/24 + 4. Verify no message is sent to zebra + +- RouteOrch: program ASIC route + 1. Create FVS and call RouteOrch::doTask() + 2. Verify route is created in SAI + 3. Verify the content of APPL_STATE_DB +- RouteOrch: delete ASIC route + 1. Create delete entry and call RouteOrch::doTask() + 2. Verify route is delete in SAI + 3. Verify the content of APPL_STATE_DB + +#### 11.2. VS Test cases + +Code coverage is satisfied by UT and the E2E flow is not possible to test with current VS infrastructure, so no VS tests are planned. + +#### 11.3. System Test cases + + +##### Figure 7. T1 test topology + +

+Figure 7. T1 test topology +

+ +In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: + +- **Functional Test** + 1. Enable ```suppress-pending-fib``` + 2. Suspend orchagent process to simulate a delay with ```kill --SIGSTOP $(pidof orchagent)``` + 3. Announces prefix *1.1.1.0/24* (*1::/64*) to DUT through *exabgp* from T0 Arista VM + 4. Make sure announced BGP route is queued in ```show ip route``` output + 5. Verify the route is not announced to Arista T2 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer + 6. Send traffic matching the prefix and verify packets are not forwarded to T0 Arista VM + 7. Restore orchagent process: ```kill --SIGCONT $(pidof orchagent)``` + 8. Make sure route is programmed in ASIC_DB + 9. Make sure announced BGP route is FIB synced in ```show ip route``` output + 10. Verify the route is announced to Arista T1 peer by executing ```show ip bgp neighbor A.B.C.D received-routes``` on the peer + 11. Send traffic matching the prefix and verify packets are forwarded to T0 Arista VM +- **Performance Test** + 1. Announce 10K routes to DUT + 2. Send traffic towards these prefixes from AUX + 3. Measure time from announce start till all packets get received + 4. The test is parametrized for suppression state for comparison + 5. The test passes if the time is under the predefined threshold +- **Stress Test** + 1. Enable ```suppress-pending-fib``` + 2. In a loop for 100 times: + 1. Announce 10 prefix to DUT through *exabgp* from T0 Arista VM + 2. Withdraw 10 prefix to DUT through *exabgp* from T0 Arista VM + 3. Once reached the required number of cycles the loop breaks after first step + 4. Consistency check is applied, there are no withdrawn routes and announced routes were successfully installed and correctly marked as offloaded in zebra (In case a race condition happens or a notification is missed for some reason this test will try to catch it) + +### 12. Open/Action items - if any From 830f95366631cebd9685fa48dd586faafab2e831 Mon Sep 17 00:00:00 2001 From: Stepan Blyschak Date: Fri, 25 Nov 2022 15:13:24 +0200 Subject: [PATCH 42/43] rename flag to suppress-fib-pending Signed-off-by: Stepan Blyschak --- doc/BGP/BGP-supress-fib-pending.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index dddb4b9169..6af24af8db 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -138,7 +138,7 @@ Configuration schema in ABNF format: ```abnf ; DEVICE_METADATA table key = DEVICE_METADATA|localhost ; Device metadata configuration table -suppress-pending-fib = "enabled"/"disabled" ; Globally enable/disable routes announcement suppression feature, +suppress-fib-pending = "enabled"/"disabled" ; Globally enable/disable routes announcement suppression feature, ; by default this flag is disabled ``` @@ -148,7 +148,7 @@ Sample of CONFIG DB snippet given below: { "DEVICE_METADATA": { "localhost": { - "suppress-pending-fib": "enabled" + "suppress-fib-pending": "enabled" } } } @@ -165,7 +165,7 @@ In case user detects an issue with a feature (i.e routes aren't marked as offloa ##### 6.1.2. Minigraph -*suppress-pending-fib* is enabled when generating CONFIG_DB out of minigraph for device role *LeafRouter*. +*suppress-fib-pending* is enabled when generating CONFIG_DB out of minigraph for device role *LeafRouter*. #### 6.2. Manifest (if the feature is an Application Extension) @@ -176,7 +176,7 @@ This feature is implemented as part of existing BGP and SWSS containers, no mani ##### 6.3.2. DEVICE_METADATA -A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```suppress-pending-fib``` which can be set to ```"enable"``` or ```"disable"```. +A new leaf is added to ```sonic-device_metadata/sonic-device_metadata/DEVICE_METADATA/localhost``` called ```suppress-fib-pending``` which can be set to ```"enable"``` or ```"disable"```. Snippet of ```sonic-device_metatadata.yang```: @@ -195,7 +195,7 @@ module sonic-device_metadata { description "DEVICE_METADATA part of config_db.json"; container localhost{ - leaf suppress-pending-fib { + leaf suppress-fib-pending { description "Enable suppress pending FIB feature. BGP will wait for route FIB installation before announcing routes."; type enumeration { @@ -227,12 +227,12 @@ A new CLI command is added to control this feature: Command to enable feature: ``` -admin@sonic:~$ sudo config suppress-pending-fib enabled +admin@sonic:~$ sudo config suppress-fib-pending enabled ``` Command to disable feature: ``` -admin@sonic:~$ sudo config suppress-pending-fib disabled +admin@sonic:~$ sudo config suppress-fib-pending disabled ``` ### 7. High-Level Design @@ -771,7 +771,7 @@ Code coverage is satisfied by UT and the E2E flow is not possible to test with c In order to test this feature end to end it is required to simulate a delay in ASIC route programming. The proposed test: - **Functional Test** - 1. Enable ```suppress-pending-fib``` + 1. Enable ```suppress-fib-pending``` 2. Suspend orchagent process to simulate a delay with ```kill --SIGSTOP $(pidof orchagent)``` 3. Announces prefix *1.1.1.0/24* (*1::/64*) to DUT through *exabgp* from T0 Arista VM 4. Make sure announced BGP route is queued in ```show ip route``` output @@ -789,7 +789,7 @@ In order to test this feature end to end it is required to simulate a delay in A 4. The test is parametrized for suppression state for comparison 5. The test passes if the time is under the predefined threshold - **Stress Test** - 1. Enable ```suppress-pending-fib``` + 1. Enable ```suppress-fib-pending``` 2. In a loop for 100 times: 1. Announce 10 prefix to DUT through *exabgp* from T0 Arista VM 2. Withdraw 10 prefix to DUT through *exabgp* from T0 Arista VM From 3ae161c4efa4ccab8443d90de3adfd4e7bfc9701 Mon Sep 17 00:00:00 2001 From: Stepan Blyschak Date: Fri, 25 Nov 2022 15:15:26 +0200 Subject: [PATCH 43/43] add performance section Signed-off-by: Stepan Blyschak --- doc/BGP/BGP-supress-fib-pending.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/BGP/BGP-supress-fib-pending.md b/doc/BGP/BGP-supress-fib-pending.md index 6af24af8db..b4ddf3dd23 100644 --- a/doc/BGP/BGP-supress-fib-pending.md +++ b/doc/BGP/BGP-supress-fib-pending.md @@ -29,6 +29,7 @@ - [11.1. Unit Test cases](#111-unit-test-cases) - [11.2. VS Test cases](#112-vs-test-cases) - [11.3. System Test cases](#113-system-test-cases) + - [11.4. Performance measurements](#114-performance-measurements) - [12. Open/Action items - if any](#12-openaction-items---if-any) @@ -796,4 +797,8 @@ In order to test this feature end to end it is required to simulate a delay in A 3. Once reached the required number of cycles the loop breaks after first step 4. Consistency check is applied, there are no withdrawn routes and announced routes were successfully installed and correctly marked as offloaded in zebra (In case a race condition happens or a notification is missed for some reason this test will try to catch it) +#### 11.4. Performance measurements + +TBD + ### 12. Open/Action items - if any