- Introduction
- Option 1 - Virtual WAN v1, IPsec API integration
- Option 2 - Customer managed VNet v1, UDR/Static Routing + NVA
- Option 3 - Customer managed VNet v2, BGP-over-IPsec with VPN Gateway + NVA
- Option 4 - Virtual WAN v2, "NVA in the Hub"
- Option 5 - Customer managed VNet v3, Azure Route Server + NVA
- Option 6 - Virtual WAN v3, BGP Endpoint + NVA
- Option 7 - Cloud Edge Colocation, Physical appliance
- Conclusion and options flow-chart
This article compliments a two part video series published on YoutTube. Part 1 here covers the strategy angle and "asking the right questions", whilst Part 2 here is the video couterpart to this article, focused on the technical integration patterns for SD-WAN integration in to Azure Virtual Networking
We will jump straight in to the technical scenarios/options. For preamble and context, please watch the videos linked above. NB. You can also click the section headers, to take you straight to video content for that particular topic.
⚠️ Check with your networking vendor for approved/reference designs before continuing
This design pattern has existed since 2018 when Azure Virtual WAN first went Generally Available. It utilises automation between the networking vendors SD-WAN controller and API's exposed by the Azure Virtual WAN Hub. These APIs offer programmatic automation for attachment of the SD-WAN to Cloud. Example, a new site comes online in Chicago, it is tagged as "cloud enabled" in the SD-WAN vendors software. IPsec tunnels are automatically built from the Branch CPE to Azure Virtual WAN.
| Design area | Details |
|---|---|
| Network Transport | IPsec only |
| Azure Networking category | Azure Virtual WAN |
| Azure/Cloud CPE device | Azure Virtual WAN VPN Gateway |
| Routing | Dynamic |
| Notes | 1000 sites, 2Gbps per site, 20Gbps aggregate 1 |
- Microsoft Azure Virtual WAN landing page for supported partners - https://docs.microsoft.com/en-us/azure/virtual-wan/virtual-wan-locations-partners#partners
- Example of vendor pattern utilising this topology, Meraki - https://documentation.meraki.com/MX/Deployment_Guides/Cisco_Meraki_MX_Branch_to_Azure_Virtual_WAN_Deployment_Guide
This design pattern has existed for nearly as long as Azure Virtual Networks have. The ability to insert Network Virtual Appliances (NVAs) is a common function leveraged by customers to insert 3rd party Firewalls in the middle of network flows between endpoints inside or to/from the Cloud. We are able to use this same pattern with SD-WAN virtual appliances.
| Design area | Details |
|---|---|
| Network Transport | SD-WAN |
| Azure Networking category | Virtual Network (customer managed) |
| Azure/Cloud CPE device | Vendor NVA |
| Routing | Static |
| Notes | Not only will you require static UDR on Spoke Virtual Networks to point SD-WAN bound traffic at your SD-WAN NVA, the SD-WAN appliance itself will have to internally originate static routes that represent your Azure address space. Consider your HA pattern, will you use Azure Load Balancer Active/Active or UDR Rewrite Active/Standby; this will largely be determined by any advanced stateful security features you may be using as part of your SD-WAN deployment. |
- Microsoft documentation for generic high availability patterns for NVAs - https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/dmz/nva-ha
This design pattern looked to enhance #2 with a workaround (trick!) related to Azure VPN Gateways. Historically, VPN Gateways were the only endpoint to which you could configure custom BGP peerings. By establishing IPsec tunnels between the NVA in the Cloud, and the Azure VPN Gateway in the Cloud (now possible using private IPs), we are able to dynamically exchange routes. I.e. the VPN Gateway was purely a bump in the wire, to take BGP information from your NVA, and use it's native fabric access to inject routes in to the Azure SDN.
| Design area | Details |
|---|---|
| Network Transport | SD-WAN |
| Azure Networking category | Virtual Network (customer managed) |
| Azure/Cloud CPE device | Vendor NVA |
| Routing | Dynamic |
| Notes | Check vendor supports eBGP multi-hop. Maximum 4 IPsec tunnels, offering approximately 1.25Gbps *4 = 5Gbps throughput aggregate. 2 |
This design pattern has existed since mid 2020. In addition to the APIs exposed within Azure Virtual WAN as part of pattern #1, additional, deeper, integrations were opened up for networking partners. In this model, after collaboratively working with Microsoft on the required integration, network vendors are able to place their native virtual appliances directly inside of the Virtual WAN Hub. These appliances then appear as any other Virtual WAN component; that is, entirely managed, with no need to worry about the actual resource, resilience, scaling, feeding and watering etc. In my opinion this offers the deepest levels of Azure integration, it is effectively a PaaS service for SD-WAN endpoint in the Cloud. This integration, also grants the NVA special access to the SDN in order to dynamically exchange routes as if it were a native service.
| Design area | Details |
|---|---|
| Network Transport | SD-WAN |
| Azure Networking category | Azure Virtual WAN |
| Azure/Cloud CPE device | Managed (PaaS) vendor NVA |
| Routing | Dynamic |
| Notes | Limited vendor support, see below link |
- Microsoft Azure Virtual WAN landing page for supported partners - https://docs.microsoft.com/en-us/azure/virtual-wan/virtual-wan-locations-partners#integrated-virtual-hub-nva-partners
Azure Route Server (ARS) went GA in September 2021. This pattern effectively supersedes pattern #3; now that we can talk BGP directly to ARS, there is no need to deal with the overhead/technical-hassle of running this routing protocol over IPsec. ARS acts as a route reflector, allowing the VNets (local and peered) to learn of your SD-WAN routes, and vice-versa. Note "route reflector", ARS does not sit in the data-path by design.
| Design area | Details |
|---|---|
| Network Transport | SD-WAN |
| Azure Networking category | Virtual Network (customer managed) |
| Azure/Cloud CPE device | vendor NVA |
| Routing | Dynamic |
| Notes | Pay attention to ARS limits, approach summarization with care, especially in migration scenarios 3. Check vendor supports eBGP multi-hop. |
- Microsoft Azure Route Server documentation - https://docs.microsoft.com/en-us/azure/route-server/overview#how-does-it-work
- Example of vendor pattern utilising this topology, Meraki - https://documentation.meraki.com/MX/Deployment_Guides/Cisco_Meraki_MX_Branch_to_Azure_Virtual_WAN_Deployment_Guide
If you are using Azure Virtual WAN and your favorite networking vendor is not available for pattern #4 then this might be the topology for you. Similar to Azure Route Server, BGP Endpoint for Virtual WAN exposes the ability to directly BGP peer. We are then able to place our NVA in an attached Virtual Network Spoke, and establish BGP Peering sessions with the hub over private IPs (over VNet peering).
| Design area | Details |
|---|---|
| Network Transport | SD-WAN |
| Azure Networking category | Azure Virtual WAN |
| Azure/Cloud CPE device | vendor NVA |
| Routing | Dynamic |
| Notes | - |
- Microsoft Azure Virtual WAN BGP endpoint doc page - https://docs.microsoft.com/en-us/azure/virtual-wan/create-bgp-peering-hub-portal
An alternative approach. Instead of stretching our SD-WAN solution all the way in to Azure, and in to our Virtual Networks that ride upon the Azure SDN, we may decide to only stretch our SD-WAN network as far as our cloud edge PoPs. This is for those customers who have racks located within collocation centres that are strategically placed in close proximity to major cloud vendors. In this model, the SD-WAN does it's job in getting your business network over the public Internet, and the "last cloud mile" is handled by traditional technologies, normally ExpressRoute. This is a great choice if you have either large (>10Gbps) throughput requirements, or extremely complicated multi-vrf networks, wherein the physical SD-WAN appliances can be sized and specified appropriately.
| Design area | Details |
|---|---|
| Network Transport | SD-WAN to Cloud Edge, ExpressRoute to Azure |
| Azure Networking category | n/a |
| Azure/Cloud CPE device | n/a |
| Routing | Dynamic |
| Notes | You can solve for the "cloud edge" box in a number of ways; a) Build it yourself, common with large Enterprise, b) Take a Virtual Edge from a company like Equinix or c) Take a Cloud Onramp service as part of your SD-WAN vendor offering. See links below. |
- Microsoft Azure Virtual WAN landing page for supported partners - https://docs.microsoft.com/en-us/azure/virtual-wan/virtual-wan-locations-partners#partners
- VMware (VeloCloud) managed SD-WAN Gateways - https://sase.vmware.com/products/component-gateways
The number of technical options may feel confusing or overwhelming, but I would keep things simple;
- If you are using customer-managed traditional hub/spoke networking, aim for pattern #5 based on Azure Route Server
- If you are using Azure Virtual WAN based networking, aim for pattern #4 "NVA in hub", otherwise pattern #6 with "BGP Endpoint".
The non tl;dr version of this is represented in the flow-chart below.
Footnotes
-
https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#virtual-wan-limits ↩
-
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways#benchmark ↩
-
Whilst a maximum of 1000 routes per NVA BGP peer can be advertised to Azure Route Server, note that if using the branch2branch feature of ARS, this limit effectively drops to 200 total routes across all BGP peers https://docs.microsoft.com/en-us/azure/route-server/route-server-faq#route-server-limits ↩