-
Notifications
You must be signed in to change notification settings - Fork 39
Static Routing
- Features by Version
- Basic Settings
- Router Interfaces
- Nexthop Routes
- Neighbours
- ECMP Routes
- IPv6 Source-specific Routing
- Disabling ICMP Redirects
- Abort Mechanism
- Route Offload Notifications
- Recommended Sysctl Configuration
- Further Resources
| Kernel Version | |
|---|---|
| 4.8 | IPv4 routing |
| 4.11 |
net.ipv4.conf.default.ignore_routes_with_linkdown support |
| 4.14 | IPv6 routing, nexthop offload indication |
| 5.2 | neighbour offload indication |
| 5.11 | Nexthop objects support |
| 5.12 | Route offload notifications |
In order for routing to work on a Linux system, forwarding must be enabled. To check if forwarding is enabled, run:
$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
$ sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 0
In this case, IPv4/IPv6 forwarding is disabled. To enable it, run:
$ sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
$ sysctl -w net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.forwarding = 1
To enable it permanently across reboots, run:
$ echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/forward.conf
$ echo "net.ipv6.conf.all.forwarding = 1" >> /etc/sysctl.d/forward.conf
In a similar way, the other IP sysctls can be adjusted. For example,
the maximum number of IPv4 neighbour entries can be adjusted via
net.ipv4.neigh.default.gc_thresh3.
Whenever an IP address is assigned to a port netdevice or one of its uppers
(e.g. bridge, team, VLAN), a router interface is automatically created in the
hardware. In the following example, two router interfaces are created, one for
sw1p1 and one for sw1p2:
$ ip addr add 192.168.0.1/24 dev sw1p1
$ ip link set dev sw1p1 up
$ ip addr add 192.168.1.1/24 dev sw1p2
$ ip link set dev sw1p2 up
For each address and its broadcast and network addresses, traps are inserted into the hardware which cause the appropriate packets to be delivered to the kernel.
Note: MAC addresses of all the router interfaces must have the same 38 MSBs.
By default, the kernel flushes all the IPv6 addresses upon interface down:
$ ip -6 address show dev sw1p1
28: sw1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2001:db8::1/32 scope global
valid_lft forever preferred_lft forever
inet6 fe80::e61d:2dff:fe45:a9f1/64 scope link
valid_lft forever preferred_lft forever
$ ip link set dev sw1p1 down
$ ip -6 address show dev sw1p1
To be consistent with IPv4 and keep static global addresses with no expiration time upon interface down, run:
$ sysctl -w net.ipv6.conf.sw1p1.keep_addr_on_down=1
$ ip -6 address show dev sw1p1
inet6 2001:db8::1/32 scope global
valid_lft forever preferred_lft forever
inet6 fe80::e61d:2dff:fe45:a9f1/64 scope link
valid_lft forever preferred_lft forever
$ ip link set dev sw1p1 down
$ ip -6 address show dev sw1p1
28: sw1p1: <BROADCAST,MULTICAST> mtu 1500 state DOWN qlen 1000
inet6 2001:db8::1/32 scope global tentative
valid_lft forever preferred_lft forever
Note that the global address 2001:db8::1/32 is still configured on the
interface, while the link-local address fe80::e61d:2dff:fe45:a9f1/64 was
flushed.
To make this option the default for all the netdevices on the system upon boot
set net.ipv6.conf.all.keep_addr_on_down=1 in the sysctl configuration
files.
As previously stated, it is possible to create router interfaces on top of bridge netdevs by assigning them an IP address. In the case of the VLAN-aware bridge, a router interface can be created for each of its upper VLAN devices.
To create a router interface for the bridge netdev itself, run:
$ ip link add name br0 type bridge vlan_filtering 1
...
$ ip addr add 192.168.0.1/24 dev br0
And for one of its upper VLAN devices, run:
$ ip link add link br0 name br0.10 type vlan id 10
$ bridge vlan add dev br0 vid 10 self
$ ip addr add 192.168.1.1/24 dev br0.10
Once the router interface is created, it is possible to add routes:
$ ip route add 192.168.2.0/24 via 192.168.0.2 dev sw1p1
$ ip route add 192.168.3.0/24 via 192.168.1.2 dev sw1p2
To list the routes, run:
$ ip route
192.168.0.0/24 dev sw1p1 proto kernel scope link src 192.168.0.1 offload
192.168.1.0/24 dev sw1p2 proto kernel scope link src 192.168.1.1 offload
192.168.2.0/24 via 192.168.0.2 dev sw1p1 offload
192.168.3.0/24 via 192.168.1.2 dev sw1p2 offload
The offload flag indicates that the route is offloaded to hardware.
A neighbour entry is created for each nexthop. To list the neighbour entries, run:
$ ip neigh
192.168.0.2 dev sw1p1 INCOMPLETE
192.168.1.2 dev sw1p2 INCOMPLETE
After neighbour discovery takes place, the output changes:
$ ip neigh
192.168.0.2 dev sw1p1 52:54:00:aa:bb:01 REACHABLE
192.168.1.2 dev sw1p2 52:54:00:aa:bb:02 REACHABLE
To add routes with multiple nexthops, run:
$ ip route add 192.168.5.0/24 nexthop via 192.168.0.2 dev sw1p1 weight 1 nexthop via 192.168.1.1 dev sw1p2 weight 1
Unlike IPv4, the kernel allows one to add and remove individual nexthops without the need to delete the entire ECMP route and re-add it with a modified nexthop configuration.
To add an ECMP route, run:
$ ip -6 route add 2001:db81::/32 \
nexthop via fe80::e61d:2dff:fea5:f341 dev sw1p1 \
nexthop via fe80::e61d:2dff:fea5:f365 dev sw1p2
$ ip -6 route show 2001:db81::/32
2001:db81::/32 metric 1024
nexthop via fe80::e61d:2dff:fea5:f341 dev sw1p1 weight 1 offload
nexthop via fe80::e61d:2dff:fea5:f365 dev sw1p2 weight 1 offload
To delete the first nexthop, run:
$ ip -6 route del 2001:db81::/32 nexthop via fe80::e61d:2dff:fea5:f341 dev sw1p1
$ ip -6 route show 2001:db81::/32
2001:db81::/32 via fe80::e61d:2dff:fea5:f365 dev sw1p2 metric 1024 offload pref medium
Alternatively, another nexthop can be added to the route.
By default, when the carrier of a netdevice goes down, the routing
subsystem does not invalidate the nexthops using it and therefore
continues to try and forward packets through them. Such nexthops
are marked using the linkdown flag. For example:
$ ip route show 192.168.100.0/24
192.168.100.0/24
nexthop via 192.168.0.1 dev sw1p17 weight 1 offload linkdown
nexthop via 192.168.1.1 dev sw1p18 weight 1 offload
It is possible to make the kernel exclude such nexthops from its ECMP groups by setting the following sysctl:
$ sysctl -w net.ipv4.conf.sw1p17.ignore_routes_with_linkdown=1
With this sysctl set, when the carrier of sw1p17 goes down the kernel
starts forwarding packets via the nexthop using sw1p18 as its nexthop
device. It also marks the nexthop as dead for various listeners in
the user space:
$ ip route show 192.168.100.0/24
192.168.100.0/24
nexthop via 192.168.0.1 dev sw1p17 weight 1 dead linkdown
nexthop via 192.168.1.1 dev sw1p18 weight 1 offload
To make this option the default for all the netdevices on the system
upon boot set net.ipv4.conf.default.ignore_routes_with_linkdown=1 in
the sysctl configuration files.
Note: The mlxsw driver currently does not support this functionality in IPv6.
When forwarding packets, the device performs the multi-path hash in accordance with the kernel's policy.
The packet fields used for the multi-path hash are controlled by the
net.ipv{4,6}.fib_multipath_hash_policy sysctls. By default, it is set to
0, which means only the source and destination IP addresses are used.
In IPv6, the flow label is also used.
If the sysctl is set to 1, a 5-tuple is used: The source and
destination IP addresses, the source and destination ports, and the IP
protocol.
Note: When the sysctl is set to 0, the kernel performs the
multi-path hash for ICMP error packets according to the inner IP
addresses. Currently, this is not supported by the device.
Note: Layer 4 fields are not considered for fragmented packets.
Traditionally, when routes were programmed to the kernel they contained both the prefix information and the nexthop information. As seen here, this results in sub-optimal insertion rates and in some cases, increased memory consumption.
Since kernel 5.3, the kernel supports nexthops as separate objects. This allows user space to program nexthops to the kernel and assign them a numeric identifier. Later, user space can program routes and associate them with nexthop objects by specifying their numeric identifier.
In the following example, a nexthop group is added with two nexthops:
# ip link set dev swp3 up
# ip link set dev swp7 up
# ip address add 192.0.2.2/28 dev swp3
# ip address add 192.0.2.18/28 dev swp7
# ip nexthop replace id 1 via 192.0.2.1 dev swp3
# ip nexthop replace id 2 via 192.0.2.17 dev swp7
# ip nexthop replace id 10 group 1/2
# ip nexthop
id 1 via 192.0.2.1 dev swp3 scope link offload
id 2 via 192.0.2.17 dev swp7 scope link offload
id 10 group 1/2 offload
A route using the nexthop group is then added:
# ip route replace 198.51.100.0/24 nhid 10
# sysctl -qw net.ipv4.nexthop_compat_mode=0
# ip route show 198.51.100.0/24
198.51.100.0/24 nhid 10 rt_offload
# sysctl -qw net.ipv4.nexthop_compat_mode=1
# ip route show 198.51.100.0/24
198.51.100.0/24 nhid 10 rt_offload
nexthop via 192.0.2.1 dev swp3 weight 1
nexthop via 192.0.2.17 dev swp7 weight 1
In the above example, the output of the ip route command changes based
on the value of the net.ipv4.nexthop_compat_mode sysctl option. It is
described as follows in the kernel documentation:
nexthop_compat_mode - BOOLEAN
New nexthop API provides a means for managing nexthops independent of
prefixes. Backwards compatibilty with old route format is enabled by
default which means route dumps and notifications contain the new
nexthop attribute but also the full, expanded nexthop definition.
Further, updates or deletes of a nexthop configuration generate route
notifications for each fib entry using the nexthop. Once a system
understands the new API, this sysctl can be disabled to achieve full
performance benefits of the new API by disabling the nexthop expansion
and extraneous notifications.
Default: true (backward compat mode)
Nexthop objects also support the following unique features which are not supported by the legacy API:
By changing a nexthop to a blackhole nexthop, all the routes using the nexthop will start to silently drop packets. Example:
# ip nexthop replace id 3 blackhole
# ip nexthop show id 3
id 3 blackhole offload
The offload flag indicates that packets are dropped by the ASIC. To
get visibility into packets dropped by such nexthops, enable the
blackhole_nexthop trap:
# devlink trap set pci/0000:01:00.0 trap blackhole_nexthop action trap
Dropped packets can then be read using utilities such as DropWatch as explained here.
IPv4 routes can be added when pointing to IPv6 nexthops. For example:
# ip nexthop replace id 4 via fe80::1 dev swp3
# ip nexthop replace id 5 via fe80::2 dev swp7
# ip nexthop replace id 20 group 4/5
# ip nexthop
id 1 via 192.0.2.1 dev swp3 scope link offload
id 2 via 192.0.2.17 dev swp7 scope link offload
id 3 blackhole offload
id 4 via fe80::1 dev swp3 scope link offload
id 5 via fe80::2 dev swp7 scope link offload
id 10 group 1/2 offload
id 20 group 4/5 offload
# ip route replace 198.51.100.0/24 nhid 20
# ip route show 198.51.100.0/24
198.51.100.0/24 nhid 20 rt_offload
nexthop via inet6 fe80::1 dev swp3 weight 1
nexthop via inet6 fe80::2 dev swp7 weight 1
With the legacy API, it was not possible to perform multi-pathing between several device-only IPv6 nexthops. This is made possible with the nexthop API, which enables multi-pathing between several GRE tunnels. See this example from the kernel source tree for example usage.
Note: iproute2 version 5.11 is required in order to display nexthop
hardware flags (e.g., offload, trap)
Note: By default, since version 7.3, FRR will try to install routes using nexthop objects
The kernel supports IPv6 source-specific routing, which allows packets to be forwarded according to the destination and source addresses. If a packet matches two routes matching the destination address without one being more specific than the other, then it is possible to use the route with the most specific source prefix to route the packet.
However, without resorting to ACLs, the ASIC performs routing solely based on the destination address. Therefore, upon the insertion of source-specific routes the abort mechanism is invoked and forwarding is performed by the kernel.
Usually, when a packet is routed through the same interface from which it ingressed the router, an ICMP redirect packet is generated and sent to the transmitting host.
In some cases (e.g., one-armed router) it is desirable to disable ICMP redirects, as routing a packet through the same interface is not indicative of sub-optimal routing tables. Disabling ICMP redirects can be either done in hosts (preventing them from processing such packets) or in routers (preventing them from generating such packets).
Hosts can be prevented from processing ICMP redirects by clearing the
following sysctls. To disable ICMP redirects processing on sw1p1:
$ sysctl -qw net.ipv4.conf.all.accept_redirects=0
$ sysctl -qw net.ipv4.conf.sw1p1.accept_redirects=0
$ sysctl -qw net.ipv6.conf.all.accept_redirects=0
$ sysctl -qw net.ipv6.conf.sw1p1.accept_redirects=0
IPv4 routers can be prevented from generating ICMP redirects by clearing
the following sysctl. To disable ICMP redirects generation on sw1p1:
$ sysctl -qw net.ipv4.conf.all.send_redirects=0
$ sysctl -qw net.ipv4.conf.sw1p1.send_redirects=0
There is no corresponding sysctl for IPv6, but the following iptables
rule can be used:
$ ip6tables -A OUTPUT -p icmpv6 --icmpv6-type redirect -j DROP
It is possible for the insertion of a prefix route into the hardware to fail.
The reason might be that maximal capacity is reached, missing feature, etc.
In that case, the abort mechanism is initiated in kernel which leads to the
removal of all routes from the hardware resulting in all the packets being
processed in the kernel.
Note: Currently, this process irreversible. One has to reboot the system to re-enable routes offloading to hardware.
Note: There is an ongoing discussion regarding fixing the abort
mechanism and make the behaviour more sane for the end user.
When routes are programmed to hardware they are marked with one of the following flags:
-
RTM_F_OFFLOADfor routes that offload traffic from the kernel -
RTM_F_TRAPfor routes that trap packets to the kernel for processing (e.g., host routes) -
RTM_F_OFFLOAD_FAILEDfor routes that failed to be installed to hardware
By default, user space does not receive RTM_NEWROUTE notifications
when these flags change. This behaviour is controlled by the
net.ipv4.fib_notify_on_flag_change and
net.ipv6.fib_notify_on_flag_change sysctls. Possible values:
0 - Do not emit notifications
1 - Emit notifications
2 - Emit notifications only for RTM_F_OFFLOAD_FAILED flag change
The default value is 0.
To monitor the notifications, use:
$ ip monitor route
The notifications can be used, for example, by a routing daemon to delay advertisement of a route until it was successfully installed in hardware.
This section contains a list of relevant sysctls and their recommended values when the system is used as a router given the Linux defaults might not be adequate for this use case. For the sake of keeping the list complete, there is an overlap with other sections that discuss various sysctls in more detail.
# Enable IPv4 and IPv6 forwarding.
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv6.conf.default.forwarding=1
# Keep IPv6 addresses on an interface when it goes down. This is
# consistent with IPv4.
net.ipv6.conf.all.keep_addr_on_down=1
net.ipv6.conf.default.keep_addr_on_down=1
# Prevent the kernel from routing packets via an interface whose link is
# down. This is not strictly necessary when a routing daemon is used as
# it will most likely evict such routes. In addition, when offloaded,
# such routes will not be considered anyway since the associated neighbour
# entries will be flushed upon the carrier going down, preventing the
# device from determining the destination MAC it should use.
net.ipv4.conf.all.ignore_routes_with_linkdown=1
net.ipv6.conf.all.ignore_routes_with_linkdown=1
net.ipv4.conf.default.ignore_routes_with_linkdown=1
net.ipv6.conf.default.ignore_routes_with_linkdown=1
# Use a standard 5-tuple to compute the multi-path hash.
net.ipv4.fib_multipath_hash_policy=1
net.ipv6.fib_multipath_hash_policy=1
# Generate an unsolicited neighbour advertisement when an interface goes
# down or its hardware address changes.
net.ipv6.conf.all.ndisc_notify=1
net.ipv6.conf.default.ndisc_notify=1
# Do not perform source validation when routing IPv4 packets. This is
# consistent with the hardware data path behavior. No configuration
# is necessary for IPv6.
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
# Do not update the SKB priority from "TOS" field in IP header after
# the packet is forwarded. This applies to both IPv4 and IPv6 packets
# which are forwarded by the device.
net.ipv4.ip_forward_update_priority=0
# Prevent the kernel from generating a netlink event for each deleted
# IPv6 route when an interface goes down. This is consistent with IPv4.
net.ipv6.route.skip_notify_on_dev_down=1
# Use neighbour information when choosing a nexthop in a multi-path
# route. Will prevent the kernel from routing the packets via a
# failed nexthop. This is consistent with the hardware behavior.
net.ipv4.fib_multipath_use_neigh=1
# Increase the maximum number of cached IPv6 routes. No configuration is
# necessary for IPv4.
net.ipv6.route.max_size=16384
# In case the number of non-permanent neighbours in the system exceeds
# this value for over 5 seconds, the garbage collector will kick in.
# Default is 512, but if the system has a larger number of interfaces or
# expected to communicate with a larger number of directly-connected
# neighbours, then it is recommended to increase this value.
net.ipv4.neigh.default.gc_thresh2=8192
net.ipv6.neigh.default.gc_thresh2=8192
# In case the number of non-permanent neighbours in the system exceeds
# this value, the garbage collector will kick in. Default is 1024, but
# if the system has a larger number of interfaces or expected to
# communicate with a larger number of directly-connected neighbours,
# then it is recommended to increase this value.
net.ipv4.neigh.default.gc_thresh3=16384
net.ipv6.neigh.default.gc_thresh3=16384
Please refer to Routing intro in order to get essential information about routing setup in Linux.
- man ip
- man sysctl.d
- man ip-nexthop
- "Improving Route Scalability: Nexthops as Separate Objects" by David Ahern
Installation
System Maintenance
Network Interface Configuration
- Switch Port Configuration
- Netdevice Statistics
- Persistent Configuration
- Quality of Service
- Queues Management
- How To Configure Lossless RoCE
- Port Mirroring
- ACLs
- OVS
- Resource Management
- Precision Time Protocol (PTP)
Layer 2
Network Virtualization
Layer 3
- Static Routing
- Virtual Routing and Forwarding (VRF)
- Tunneling
- Multicast Routing
- Virtual Router Redundancy Protocol (VRRP)
Debugging