BGP Unnumbered IPv6 Underlay in an EVPN-VXLAN Data Center

Overview

Today many enterprises and hyper-scale data centers use BGP as the underlay routing protocol. Unlike traditional IGPs such as OSPF and IS-IS, BGP typically requires that you explicitly configure peering, autonomous system (AS) numbers, and routing policies to control route exchanges.

Many operators are still relatively inexperienced with IPv6. Using BGP unnumbered peering, which dynamically discovers IPv6 neighbors, reduces the burden of manually configuring an IPv6 underlay in an EVPN-VXLAN data center (DC) fabric. Junos OS builds on the baseline IPv6 functionality by supporting BGP group configuration. BGP group configuration provides support for dynamic peering parameters (such as allowed remote AS numbers) used to support an unnumbered IPv6 fabric.

You can configure an EVPN-VXLAN DC fabric manually, or by using the BGP unnumbered peering feature. This use case describes the complexity of manually configuring the fabric and why using the BGP unnumbered peering feature provides a much easier solution.

Consider a simple two-tier data center. This modest size fabric consists of four spine devices and 32 leaf devices. Each spine device has 32 links which attach to the leaf devices and each leaf has two fabric links, one for each spine device.

In this manual configuration, you first need to need to assign the IP addresses for the network. For this fabric you'll need to configure 4*32=128 IPv6 IP addresses. Each network requires two host address assignments.

Next, you configure the BGP peers and their associated AS numbers. For each end of every fabric link, you need one BGP peering session. In our example fabric, this calculation equates to a total of 4*32*2=256 BGP peer definitions, each of which requires a unique peering IP and remote AS number.

Manually defining 256 BGP peerings can be cumbersome and is also prone to error. In a complex fabric, a simple misconfiguration can be difficult to isolate. Let's say that the fabric supports 128 leaf devices. You must now configure 4*128=512 IP IPv6 networks. It is clear from the math that the complexity of manually provisioning a large fabric quickly becomes a burden. Also, for IPv4 fabrics, an often overlooked factor is the large number of IPv4 addresses the underlay consumes. In many networks, IPv4 addressing space is at a premium.

In contrast, BGP unnumbered peering requires no routable IP network assignments on underlay links. All BGP peering in the underlay uses only link-local IP addresses. Using link-local IP addresses means less configuration, less complexity, smaller routing tables, and IP address preservation.

Configure BGP Unnumbered EVPN Fabric

When configuring BGP unnumbered peering, you only need a minimum number of configuration statements. These configuration statements are used to quickly bootstrap an IPv6 based underlay. This underlay supports the EVPN-VXLAN overlay. Although the underlay uses native IPv6, it also supports IPv4 routes with IPv6 next-hops (RFC5549) . In other words, the underlay supports both IPv4 and IPv6 workloads and their related virtual networks in the EVPN-VXLAN overlay. A complete discussion of IPv6 stateless auto-configuration is beyond the scope of this document.

The following list highlights the key features of BGP unnumbered peering:

• Automatically configures stateless link-local IPv6 addresses on all IPv6 enabled interfaces.
• Supports a list of allowed AS numbers to simplify peering to remote fabric devices.
• Uses IPv6 Router Advertisements (RAs) that provides dynamic discovery of directly attached neighbors.
• Uses IPv6 neighbor discovery to resolve the neighbor's link-local IP to the corresponding MAC address to facilitate link level communications.
• The local end uses the discovered peer link-local and MAC addresses to send a BGP open message to directly attached neighbors. This open message contains the local peer's AS number. The remote peer matches this against its list of allowed AS numbers to decide if the session is allowed. Likewise, the local peer matches on the remote peer's AS number, as returned in the remote peer's open message.
• Provides you with a simple BGP policy that advertises all directly connected networks (at a minimum, the loopback address of each fabric device must be advertised).
• Uses the default EBGP policy to re-advertise the routes learned from other fabric devices.
• Uses the BGP AS path length to prevent loops and to provide optimum route selection for ECMP load balancing.
• Simplifies adding an EVPN-VXLAN overlay since the underlay provides loopback reachability.

Note:

BGP unnumbered peering only supports EBGP. Multihop EBGP and IBGP are not supported.

Topology

The following diagram shows a simple two spine and two leaf topology.

• The fabric underlay is pure IPv6, including the device loopbacks.

• The attached servers are IPv4-based, which demonstrates support for IPv4 over IPv6 (see RFC 5549). IPv6 workloads are also supported in the overlay. Assume these servers are bare-metal servers (BMS), and are preconfigured with the IP addressing that is shown. The VLAN membership is used to map these workloads into overlay virtual networks that are isolated by VXLAN encapsulation. The access interfaces are untagged for this example.

• You expect to have routed connectivity between the server workloads when you add an overlay. The overlay can be either CRB or ERB. The BGP unnumbered underlay supports either type of overlay.

Step-by-Step Configuration

1. Enable the inet6 family on all fabric interfaces. These fabric interfaces attach Leaf 1 to the spine devices. The inet6 family provides support for IPv6 stateless auto-configuration and neighbor discovery. To support IPv4 workloads you must also add the inet family.

2. Create the loopback interface lo0 and configure the IPv6 address. The loopback address is used to support BGP peering when you add the EVPN-VXLAN overlay.

3. Configure a policy that specifies the list of BGP AS numbers you want to allow for dynamic BGP peering.

4. Configure a per-packet load balancing policy. A load balancing policy allows multiple equal-cost next-hops to be installed in the forwarding table, which provides rapid fail-over to alternative equal-cost fabric hops in the event of a link failure.

5. Apply the per-packet load balancing policy to the forwarding table.

6. Configure a policy to advertise direct routes. Because link-local subnets are not exported, in this example, this policy advertises only the loopback address. You'll use this same loopback address later when you configure BGP peering in the overlay.

7. Configure the router ID (RID). Because this example is for a native IPv6 fabric, you must ensure the configuration includes an IPv4 formatted router ID. In most cases, the RID is automatically derived from the loopback address, but in this example, the loopback is IPv6 only.

8. Configure a router advertisement (RA) for each fabric interface. When the RA is enabled, the interface periodically sends RA messages. RA messages are used to discover the remote neighbor's link-local IP, which in turn kicks off neighbor discovery and the rest of the dynamic peering process.

9. Configure a BGP group to support unnumbered peering. We apply a policy to this group that exports direct routes. You must include all fabric interfaces in this group and you must enable the interfaces with peer-auto-discovery under the [protocols bgp group <group-name> dynamic-neighbor <neighbor-group-name>] hierarchy. This group is linked to the policy that defines the AS numbers that are allowed for dynamic peering.

   You enable load balancing over multiple paths and multiple AS numbers. Remember that in this fabric, each node uses a unique AS number. Multipath load balancing (ECMP) to multiple AS numbers enables fast fail-over by allowing the RIB to install multiple next hops that point to these different AS numbers. Adding support for extended IPv4 nexthops allows IPv4 route exchange over the IPv6 BGP sessions.

Results

Recall that all fabric devices have similar configurations. For brevity, only the configuration delta for Leaf 1 is shown.

Quick Configurations

To get you up and running quickly, we've provided you with quick configurations for each node in the topology. Copy and edit the following commands as needed for the specifics of your environment (you usually would not use the commands exactly as shown here). Paste the modified commands into the related fabric device's terminal window. You must be in configuration mode at the [edit] hierarchy:

The quick configuration for Leaf 1:

The quick configuration for Leaf 2:

The quick configuration for Spine 1:

The quick configuration for Spine 2:

Be sure to commit the configuration changes on all devices.

Validation

Use these commands and sample outputs to confirm the proper operation of your unnumbered underlay. The configuration and operation of all nodes are similar. Below we only show commands and output for Leaf 1. The same commands apply to all nodes, and similar output is expected for all devices.

Overall, the main verification task is to confirm all nodes have expected BGP sessions established using link-local addresses, and that the fabric devices are properly exchanging loopback routes. We take a structured bottom-up approach that confirms each aspect needed to successfully establish BGP sessions.

Confirm fabric interfaces are up and operational.

The output shows that the Leaf 1's fabric interfaces are operational. Also seen is the lack of an explicit IPv4 or IPv6 address. Only the inet and inet6 families are configured on the interface. As a result, only IPv6 link-local IPv6 addresses are present as part of IPv6 stateless address configuration. We also note the loopback address has the expected IPv6 address assignment.

The output of the show interfaces terse CLI command makes it easy to verify the state and configuration of all interfaces, including the loopback address.

Confirm all fabric interfaces are sending and receiving IPv6 router advertisements (RAs).

The output confirms that Leaf 1 is sending, and receiving RAs, to and from both spine devices. The RAs correctly report the sending end's link-local address.

Confirm the fabric devices have learned the MAC to link-local address bindings of all directly attached IPv6 neighbors using IPv6 ND.

The output confirms that Leaf 1 has successfully resolved the MAC to link-local address of both its neighbors, that is, the two spine devices.

Confirm that you can ping a neighbor using its link-local address.

The ping from Leaf 1 to Spine 1 succeeds. This result confirms IPv6 connectivity using link-local addresses.

Confirm that all fabric devices have established BGP peering sessions to the directly connected neighbors.

As expected, the output shows that the Leaf 1 device has two BGP sessions established. The display confirms that IPv6 routes are exchanged. We include more on the confirmation later. If desired, show details about a BGP neighbor:

Confirm that all nodes are advertising their loopback address while learning the loopback address of other nodes.

As expected, the output confirms that Leaf 1 has learned the loopback addresses of all other fabric devices. Note that for Leaf 2, it shows two equal costs paths, one path through Spine 1 and the other through Spine 2.

If needed, you can help isolate routing problems by displaying the specific routes advertised to, or received from, a given BGP neighbor. Note that as with a ping to a link-local address, you must qualify the peer's link-local IP with the corresponding egress interface:

Confirm that the fabric supports ECMP load balancing. Display details about the routes Leaf 1 use to reach Leaf 2. You expect two equal costs routes, one through each spine device. In the section below, you confirm ECMP in both the routing and forwarding tables (RIB and FIB).

The output confirms that both paths between Leaf 1 and Leaf 2 are actively used to forward traffic and are part of a multipath route.

Confirm that Leaf 1 and Leaf 2 have connectivity over the underlay.

The output confirms that Leaf 1 and Leaf 2 have underlay connectivity. The path shows that this flow went through Spine 1, as indicated by its loopback address as the first hop.

This underlay is now ready to support your choice of EVPN-VXLAN overlay. See Understanding EVPN with VXLAN Data Plane Encapsulation for details on available overlay options.