Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

IP Fabric Underlay Network Design and Implementation

 

For an overview of the IP Fabric underlay components used in this design, see the IP Fabric Underlay Network section in Data Center Fabric Blueprint Architecture Components.

The IP underlay network building block is arranged in a Clos-based fabric topology. The underlay network uses EBGP as the routing protocol in place of a traditional IGP like OSPF. You can use other routing protocols in the underlay protocol in your data center; the usage of those routing protocols is beyond the scope of this document.

Aggregated Ethernet interfaces with MicroBFD are also used in this building block. MicroBFD improves fault detection in an aggregated Ethernet interface by running BFD on individual links of the aggregated Ethernet interface.

Figure 1 provides a high-level illustration of the Clos-based IP fabric underlay network.

Figure 1: IP Fabric Underlay Network Overview
IP
Fabric Underlay Network Overview

Configuring the Aggregated Ethernet Interfaces Connecting Spine Devices to Leaf Devices

In this design each spine device is interconnected to each leaf device using a single link or a two-member aggregated Ethernet interface. The decision to use a single link or an aggregated Ethernet interface largely depends on the needs of your network; see Data Center Fabric Reference Design Overview and Validated Topology for more information on interface requirements.

The majority of IP Fabric topologies do not use aggregated Ethernet interfaces to interconnect spine and leaf devices. You can skip this section if you are connecting your spine and leaf devices using single links.

Use the following instructions to configure the interfaces that interconnect spine and leaf devices as aggregated Ethernet interfaces with two member links. An IPv4 address is assigned to each aggregated Ethernet interface. LACP with a fast periodic interval is also enabled.

Figure 2 shows the spine device interfaces that are configured in this procedure:

Figure 2: Spine 1 Interfaces
Spine 1 Interfaces

Figure 3 shows the leaf device interfaces that are configured in this procedure:

Figure 3: Leaf 1 Interfaces
Leaf 1 Interfaces

To configure aggregated Ethernet interfaces with fast LACP:

  1. Set the maximum number of aggregated Ethernet interfaces permitted on the device.

    We recommend setting this number to the exact number of aggregated Ethernet interfaces on your device, including aggregated Ethernet interfaces that are not used for spine to leaf device connections.

    In this example, the aggregated Ethernet device count value is set at 10 for a leaf device and 100 for a spine device.

    Leaf Device:

    Spine Device:

  2. Create and name the aggregated Ethernet interfaces, and optionally assign a description to each interface.

    This step shows how to create three aggregated Ethernet interfaces on spine 1 and four aggregated ethernet interfaces on leaf 1.

    Repeat his procedure for every aggregated Ethernet interface connecting a spine device to a leaf device.

    Spine 1:

    Leaf 1:

  3. Assign interfaces to each aggregated Ethernet interface on your device.

    Spine 1:

    Leaf 1:

  4. Assign an IP address to each aggregated Ethernet interface on the device.

    Spine 1:

    Leaf 1:

  5. Enable fast LACP on every aggregated Ethernet interface on the device.

    LACP is enabled using the fast periodic interval, which configures LACP to send a packet every second.

    Spine 1:

    Leaf 1:

  6. After the configuration is committed, confirm that the aggregated Ethernet interfaces are enabled, that the physical links are up, and that packets are being transmitted if traffic has been sent.

    The output below provides this confirmation information for ae1 on Spine 1.

  7. Confirm the LACP receive state is Current and that the transmit state is Fast for each link in each aggregated Ethernet interface bundle.

    The output below provides LACP status for interface ae1.

  8. Repeat this procedure for every device in your topology.

    The guide presumes that spine and leaf devices are interconnected by two-member aggregated Ethernet interfaces in other sections. When you are configuring or monitoring a single link instead of an aggregated Ethernet link, substitute the physical interface name of the single link interface in place of the aggregated Ethernet interface name.

Configuring an IP Address for an Individual Link

This section covers the procedure to add an IP address to a single link interface connecting a spine or leaf device. The process for adding an IP address to an aggregated Ethernet interface is covered in Configuring the Aggregated Ethernet Interfaces Connecting Spine Devices to Leaf Devices.

To add an IP address to a single link interface:

  1. Assign an IP address to each interface.

    Spine 1 Example:

    Leaf 1 Example:

  2. After the interface configuration is committed, confirm that the interfaces are enabled, that the physical links are up, and that packets are being transmitted if traffic has been sent.

    The output below provides this confirmation information for et-0/0/64 on Spine 1.

Enabling EBGP as the Routing Protocol in the Underlay Network

In this design, EBGP is the routing protocol of the underlay network and each device in the IP fabric is assigned a unique 32-bit autonomous system number (ASN). The underlay routing configuration ensures that all devices in the underlay IP fabric are reliably reachable from one another. Reachability between VTEP across the underlay IP Fabric is also required to support overlay networking with VXLAN.

Figure 4 shows the EBGP configuration of the underlay network.

Figure 4: EBGP Underlay Network Overview
EBGP Underlay
Network Overview

To enable EBGP as the routing protocol for the underlay network on a device:

  1. Create and name the BGP peer group. EBGP is enabled as part of this step.

    The underlay EBGP group is named UNDERLAY in this design.

    Spine or Leaf Device:

  2. Configure the ASN for each device in the underlay.

    Recall that in this design, every device is assigned a unique ASN in the underlay network. The ASN for EBGP in the underlay network is configured at the BGP peer group level using the local-as statement because the system ASN setting is used for MP-IBGP signaling in the overlay network.

    The examples below show how to configure the ASN for EBGP for Spine 1 and Leaf 1.

    Spine 1:

    Leaf 1:

  3. Configure BGP peers by specifying the ASN of each BGP peer in the underlay network on each spine and leaf device.

    In this design, for a spine device, every leaf device is a BGP peer, and for a leaf device, every spine device is a BGP peer.

    The example below demonstrates how to configure the peer ASN in this design.

    Spine 1:

    Leaf 1:

  4. Set the BGP hold time. The BGP hold time is the length of time, in seconds, that a peer waits for a BGP message—typically a keepalive, update, or notification message—before closing a BGP connection.

    Shorter BGP hold time values guard against BGP sessions staying open for unnecessarily long times in scenarios where problems occur, such as keepalives not being sent. A longer BGP hold time ensures BGP sessions remain active even during problem periods.

    The BGP hold time is configured at 10 seconds for every device in this design.

    Spine or Leaf Device:

  5. Configure EBGP to signal the unicast address family for the underlay BGP peer group.

    Spine or Leaf Device:

  6. Configure an export routing policy that advertises the IP address of the loopback interface to EBGP peering devices.

    This export routing policy is used to make the IP address of the loopback interface reachable from all devices in the IP Fabric. Loopback IP address reachability is required to allow leaf and spine device peering using MP-IBGP in the overlay network. IBGP peering in the overlay network must be established to allow devices in the fabric to share EVPN routes. See Configuring IBGP for the Overlay.

    The route filter IP address in this step—192.168.1.10—is the loopback address of the leaf device.

    Leaf 1:

  7. After committing the configuration, enter the show bgp summary command on each device to confirm that the BGP state is established and that traffic paths are active.

    Issue the show bgp summary command on Spine 1 to verify EBGP status.

  8. Repeat this procedure for every spine and leaf device in your topology.

Enabling Load Balancing

ECMP load balancing allows traffic to be sent to the same destination over multiple equal cost paths. Load balancing must be enabled on all spine and leaf devices to ensure that traffic is sent over all available paths provided by the IP Fabric.

Traffic is load balanced per Layer 4 flow on Junos devices. The ECMP algorithm load balances each traffic flow over one of the multiple paths, and all traffic for that flow is transmitted using the selected link.

To enable ECMP-based load balancing on a device:

  1. Enable multipath with the multiple AS option in BGP on all devices in the IP Fabric.

    EBGP, by default, selects one best path for each prefix and installs that route in the forwarding table. When BGP multipath is enabled, all equal-cost paths to a given destination are installed into the forwarding table. The multiple-as option enables load balancing between EBGP neighbors in different autonomous systems.

    All Spine and Leaf Devices:

  2. Create a policy statement that enables per-packet load balancing.

    All Spine and Leaf Devices:

  3. Export the policy statement to the forwarding table.

    All Spine and Leaf Devices:

Configuring Micro Bidirectional Forwarding Detection on Member Links in Aggregated Ethernet Interfaces

BFD is a simple bidirectional fault detection protocol that verifies bidirectional connectivity between directly-connected devices by periodically and rapidly sending a simple hello packet over the link or links that interconnect the devices. BFD can detect and communicate link faults in sub-second time frames to allow the control-plane software to quickly switch to an alternate path.

MicroBFD allows BFD to run on individual member links in an aggregated Ethernet interface.

MicroBFD is supported on QFX10000 switch to QFX10000 switch connections only in this design.

To enable microBFD:

  1. Set the minimum interval for BFD.

    The minimum interval is the amount of time, in milliseconds, that increments before BFD sends a hello packet to a directly-connected neighbor. The minimum interval is also the amount of time that BFD waits to receive a response to the hello packet before directing traffic destined to that link to other member links in the aggregated Ethernet interface.

    If the minimum interval is set at 100, for instance, BFD sends a hello packet every 100 milliseconds and starts directing traffic to other member links in an aggregated Ethernet interface if the hello packet does not receive a response in 100 milliseconds

    Set the minimum interval to a lower value if you want to ensure traffic is quickly redirected after a link failure. Set the minimum interval to a higher value if you want to minimize the chance that an active link is not accidentally misidentified as a down link by BFD.

    The minimum interval is set to 100 milliseconds on all member links in this design.

    The output below shows how to configure the minimum interval to 100 milliseconds on aggregated Ethernet interface 1.

    Spine and Leaf Device:

  2. Configure the IP address used by MicroBFD over the aggregated Ethernet bundle. For bridged AE interfaces, this address should be the IP address used by MicroBFD over the aggregated Ethernet bundle.

    The output below show how to assign the loopback IP address to ae1 on the leaf device.

    Spine and Leaf Device:

  3. Specify the loopback IP address of the neighbor device on the other end of the aggregated Ethernet interface.

    The output below is from aggregated Ethernet interface 1 on leaf device 1, which is used to connect to spine device 1. Spine device 1 uses 192.168.0.1 as it’s loopback IP address.

    Spine and Leaf Device:

  4. Specify the minimum number of links that have to remain up before all links in the aggregated Ethernet interface stop sending and receiving traffic.

    Setting the minimum links to 1 ensures the aggregated Ethernet bundle always transmits traffic unless all links in the aggregated Ethernet interface are unable to forward or receive traffic. Consider setting the minimum links value to a higher number if you feel performance could be significantly degraded in your network if a number of member links in your aggregated Ethernet interface stopped forwarding traffic.

    All devices in the design set the minimum links value to 1.

    Spine and Leaf Device:

    Repeat the above steps for each aggregated Ethernet interface in your topology to enable microBFD.

  5. To verify BFD sessions after enabling microBFD, enter the show bfd session command:

    Leaf 10 Example:

IP Fabric Underlay Network — Release History

Table 1 provides a history of all of the features in this section and their support within this reference design.

Table 1: IP Fabric Underlay Network in the Cloud Data Center Reference Design Release History

Release

Description

17.3R1-S1

All features documented in this section are supported on all devices within the design running Junos OS Release 17.3R1-S1 or later with the following exceptions:

  • MicroBFD is supported on QFX10000 series switches only.