Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Edge-Routed Bridging Overlay Design and Implementation

 

A second overlay option for this reference design is the edge-routed bridging overlay, as shown in Figure 1.

Figure 1: Edge-Routed Bridging Overlay
Edge-Routed Bridging Overlay

The edge-routed bridging overlay performs routing at IRB interfaces located at the edge of the overlay (most often at the leaf devices). As a result, Ethernet bridging and IP routing happen as close to the end systems as possible, but still support Ethernet dependent applications at the end system level.

For a list of switches that we support as lean spine and leaf devices in an edge-routed bridging overlay, see the Data Center Fabric Reference Design Supported Hardware Summary.

Lean spine devices handle only IP traffic, which removes the need to extend the bridging overlay to the lean spine devices. With this limited role, you must configure only the IP fabric underlay and IBGP overlay on these devices.

On the leaf devices, you can configure an edge-routed bridging overlay using the default switch instance or using MAC-VRF instances. Some configuration steps that affect the Layer 2 configuration differ with MAC-VRF instances. The leaf device configuration includes the following steps:

  • Configure the default instance with the loopback interface as a VTEP source interface. Or if your configuration uses MAC-VRF instances, configure a MAC-VRF instance with the loopback interface as a VTEP source interface. In each MAC-VRF instance, you also configure a service type, a route distinguisher, and a route target.

  • Configure a leaf-to-end system aggregated Ethernet interface as a trunk to carry multiple VLANs. With MAC-VRF instances, you also include the interface in the MAC-VRF instance.

  • Establish LACP and ESI functionality.

  • Map VLANs to VNIs. For a MAC-VRF instance configuration, you configure the VLAN to VNI mappings in the MAC-VRF instance.

  • Configure proxy-macip-advertisement, virtual gateways, and static MAC addresses on the IRB interfaces.

  • Configure EVPN/VXLAN in the default instance or in the MAC-VRF instance.

  • Enable VRF routing instances and IP prefix route properties for EVPN Type 5.

For an overview of edge-routed bridging overlays, see the Edge-Routed Bridging Overlay section in Data Center Fabric Blueprint Architecture Components.

For more information about MAC-VRF instances and using them in an example customer use case with an edge-routed bridging overlay, see EVPN-VXLAN DC IP Fabric MAC-VRF L2 Services  .

The following sections show the steps of how to configure and verify the edge-routed bridging overlay:

Configuring an Edge-Routed Bridging Overlay on a Lean Spine Device

To enable the edge-routed bridging overlay on a lean spine device, perform the following:

Note

The following example shows the configuration for Spine 1, as shown in Figure 2.

Figure 2: Edge-Routed Bridging Overlay – Lean Spine Devices
Edge-Routed Bridging Overlay – Lean
Spine Devices
  1. Ensure the IP fabric underlay is in place. To see the steps required to configure an IP fabric on a spine device, see IP Fabric Underlay Network Design and Implementation.
  2. Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your spine device, see Configuring IBGP for the Overlay.

Verifying the Edge-Routed Bridging Overlay on a Lean Spine Device

To verify that IBGP is functional on a lean spine device, use the show bgp summary command as described in Configuring IBGP for the Overlay. In the output that displays, ensure that the state of the lean spine device and its peers is Establ (established).

Configuring an Edge-Routed Bridging Overlay on a Leaf Device

To enable the edge-routed bridging overlay on a leaf device, perform the following:

Note

The following example shows the configuration for Leaf 10, as shown in Figure 3.

Figure 3: Edge-Routed Bridging Overlay – Leaf Devices
Edge-Routed Bridging Overlay – Leaf
Devices
  1. Ensure the IP fabric underlay is in place. To see the steps required to configure an IP fabric on a leaf device, see IP Fabric Underlay Network Design and Implementation.
  2. Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your leaf device, see Configuring IBGP for the Overlay.
  3. Configure the loopback interface as a VTEP source interface.

    If your configuration uses the default instance, you use statements in the [edit switch-options] hierarchy, as follows:

    Leaf 10 (Default Instance):

    If your configuration uses MAC-VRF instances, define a routing instance of type mac-vrf, and configure these statements at that MAC-VRF routing instance hierarchy level. You also must configure a service type for the MAC-VRF instance. We use the vlan-aware service type here so you can associate multiple VLANs with the MAC-VRF instance. This setting is consistent with the alternative configuration that uses the default instance.

    Leaf 10 (MAC-VRF Instance):

  4. (MAC-VRF instances only) Enable shared tunnels on the device.

    A device can have problems with VTEP scaling when the configuration uses multiple MAC-VRF instances. As a result, to avoid this problem, we require that you enable the shared tunnels feature on the QFX5000 line of switches with a MAC-VRF instance configuration. When you configure the shared-tunnels option, the device minimizes the number of next-hop entries to reach remote VTEPs. The following statement globally enables shared VXLAN tunnels on the device:

    This statement is optional on the QFX10000 line of switches, which can handle higher VTEP scaling than QFX5000 switches.

    Note

    This setting requires you to reboot the device.

  5. Configure the leaf-to-end system aggregated Ethernet interface as a trunk carrying four VLANs. Include the appropriate ESI and LACP values for your topology.

    Leaf 10:

    Note

    When configuring ESI-LAGs on the QFX5000 line of switches that serve as leaf devices in an edge-routed bridging overlay, keep in mind that we currently support only the Enterprise style of interface configuration, which is shown in this step.

    If your configuration uses MAC-VRF instances, you must also add the configured aggregated Ethernet interface to the MAC-VRF instance:

  6. Configure the mapping of VLANs to VNIs and associate one IRB interface per VLAN.

    This step shows the VLAN to VNI mapping and IRB interface association either in the default instance or in a MAC-VRF instance configuration.

    Leaf 10 (Default Instance):

    Leaf 10 (MAC-VRF Instance):

    The only difference with a MAC-VRF instance configuration is that you configure these statements in the MAC-VRF instance at the [edit routing-instances mac-vrf-instance-name] hierarchy level.

  7. Configure the IRB interfaces for VNIs 50000 and 60000 with both IPv4 and IPv6 dual stack addresses for both the IRB IP address and virtual gateway IP address.

    There are two methods for configuring gateways for IRB interfaces:

    • Method 1: unique IRB IP Address with Virtual Gateway IP Address, which is shown in step 6.

    • Method 2: IRB with Anycast IP Address and MAC Address, which is shown in step 7.

    Leaf 10:

  8. Configure the IRB interfaces for VNIs 70000 and 80000 with a dual stack Anycast IP address.

    Leaf 10:

    For more information about IRB and virtual gateway IP address configuration, see the IRB Addressing Models in Bridging Overlays section in Data Center Fabric Blueprint Architecture Components.

  9. Enable the ping operation for IRB interfaces 500 and 600, which are configured in step 6.

  10. Configure the EVPN protocol with VXLAN encapsulation on the leaf device.

    This step shows how to configure either the default instance or a MAC-VRF instance.

    Leaf 10 (Default Instance):

    Leaf 10 (MAC-VRF Instance):

    The only difference with a MAC-VRF configuration is that you configure these statements in the MAC-VRF instance at the [edit routing-instances mac-vrf-instance-name] hierarchy.

  11. Configure a policy called EXPORT_HOST_ROUTES to match on and accept /32 and /128 host routes, direct routes, and static routes. You will use this policy in step 12.
  12. Configure the loopback interface with two logical interfaces. (You will assign one logical interface to each VRF routing instance in the next step).
  13. Configure two tenant VRF routing instances, one for VNIs 50000 and 60000 (VRF 3), and one for VNIs 70000 and 80000 (VRF 4). Assign one logical interface from the loopback to each routing instance so that the VXLAN gateway can resolve ARP requests. Configure IP prefix route properties for EVPN type-5 to advertise ARP routes to the spine devices. Set up dummy IPv4 and IPv6 static routes, which you can use to discard traffic that results from denial of service (DoS) attacks. Enable load balancing for Layer 3 VPNs.

    Leaf 10:

  14. If you are configuring a QFX5110, QFX5120-48Y, or QFX5120-32C switch, you must perform this step to support pure EVPN Type 5 routes on ingress EVPN traffic.
    Note

    Entering the overlay-ecmp statement causes the Packet Forwarding Engine to restart, which interrupts forwarding operations. We recommend using this configuration statement before the EVPN-VXLAN network becomes operational.

  15. If you are configuring a QFX5110, QFX5120-48Y, or QFX5120-32C switch, and you expect that there will be more than 8000 ARP table entries and IPv6 neighbor entries, perform this step.

    Configure the maximum number of next hops reserved for use in the EVPN-VXLAN overlay network. By default, the switch allocates 8000 next hops for use in the overlay network. See next-hop for more details.

    Note

    Changing the number of next hops causes the Packet Forwarding Engine to restart, which interrupts forwarding operations. We recommend using this configuration statement before the EVPN-VXLAN network becomes operational.

Verifying the Edge-Routed Bridging Overlay on a Leaf Device

To verify that the edge-routed bridging overlay is working, run the following commands.

The commands here show output for a default instance configuration. With a MAC-VRF instance configuration, you can alternatively use:

  • show mac-vrf forwarding commands that are aliases for the show ethernet-switching commands in this section.

  • The show mac-vrf routing database command, which is an alias for the show evpn database command in this section.

The output with a MAC-VRF instance configuration displays similar information for MAC-VRF routing instances as this section shows for the default instance. One main difference you might see is in the output with MAC-VRF instances on devices where you enable the shared tunnels feature. With shared tunnels enabled, you see VTEP interfaces in the following format:

where:

  • index is the index associated with the MAC-VRF routing instance.

  • shared-tunnel-unit is the unit number associated with the shared tunnel remote VTEP logical interface.

For example, if a device has a MAC-VRF instance with index 26 and the instance connects to two remote VTEPs, the shared tunnel VTEP logical interfaces might look like this:

  1. Verify that the aggregated Ethernet interface is operational.
    user@leaf-10> show interfaces terse ae11
  2. Verify the VLAN information (associated ESIs, VTEPs, etc.).
    user@leaf-10> show vlans

    Note: esi.7585 is the ESI of the remote aggregated Ethernet link for Leaf 4, Leaf 5, and Leaf 6.

    user@leaf-10> show ethernet-switching vxlan-tunnel-end-point esi | find esi.7585

    Note: esi.7587 is the ESI for all leaf devices that have the same VNI number (Leaf 4, Leaf 5, Leaf 6, Leaf 11, and Leaf 12).

    user@leaf-10> show ethernet-switching vxlan-tunnel-end-point esi | find esi.7587

    Note: esi.8133 is the ESI for the local aggregated Ethernet interface shared with Leaf 11 and Leaf 12.

    user@leaf-10> show ethernet-switching vxlan-tunnel-end-point esi | find esi.8133
  3. Verify the ARP table.

    Note: 10.1.4.201 and 10.1.5.201 are remote end systems connected to the QFX5110 switches; and 10.1.4.202 and 10.1.5.202 are local end systems connected to Leaf 10 through interface ae11.

    user@leaf-10> show arp no-resolve vpn VRF_3
    user@leaf-10> show arp no-resolve vpn VRF_4
    user@leaf-10> show ipv6 neighbors
  4. Verify the MAC addresses and ARP information in the EVPN database.
    user@leaf-10> show evpn database mac-address 02:0c:10:04:02:01 extensive
    user@leaf-10> show evpn database mac-address 02:0c:10:04:02:02 extensive
  5. Verify the IPv4 and IPv6 end system routes appear in the forwarding table.
    user@leaf-10> show route forwarding-table table VRF_1 destination 10.1.4.202 extensive
    user@leaf-10> show route forwarding-table table VRF_1 destination 2001:db8::10:1:4:202 extensive

Edge-Routed Bridging Overlay — Release History

Table 1 provides a history of all of the features in this section and their support within this reference design.

Table 1: Edge-Routed Bridging Overlay in the Cloud Data Center Reference Design– Release History

Release

Description

19.1R2

QFX10002-60C and QFX5120-32C switches running Junos OS Release 19.1R2 and later releases in the same release train support edge-routed bridging overlays.

18.4R2

QFX5120-48Y switches running Junos OS Release 18.4R2 and later releases in the same release train support edge-routed bridging overlays.

18.1R3-S3

QFX5110 switches running Junos OS Release 18.1R3-S3 and later releases in the same release train support edge-routed bridging overlays.

17.3R3-S1

QFX10002-36Q/72Q switches running Junos 17.3R3-S1 and later releases in the same release train support edge-routed bridging overlays.