Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Example: Configure an EVPN-VXLAN Centrally-Routed Bridging Fabric Using MX Routers as Spines

This example shows how to configure EVPN and VXLAN on an IP fabric to support optimal forwarding of Ethernet frames, provide network segmentation on a broad scale, enable control plane-based MAC learning, and many other advantages. This example is based on a centrally-routed with bridging (CRB) EVPN architecture in a 5-stage Clos fabric.

In the CRB architecture IRB interfaces provide Layer 3 connectivity to servers and VMS that belong to different VLANs and networks. These IRB interfaces serve as the default gateway for inter-VLAN traffic within a fabric, and also for destinations that are remote to the fabric, for example, in the case of Data Center Interconnect (DCI). In a CRB design you define the IRB interfaces on the spine devices only. Such a design is therefore referred to as being centrally routed, as all routing occurs on the spines.

For an example of an edge-routed bridging (ERB) design, see Example: Configuring an EVPN-VXLAN Edge-Routed Bridging Fabric with an Anycast Gateway

For background information on EVPN-VXLAN technology and supported architectures, see EVPN Primer.

Requirements

The original example used the following hardware and software components:

  • Two Juniper Networks MX Series routers to act as IP gateways for the EVPN overlay

  • Four Juniper Networks QFX5100 switches. Two of these switches act as PE devices in the EVPN topology, and the other two switches act as pure IP transport for the underlay.

  • Junos OS Release 16.1 or later.

    • Updated and re-validated using Junos OS Release 21.3R1.9

  • Starting with Junos OS Release 17.3R1, EVPN-VXLAN is also supported on EX9200 switches. Previously, only MPLS encapsulation was supported. In this example, the EX9200 switch would function as an IP gateway for the EVPN overlay. There are some configuration differences between MX Series routers and EX9200 switches. The configuration section later in this topic has more information about the configuration that is specific to an EX9200.

Overview

Ethernet VPNs (EVPNs) enable you to connect groups of dispersed customer sites using Layer 2 virtual bridges, and Virtual Extensible LANs (VXLANs) enable you to stretch Layer 2 connection over an intervening Layer 3 network, while providing network segmentation like a VLAN, but without the scaling limitation of traditional VLANs. EVPN with VXLAN encapsulation handles Layer 2 connectivity at the scale required by cloud service providers, and replaces limiting protocols like STP, freeing up your Layer 3 network to use more robust routing protocols.

This example configuration shows how to configure EVPN with VXLAN encapsulation. In this example, the MX Series routers are named Core-1 and Core-2. The QFX5100 switches are named Leaf-1, Leaf-2, Spine-1, and Spine-2. The core routers act as IP gateways for the EVPN overlay, the leaf switches act as PE devices in the EVPN topology, and the spine switches act as pure IP transport for the underlay (also known as a "lean spine").

Topology

In our sample topology we demonstrate server access using both untagged and trunked (tagged) interfaces. A trunk interface uses explicit VLAN tagging. Both server A and C are configured for trunking while server B uses an untagged access interface to both leaves.

Configuration

CLI Quick Configuration

To quickly configure this example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, and then copy and paste the commands into the CLI at the [edit] hierarchy level.

Leaf-1

Leaf-2

Spine-1

Spine-2

Core-1

Core-2

EX9200 Configuration

On EX9200 switches, the vlans statement is used instead of bridge-domains, and the l3-interface statement is used instead of routing-interface.

The following example shows how to configure these statements. All other configuration shown for MX Series routers in this example also applies to EX9200 switches.

Note:

In this example, wherever bridge-domains or routing-interface statements are used, to configure on EX9200 switches, use vlans and l3-interface instead.

Configuring Leaf-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

Note:

The steps for configuring Leaf-2 are similar to Leaf-1 and therefore we will only show the step-by-step procedures for Leaf-1.

To configure Leaf-1:

  1. Set the system hostname.

  2. Configure routing options. The load-balance export policy is configured in the next step.

  3. Configure the load balancing policy.

  4. Configure the underlay EBGP to the spine devices. The lo0 export policy is configured in the next step.

  5. Configure a policy to advertise the loopback address into the underlay. In this example you write a portable policy that is loopback address agnostic, by matching only direct routes with a /32 prefix length. The result is a policy that matches any loopback address and is reusable across all devices in the topology.

  6. Configure switch options The virtual tunnel endpoint interface is lo0.0, which must be reachable through the underlay routing protocol. The route distinguisher must be unique across all switches in the network to ensure all route advertisements within MP-BGP overlay are globally unique. The VRF table target on the QFX Series switch is, at a minimum, the community the switch sends attaches to all ESI (Type-1) routes. The vrf-import vrf-imp statement defines the target community list, which is imported into the default-switch.evpn.0 instance from the bgp.evpn.0 table.

  7. Configure the VRF table import policy.

  8. Configure the related communities.

  9. Configure the extended virtual network identifier (VNI) list to establish the VNIs you want to be part of the EVPN domain. You also configure ingress replication; in EVPN-VXLAN ingress-replication is used to handle multicast without requiring a multicast capable underlay. Different route targets are specified for each VXLAN network identifier instance under vni-routing-options.

  10. Map locally significant VLAN IDs to globally significant VXLAN network identifiers.

  11. Configure the EVPN capable IBGP overlay sessions.

    Note:

    Some IP fabrics use an EBGP based EVPN-VXLAN overlay. For an example of an IP fabric that uses EBGP for both the underlay and overlay, see Example: Configuring an EVPN-VXLAN Edge-Routed Bridging Fabric with an Anycast Gateway. Note that the choice of EBGP vs IBGP for the overlay does not impact on the fabric architecture. Both CRB and edge-routed bridging (ERB) designs support either type of overlay.

  12. Configure the fabric interfaces.

  13. Configure the access interfaces. Note again that we demonstrate a mix of access and trunk interfaces for server attachment.

  14. Configure the LACP-enabled LAG interface. The ESI value is globally unique across the entire EVPN domain. The all-active configuration statement ensures that all PE routers to which this multihomed tenant is attached to can forward traffic from the CE device, such that all CE links are actively used.

  15. Configure the loopback interface address.

Configuring Spine-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

Note:

The steps for configuring Spine-2 are similar to Spine-1 and therefore we will only show the step-by-step procedures for Spine-1.

To configure Spine-1:

  1. Set the system hostname.

  2. Configure the routing options.

  3. Configure a load balancing policy.

  4. Configure the EBGP underlay with peering to the leaf and core devices. The lo0 policy that advertises the lo0 address is applied in this step; the configuration of the policy itself is shown in the next step.

  5. Configure a policy named lo0 to advertise /32 routes. The policy matches on the loopback address, without specifying any specific IP. In this way the same policy is reusable on any fabric device.

Configuring Core-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

Note:

The steps for configuring Core-2 are similar to Core-1 and therefore we will only show the step-by-step procedures for Core-1.

To configure Core-1:

  1. Set the system hostname.

  2. Configure the routing options. The load-balance policy is applied during this step. You create the policy in the next step

  3. Configure a load balancing policy named load-balance.

  4. Configure the BGP underlay peering. The lo0 policy that advertises the loopback address is applied during this step. You configure this policy in the next step.

  5. Configure a policy named lo0 to advertise loopback routes.

  6. A large portion of Core-1’s configuration takes place in the [routing-instance] hierarchy. Configure the virtual routers and configure a unique VRF table import policy for each virtual switch.

  7. Configure the policy for each routing instance.

  8. Configure the communities . Make sure that the comm-leaf policy accepts routes tagged with target 65000:1. This ensures that all virtual switches import the Type-1 ESI routes from all leafs.

  9. Configure the IRB interfaces. Every IRB has a virtual gateway address, which is a shared MAC address and IP address across Core-1 and Core-2.

  10. Configure the IBGP overlay sessions towards Leaf-1 and Leaf-2. We've include a peering between the Core devices for route sharing between Core devices.

Verification

Verifying MAC Reachability to a Single-Homed CE Device (Leaf-1)

Purpose

Verify MAC reachability to Tenant_A. This user is single-homed to Leaf-1. First, verify that the MAC address is learned locally on Leaf-1. Leaf-1 generates the Type-2 EVPN route only after it learns the MAC address.

Action

Verify that the MAC address is learned locally on Leaf-1.

Meaning

The output shows that MAC 56:04:15:00:bb:02 is successfully learned from the Tenant_A CE device, which is Server A on the xe-0/0/3.0 interface.

Verifying MAC Reachability to a Single-Homed CE Device (Type-2)

Purpose

Verify MAC reachability to a single-homed CE device (Type-2)

Action

Verify the generation of the Type-2 route to Core-1.

Meaning

The output shows that the MAC and MAC/IP are being advertised.

On Core-1, the EVPN Type-2 route is received into bgp.evpn.0.

The output shows the Type-2 routes for 56:04:15:00:bb:02. The route distinguisher is from Leaf-1 and is set to 10.1.255.111:1.

Verifying Imported Routes

Purpose

Verify that the EVPN Type-2 route is imported.

Action

On Core-1, verify whether EVPN Type-2 routes are successfully imported from the bgp.evpn.0 table into the EVPN switch instance.

Meaning

The output shows that, in Tenant_A's virtual switch, the EVPN Type-2 route is advertised with the correct target, target:1:101. Use the extensive option to review the Type-2 route in greater detail.

The output shows that Core-1 receives two copies. The first is the advertisement from Leaf-1 (Source: 10.1.255.111). The second is the advertisement from Core-2 (Source: 10.1.255.2).

Verifying the Layer 2 Address Learning Daemon Copy

Purpose

Verify the Layer 2 address learning daemon copy.

Action

Verify the Layer 2 address learning daemon copy by entering the show bridge-mac table command.

Meaning

The output shows that 56:04:15:00:bb:02 is reachable through the vtep.32771 logical interface to Leaf-1.

Note:

On EX9200 switches, the show ethernet-switching table-instance instance-name command corresponds to the show bridge mac-table instance instance-name command used here for MX Series routers

Verifying the Kernel-Level Forwarding Table

Purpose

Verify the kernel-level forwarding table, next hop identifier, and Layer 2 MAC table and hardware.

Action

Query the kernel-level forwarding table, correlate the index next hop identifier with the correct virtual network identifier, and review the Layer 2 MAC table and hardware.

Meaning

Tenant_A’s MAC, 56:04:15:00:bb:02, is reachable through index 687.

Correlate index 687 (NH-Id) with the correct virtual network identifier 101 and remote VTEP-ID of 10.1.255.111.

Note:

On EX9200 switches, the show ethernet-switching command corresponds to the show l2-learning command show here for MX Series routers.

Verifying MAC Reachability to a Multihomed CE Device

Purpose

Verify MAC reachability to the multihomed Tenant_B CE device on Leaf-1 and Leaf-2.

Action

Verify that Leaf-1 and Leaf-2 are advertising both Type-1 and Type-2 reachability towards the multihomed CE device.

Meaning

The output shows that 2c:6b:f5:43:12:c0 represents the MAC of the Tenant_B attached to Leaf-1 and Leaf-2.

Verifying EVPN, Layer 2 Address Learning Daemon, and the Kernel-Forwarding Tables for Multihomed CE Device

Purpose

Verify the Tenant B’s EVPN table, and Core-1’s Layer 2 address learning daemon table and kernel-forwarding table.

Action

In Core-1, display the Tenant B’s EVPN table.

Display Core-1’s Layer 2 address learning daemon table.

Note:

On EX9200 switches, the show ethernet-switching table-instance instance-name command corresponds to the show bridge mac-table instance instance-name command show here for MX Series routers

Display Core-1’s kernel forwarding table.

Meaning

For the Tenant_B CE device, four different routes are listed for ESI 00:01:01:01:01:01:01:01:01:01:

  • 1:10.1.255.111:0::010101010101010101::FFFF:FFFF/192 AD/ESI

    This per-Ethernet Segment A-D Type-1 EVPN route originated from Leaf-1. The route distinguisher is obtained from global-level routing-options. Core-1 receives this Type-1 route, originated from Leaf-1, from both Leaf-1 and Leaf-2.

  • 1:10.1.255.111:1::010101010101010101::0/192 AD/EVI

    This is the per-EVI A-D Type-1 EVPN route. The route distinguisher is obtained from the routing instance, or in the case of QFX5100, the switch-options. Core-1 receives this Type-1 route, originated from Leaf-1, from both Leaf-1 and Leaf-2.

  • 1:10.1.255.112:0::010101010101010101::FFFF:FFFF/192 AD/ESI

    This is the per-Ethernet Segment A-D Type-1 EVPN route originated from Leaf-2. The route distinguisher is obtained from global-level routing-options. Core-1 receives this Type-1 route, originated from Leaf-2, from both Leaf-2 and Leaf-1.

  • 1:10.1.255.112:1::010101010101010101::0/192 AD/EVI

    This is the per-EVI A-D Type-1 EVPN route. The route distinguisher is obtained from the routing instance, or in the case of QFX5100, switch-options. Core-1 receives this Type-1 route, originated from Leaf-2, from both Leaf-2 and Leaf-1.

The Type-2 routes for the two physical and one virtual MAC associated with the Tenant_B multihomed CE device are originated as expected.

From the output we cannot yet determine what VTEPs are used to forward to ESI 00:01:01:01:01:01:01:01:01:01. To determine the VTEPS, display the VXLAN tunnel endpoint ESIs.

Note:

On EX9200 switches, the show ethernet-switching command corresponds to the show l2-learning command show here for MX Series routers.

The output shows active load-balancing on the VTEP interfaces to both Leaf-1 and Leaf-2 for the MAC addresses on this ESI, which validates the all-active configuration on Leaf-1 and Leaf-2.

Related Documentation