Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Example: Configuring an EVPN Control Plane and VXLAN Data Plane

 

This example shows how to configure EVPN and VXLAN on a network to support Data Center Interconnect (DCI), allow for optimal forwarding of Ethernet frames, provide network segmentation on a broad scale, enable control plane-based MAC learning, and many other advantages.

Requirements

This example uses the following hardware and software components:

  • Two Juniper Networks MX Series routers to act as IP gateways for the EVPN overlay

  • Four Juniper Networks QFX5100 switches. Two of these switches act as PE devices in the EVPN topology, and the other two switches act as pure IP transport for the overlay.

  • Junos OS Release 16.1 or later.

  • Starting with Junos OS Release 17.3R1, EVPN-VXLAN is also supported on EX9200 switches. Previously, only MPLS encapsulation was supported. In this example, the EX9200 switch would funciton as an IP gateway for the EVPN overlay. There are some configuration differences between MX Series routers and EX9200 switches, See EX9200 Configuration for more information about specific EX9200 configuration.

Overview

Ethernet VPNs (EVPNs) enable you to connect groups of dispersed customer sites using Layer 2 virtual bridges, and Virtual Extensible LANs (VXLANs) enable you to stretch Layer 2 connection over an intervening Layer 3 network, while providing network segmentation like a VLAN, but without the scaling limitation of traditional VLANs. EVPN with VXLAN encapsulation handles Layer 2 connectivity at the scale required by cloud server providers and replaces limiting protocols like STP, freeing up your Layer 3 network to use more robust routing protocols.

This example configuration shows how to configure EVPN with VXLAN encapsulation. For the purposes of this example, the MX Series routers are named Core-1 and Core-2. The QFX5100 switches are named Leaf-1, Leaf-2, Spine-1, and Spine-2. The core routers act as IP gateways for the EVPN overlay, the leaf switches act as PE devices in the EVPN topology, and the spine switches act as pure IP transport for the overlay.

Configuration

CLI Quick Configuration

To quickly configure this example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, and then copy and paste the commands into the CLI at the [edit] hierarchy level.

Leaf-1

Leaf-2

Spine-1

Spine-2

Core-1

Core-2

EX9200 Configuration

On EX9200 switches, the vlans statement is used instead of bridge-domains, and the l3-interface statement is used instead of routing-interface.

The following example shows how to configure these statements. All other configuration shown for MX Series routers in this example applies to EX9200 switches.

Note

In this example, wherever bridge-domains or routing-interface statements are used, to configure on EX9200 switches, use vlans and l3-interface instead.

Configuring Leaf-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

Configure the QFX5100 switch called Leaf-1:

  1. Set the system hostname.
  2. Configure routing options.
  3. Configure load balancing.
  4. Make sure lo0 is exported and thus advertised into the underlay, then configure family inet unicast loops 2. Doing this is necessary because of the design choice to re-use the same autonomous system (AS) number within a tier.
  5. Configure policy options for the loopback address.
  6. Configure switch options. At the [switch-options] hierarchy level, the virtual tunnel endpoint interface is lo0.0, which must be reachable through the underlay routing protocol. The route distinguisher must be unique across all switches in the network to ensure all route advertisements within MP-BGP are globally unique. The VRF table target on the QFX Series switch is, at a minimum, the community with which the switch sends all ESI (Type-1) routes. The vrf-import vrf-imp statement defines the target community list, which is imported into the default-switch.evpn.0 instance from bgp.evpn.0.
  7. Configure the VRF table import policy.
  8. Configure the community policy options.
  9. Configure the extended virtual network identifier list to establish which VXLAN network identifiers you want to be part of the EVPN and VXLAN MP-BGP domain. Next, set up the ingress replication; this EVPN and VXLAN ingress-replication is used instead of a multicast underlay. Then configure different route targets for each VXLAN network identifier instance under vni-routing-options.
  10. Map locally significant VLAN IDs to globally significant VXLAN network identifiers.
  11. Configure the EVPN MP-BGP sessions.
  12. Configure the Gigabit Ethernet interfaces.
  13. Configure two LACP-enabled LAG interfaces. The ESI value is globally unique across the entire EVPN domain. The all-active configuration statement ensures that all PE routers to which this multihomed tenant is attached to can forward traffic from the CE device, such that all CE links are actively used.
  14. Configure the loopback interface address.

Configuring Leaf-2

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

The configuration of Leaf-2 is very similar to the configuration of Leaf-1. Configure the QFX5100 switch called Leaf-2:

  1. Set the system hostname.
  2. Configure routing options.
  3. Configure load balancing.
  4. Make sure lo0 is exported and thus advertised into the underlay, then configure family inet unicast loops 2. Doing this is necessary because of the design choice to re-use the same autonomous system (AS) number within a tier.
  5. Configure policy options for the loopback address.
  6. Configure switch options. At the [switch-options] hierarchy level, the virtual tunnel endpoint interface is lo0.0, which must be reachable through the underlay routing protocol. The route distinguisher must be unique across all switches in the network to ensure all route advertisements within MP-BGP are globally unique. The VRF table target on the QFX Series switch is, at a minimum, the community with which the switch sends all ESI (Type-1) routes. The vrf-import vrf-imp statement defines the target community list, which is imported into the default-switch.evpn.0 instance from bgp.evpn.0.
  7. Configure the VRF table import policy.
  8. Configure the community policy options.
  9. Configure the extended virtual network identifier list to establish which VXLAN network identifiers you want to be part of the EVPN and VXLAN MP-BGP domain. Next, set up the ingress replication; this EVPN and VXLAN ingress-replication is used instead of a multicast underlay. Then configure different route targets for each VXLAN network identifier instance under vni-routing-options.
  10. Map locally significant VLAN IDs to globally significant VXLAN network identifiers.
  11. Configure the EVPN MP-BGP sessions.
  12. Configure the Gigabit Ethernet interfaces.
  13. Configure two LACP-enabled LAG interfaces. The ESI value is globally unique across the entire EVPN domain. The all-active configuration statement ensures that all PE routers to which this multihomed tenant is attached to can forward traffic from the CE device, such that all CE links are actively used.
  14. Configure the loopback interface address.

Configuring Spine-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

The configuration of the spine switches is similar to the leaf switches. To configure Spine-1:

  1. Set the system hostname.
  2. Configure the routing options.
  3. Configure load balance policy options.
  4. Configure the underlay Leaf group with the advertise-peer-as statement to enable Spine-1 to bypass EBGP rules and re-advertise a route to the same autonomous system (AS) number.
  5. Configure the underlay Core group. The MX Series Core routers are not EVPN peered, and do not require reachability to the other’s loopback, so including the advertise-peer-as configuration statement is optional.
  6. Configure policy options for the loopback address.

Configuring Spine-2

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

The configuration of the spine switches is similar to the leaf switches. To configure Spine-2:

  1. Set the system hostname.
  2. Configure the routing options.
  3. Configure load balance policy options.
  4. Configure the underlay Leaf group with the advertise-peer-as statement to enable Spine-2 to bypass EBGP rules and re-advertise a route to the same autonomous system (AS) number.
  5. Configure the underlay Core group. The MX Series Core routers are not EVPN peered, and do not require reachability to the other’s loopback, so including the advertise-peer-as configuration statement is optional.
  6. Configure policy options for the loopback address.

Configuring Core-1

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

To configure Core-1:

  1. Set the system hostname.
  2. Configure the router ID and other routing options.
  3. Configure the load balance policy options.
  4. Configure the BGP underlay group.
  5. Configure the loopback address policy options.
  6. A large portion of Core-1’s configuration takes place in the [routing-instance] hierarchy. Configure the virtual routers and configure a unique VRF table import policy for each virtual switch.
  7. Configure the individual policy statements.
  8. Configure the community policy options. Make sure the comm-leaf_esi policy option accepts target 9999:9999 to ensure that all virtual switches import the Type-1 ESI routes from all leafs.
  9. Configure the IRB interfaces. Every IRB has a virtual gateway address, which is a shared MAC address and IP address across Core-1 and Core-2.
  10. Configure MP-BGP sessions towards Leaf-1 and Leaf-2.

Configuring Core-2

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For information about navigating the CLI, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

To configure Core-2:

  1. Set the system hostname.
  2. Configure the router ID and other routing options.
  3. Configure the load balance policy options.
  4. Configure the BGP underlay group.
  5. Configure the loopback address policy options.
  6. A large portion of Core-2’s configuration takes place in the [routing-instance] hierarchy. Configure the virtual routers and configure a unique VRF table import policy for each virtual switch.
  7. Configure the individual policy statements.
  8. Configure the community policy options. Make sure the comm-leaf_esi policy option accepts target 9999:9999 to ensure that all virtual switches import the Type-1 ESI routes from all leafs.
  9. Configure the IRB interfaces. Every IRB has a virtual gateway address, which is a shared MAC address and IP address across Core-1 and Core-2.
  10. Configure MP-BGP sessions towards Leaf-1 and Leaf-2.

Verification

After you configure both the underlay and EVPN overlay we recommend that you verify that the configurations work as you intended.

Verifying Leaf-1 Configuration

Purpose

Verify that Leaf-1 is properly configured.

Action

Verify that the routing options are properly configured.

user@leaf-1> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@leaf-1> show configuration policy-options policy-statement

Verify the underlay BGP group configuration.

user@leaf-1> show configuration protocols bgp group underlay

Verify that the loopback address is properly configured.

user@leaf-1> show configuration policy-options policy-statement lo0

Verify the configuration of the switch options.

user@leaf-1> show configuration switch-options

Verify the configuration of the VRF table import policy statement.

user@leaf-1> show configuration policy-options policy-statement

Verify the policy options for GREP.

user@leaf-1> show configuration policy-options | grep members

Verify that EVPN is properly configured with VXLAN details.

user@leaf-1> show configuration protocols evpn

Verify the EVPN-VXLAN Core BGP group configuration.

user@leaf-1> show configuration protocols bgp group EVPN_VXLAN_CORE

Verify the EVPN-VXLAN Leaf BGP group configuration.

user@leaf-1> show configuration protocols bgp group EVPN_VXLAN_LEAF

Verify the configuration for the xe-0/0/32 interface.

user@leaf-1> show configuration interfaces xe-0/0/32

Verify the configuration for the xe-0/0/33 interface.

user@leaf-1> show configuration interfaces xe-0/0/33

Verify the configuration for the ae0 interface.

user@leaf-1> show configuration interfaces ae0

Verify the configuration for the ae1 interface.

user@leaf-1> show configuration interfaces ae1

Verifying Leaf-2 Configuration

Purpose

Verify that Leaf-2 is properly configured.

Action

Verify that the routing options are properly configured.

user@leaf-2> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@leaf-2> show configuration policy-options policy-statement

Verify the underlay BGP group configuration.

user@leaf-2> show configuration protocols bgp group underlay

Verify that the loopback address is properly configured.

user@leaf-2> show configuration policy-options policy-statement lo0

Verify the configuration of the switch options.

user@leaf-2> show configuration switch-options

Verify the configuration of the VRF table import policy statement.

user@leaf-2> show configuration policy-options policy-statement

Verify the policy options for GREP.

user@leaf-2> show configuration policy-options | grep members

Verify that EVPN is properly configured with VXLAN details.

user@leaf-2> show configuration protocols evpn

Verify the EVPN-VXLAN Core BGP group configuration.

user@leaf-2> show configuration protocols bgp group EVPN_VXLAN_CORE

Verify the EVPN-VXLAN Leaf BGP group configuration.

user@leaf-2> show configuration protocols bgp group EVPN_VXLAN_LEAF

Verify the configuration for the xe-0/0/36 interface.

user@leaf-2> show configuration interfaces xe-0/0/36

Verify the configuration for the xe-0/0/37 interface.

user@leaf-2> show configuration interfaces xe-0/0/37

Verify the configuration for the ae0 interface.

user@leaf-2> show configuration interfaces ae0

Verify the configuration for the ae1 interface.

user@leaf-2> show configuration interfaces ae1

Verifying Spine-1 Configuration

Purpose

Verify that Spine-1 is properly configured.

Action

Verify that the routing options are properly configured.

user@spine-1> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@spine-1> show configuration policy-options policy-statement load-balance

Verify the underlay Leaf BGP group configuration.

user@spine-1> show configuration protocols bgp group underlay-leaf

Verify the underlay Core BGP group configuration.

user@spine-1> show configuration protocols bgp group underlay-core

Verify that the loopback address is properly configured.

user@spine-1> show configuration policy-options policy-statement lo0

Verifying Spine-2 Configuration

Purpose

Verify that Spine-2 is properly configured.

Action

Verify that the routing options are properly configured.

user@spine-2> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@spine-2> show configuration policy-options policy-statement load-balance

Verify the underlay Leaf BGP group configuration.

user@spine-2> show configuration protocols bgp group underlay-leaf

Verify the underlay Core BGP group configuration.

user@spine-2> show configuration protocols bgp group underlay-core

Verify that the loopback address is properly configured.

user@spine-2> show configuration policy-options policy-statement lo0

Verifying Core-1 Configuration

Purpose

Verify that Core-1 is properly configured.

Action

Verify that the routing options are properly configured.

user@core-1> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@core-1> show configuration policy-options policy-statement load-balance

Verify the underlay BGP group configuration.

user@core-1> show configuration protocols bgp

Verify that the loopback address is properly configured.

user@core-1> show configuration policy-options policy-statement lo0

Verify that the routing instances are properly configured.

user@core-1> show configuration routing-instances

Verify that the policy options are properly configured.

user@core-1> show configuration policy-options

Verify the configuration of the IRB interface.

user@core-1> show configuration interfaces irb

Verify the configuration of the EVPN-VXLAN BGP group.

user@core-1> show configuration protocols bgp group EVPN_VXLAN

Verifying Core-2 Configuration

Purpose

Verify that Core-2 is properly configured.

Action

Verify that the routing options are properly configured.

user@core-2> show configuration routing-options

Verify that the load-balance policy statement is properly configured.

user@core-2> show configuration policy-options policy-statement load-balance

Verify the underlay BGP group configuration.

user@core-2> show configuration protocols bgp

Verify that the loopback address is properly configured.

user@core-2> show configuration policy-options policy-statement lo0

Verify that the routing instances are properly configured.

user@core-2> show configuration routing-instances

Verify that the policy options are properly configured.

user@core-2> show configuration policy-options

Verify the configuration of the IRB interface.

user@core-2> show configuration interfaces irb

Verify the configuration of the EVPN-VXLAN BGP group.

user@core-2> show configuration protocols bgp group EVPN_VXLAN

Verifying MAC Reachability to a Single-Homed CE Device (Leaf-1)

Purpose

Verify MAC reachability to Tenant_C single-homed to Leaf-1. First you need to verify that the MAC address is learned locally on Leaf-1. Leaf-1 generates the Type-2 route only after it has learned the MAC address.

Action

Verify that the MAC address is learned locally on Leaf-1.

Meaning

The output shows that MAC 00:21:59:c8:24:65 is successfully learned towards the Tenant_C CE device on xe-0/0/34.0, and from the output below, we can see that we generate the Type-2 route to Core-1.

Verifying MAC Reachability to a Single-Homed CE Device (Type-2)

Purpose

Verify MAC reachability to a single-homed CE device (Type-2)

Action

Verify the generation of the Type-2 route to Core-1.

Meaning

The output shows that the same MAC address is advertised twice, with two different route targets. On xe-0/0/34, Tenant_C and Tenant_D are on the same physical server, running on different virtual machines. Tenant isolation is preserved by advertising these MACs on different VNIs and RTs.

On Core-1, the Type-2 route is received into bgp.evpn.0.

The output shows two Type-2 routes for 00:21:59:c8:24:65, but with different route targets. The route distinguisher is from Leaf-1, set as 192.0.2.2:1.

Verifying the Imported Route

Purpose

Verify that the Type-2 route is imported.

Action

On Core-1, verify whether this Type-2 route is successfully imported from the bgp.evpn.0 table into the EVPN switch instance.

Meaning

The output shows that, in Tenant_C’s virtual switch, the Type-2 route is advertised with the correct target, target:1:1300.

You can use the extensive option to review the Type-2 route in greater detail.

The output shows that Core-1 receives two copies. The first is the direct advertisement from Leaf-1 (Source: 192.0.2.2). The second is the indirect advertisement from Leaf-1 -> Leaf-2 -> Core-1 (Source: 10.255.255.5). Because we configured no-nexthop-change on Leaf-2, the protocol next hop of 192.0.2.2 is correct.

Note

The no-nexthop-change command must be configured on multihomed provider edge (PE) devices with EBGP configuration. This allows other PEs to load balance the traffic on the multihomed PEs.

Verifying the Layer 2 Address Learning Daemon Copy

Purpose

Verify the Layer 2 address learning daemon copy.

Action

Verify the Layer 2 address learning daemon copy by entering the show bridge-mac table command.

Meaning

The output shows that 00:21:59:c8:24:65 is reachable through the vtep.32774 logical interface to Leaf-1.

Note

On EX9200 switches, the show ethernet-switching table-instance instance-name command corresponds to the show bridge mac-table instance instance-name command on MX Series routers

Verifying the Kernel-Level Forwarding Table

Purpose

Verify the kernel-level forwarding table, next hop identifier, and Layer 2 MAC table and hardware.

Action

Query the kernel-level forwarding table, correlate the index next hop identifier with the correct virtual network identifier, and review the Layer 2 MAC table and hardware.

Meaning

Tenant_C’s MAC, 00:21:59:c8:24:65, is reachable through index 699.

Correlate index 699 (NH-Id) with the correct virtual network identifier 1300 and remote VTEP-ID of 192.0.2.2.

Note

On EX9200 switches, the show ethernet-switching command corresponds to the show l2-learning command on MX Series routers.

The output shows that 00:21:59:c8:24:65 is successfully programmed in Packet Forwarding Engine hardware.

Note

On EX9200 switches, the show ethernet-switching command corresponds to the show l2-learning command on MX Series routers.

Verifying MAC Reachability to a Multihomed CE Device

Purpose

Verify MAC reachability to the multihomed Tenant_B CE device on Leaf-1 and Leaf-2.

Action

Verify that Leaf-1 and Leaf-2 are advertising both Type-1 and Type-2 reachability towards the multihomed CE device.

Verify that interface ae1 belongs on ESI 00:02:02:02:02:02:02:02:02:02.

Meaning

The output shows that 00:21:59:c8:24:64 represents the physical MAC of the Tenant_B NIC physically attached to Leaf-1. 00:21:59:c8:24:68 represents the physical MAC of the Tenant_B NIC physically attached to Leaf-2. 00:21:59:c8:24:41 is a virtual MAC that, at the time of the capture, has been learned only on Leaf-2.

Verifying EVPN, Layer 2 Address Learning Daemon, and the Kernel-Forwarding Tables

Purpose

Verify the Tenant B’s EVPN table, and Core-1’s Layer 2 address learning daemon table and kernel-forwarding table.

Action

In Core-1, display the Tenant B’s EVPN table.

Display Core-1’s Layer 2 address learning daemon table.

Note

On EX9200 switches, the show ethernet-switching table-instance instance-name command corresponds to the show bridge mac-table instance instance-name command on MX Series routers

Display Core-1’s kernel forwarding table.

Meaning

For the Tenant_B CE device, four different routes are listed for ESI 00:02:02:02:02:02:02:02:02:02.

  • 1:192.0.2.2:0::020202020202020202::FFFF:FFFF/304

    This per-Ethernet Segment A-D Type-1 EVPN route originated from Leaf-1. The route distinguisher is taken from global-level routing-options. Core-1 receives this Type-1 route, originated from Leaf-1, from both Leaf-1 and Leaf-2.

  • 1:192.0.2.2:1::020202020202020202::0/304

    This is the per-EVI A-D Type-1 EVPN route. The route distinguisher is taken from the routing instance, or in the case of QFX5100, switch-options. Core-1 receives this Type-1 route, originated from Leaf-1, from both Leaf-1 and Leaf-2.

  • 1:10.255.255.5:0::020202020202020202::FFFF:FFFF/304

    This is the per-Ethernet Segment A-D Type-1 EVPN route originated from Leaf-2. The route distinguisher is taken from global-level routing-options. Core-1 receives this Type-1 route, originated from Leaf-2, from both Leaf-2 and Leaf-1.

  • 1:10.255.255.5:1::020202020202020202::0/304

    This is the per-EVI A-D Type-1 EVPN route. The route distinguisher is taken from the routing instance, or in the case of QFX5100, switch-options. Core-1 receives this Type-1 route, originated from Leaf-2, from both Leaf-2 and Leaf-1.

Type-2 routes for the two physical and one virtual MAC belonging to the Tenant_B CE device originated as expected.

From the output we cannot yet determine what VTEPs are being used to forward to ESI 00:02:02:02:02:02:02:02:02:02. To determine the VTEPS, list the VXLAN tunnel endpoint ESIs.

Note

On EX9200 switches, the show ethernet-switching command corresponds to the show l2-learning command on MX Series routers.

The output shows active load-balancing on the VTEP interfaces to both Leaf-1 and Leaf-2 for MACs on this ESI, which validates the all-active configuration on Leaf-1 and Leaf-2.

The reference to esi.703 might be confusing. This is an internal unilist next hop that comprises multiple, valid, individual ECMP next hops. You can correlate these individual next hops to the unilist by executing this shell level command:

From the output we can reference index 720 with vtep.32778 to Leaf-2 and index 702 with vtep.32774 to Leaf-1.

Verifying an EVPN Anycast Gateway

Purpose

Verifying the EVPN anycast gateway is similar to the multihomed CE device scenario. For this example, verify from the leaf (QFX5100) perspective.

Action

For this example, verify the anycast gateway on both Core-1 and Core-2 for VNI 1100 (Vlan 100 on both Leaf-1 and Leaf-2).

Verify the same Type-1, same anycast Type-2, and different physical Type-2 (a.b.c.3) from Core-2.

Similarly, if we look at the default-switch.evpn.0 table on leaf-1, we see for the ESI for VNI1110:

Meaning

The output shows that Leaf-1 receives Type-1 advertisements for the autogenerated ESI’s for the anycast gateway. The outputs also show the anycast and physical MACs (a.b.c.2) for each bridge domain.

Two unique advertisements are received for this ESI. One from Core-1, and the other from Core-2, with corresponding route distinguishers. Because Leaf-1 is iBGP peered with Leaf-2, the extra copy for each route from Leaf-2 displays with the original protocol-next hop preserved.

Leaf-1’s Layer 2 address learning daemon table displays:

Leaf-1’s kernel table displays:

From the output, Leaf-1 has selected index 1724, correlating with vtep.32769, to forward all traffic for the anycast gateway on VNI 1100, and vtep.32769 is the VTEP to Core-1.