Bridged Overlay Design and Implementation
A bridged overlay provides Ethernet bridging between leaf devices in an EVPN network, as shown in Figure 1. This overlay type simply extends VLANs between the leaf devices across VXLAN tunnels. Bridged overlays provide an entry level overlay style for data center networks that require Ethernet connectivity but do not need routing services between the VLANs.
In this example, loopback interfaces on the leaf devices act as VXLAN tunnel endpoints (VTEPs). The tunnels enable the leaf devices to send VLAN traffic to other leaf devices and Ethernet-connected end systems in the data center. The spine devices only provide basic EBGP underlay and IBGP overlay connectivity for these leaf-to-leaf VXLAN tunnels.
If inter-VLAN routing is required for a bridged overlay, you can use an MX Series router or SRX Series security device that is external to the EVPN/VXLAN fabric. Otherwise, you can select one of the other overlay types that incorporate routing (such as an edge-routed bridging overlay, a centrally-routed bridging overlay, or a routed overlay) discussed in this Cloud Data Center Architecture Guide.
The following sections provide the detailed steps of how to configure a bridged overlay:
Configuring a Bridged Overlay
Bridged overlays are supported on all platforms included in this reference design. To configure a bridged overlay, you configure VNIs, VLANs, and VTEPs on the leaf devices, and BGP on the spine devices. We support either an IPv4 Fabric or an IPv6 Fabric (with supported platforms) as the fabric infrastructure with bridged overlay architectures.
When you implement this style of overlay on a spine device, the focus is on providing overlay transport services between the leaf devices. Consequently, you configure an IP Fabric underlay and IBGP overlay peering with IPv4, or an IPv6 Fabric underlay with EBGP IPv6 overlay peering. There are no VTEPs or IRB interfaces needed, because the spine device does not provide any routing functionality or EVPN/VXLAN capabilities in a bridged overlay.
On the leaf devices, you can configure a bridged overlay using the default switch instance or using MAC-VRF instances.
We support EVPN-VXLAN on devices running Junos OS Evolved only with MAC-VRF instance configurations.
In addition, we support the IPv6 Fabric infrastructure design only with MAC-VRF instance configurations.
Some configuration steps that affect the Layer 2 configuration differ with MAC-VRF instances. Likewise, one or two steps might differ for IPv6 Fabric configurations compared to IPv4 Fabric configurations. The leaf device configuration includes the following steps:
Enable EVPN with VXLAN encapsulation to connect to other leaf devices, and configure the loopback interface as a VTEP source interface. If you are using MAC-VRF instances instead of the default switching instance, configure a MAC-VRF instance with these parameters in the MAC-VRF instance. If your fabric uses an IPv6 Fabric, you configure the VTEP source interface as an IPv6 interface.
Establish route targets and route distinguishers. If you are using MAC-VRF instances instead of the default switching instance, configure a MAC-VRF instance with these parameters in the MAC-VRF instance.
Configure Ethernet Segment Identifier (ESI) settings.
Map VLANs to VNIs.
Again, you do not include IRB interfaces or routing on the leaf devices for this overlay method.
The following sections provide the detailed steps of how to configure and verify the bridged overlay:
Configuring a Bridged Overlay on the Spine Device
To configure a bridged overlay on a spine device, perform the following steps:
The following example shows the configuration for Spine 1, as shown in Figure 2.
- Ensure the IP fabric
underlay is in place. To configure an IP fabric on a spine device,
see IP Fabric Underlay Network Design and Implementation.
If you are using an IPv6 Fabric, see IPv6 Fabric Underlay and Overlay Network Design and Implementation with EBGP instead. Those instructions include how to configure the IPv6 underlay connectivity with EBGP and IPv6 overlay peering.
- Confirm that your IBGP overlay is up and running. To configure
an IBGP overlay on your spine devices, see Configure IBGP for the Overlay.
If you are using an IPv6 Fabric, you don’t need this step. Step 1 also covers how to configure the EBGP IPv6 overlay peering that corresponds to the IPv6 underlay connectivity configuration.
Verifying a Bridged Overlay on the Spine Device
Issue the following commands to verify that the overlay is working properly on your spine devices:
- Verify that the spine device has reachability to the leaf
devices. This output shows the possible routes to Leaf 1.
(With an IPv6 Fabric, enter this command with the IPv6 address of the spine device instead of an IPv4 address.)
user@spine-1> show route 192.168.1.1inet.0: 446 destinations, 19761 routes (446 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 192.168.1.1/32 *[BGP/170] 00:06:29, localpref 100 AS path: 4200000010 I, validation-state: unverified > to 172.16.1.1 via ae1.0 ... [BGP/170] 00:06:18, localpref 100 AS path: 4200000106 4200000004 4200000010 I, validation-state: unverified > to 172.16.96.1 via ae96.0 - Verify that IBGP is functional on the spine devices acting
as a route reflector cluster. You should see peer relationships with
all spine device loopback interfaces (192.168.0.1 through 192.168.0.4)
and all leaf device loopback interfaces (192.168.1.1 through 192.168.1.96).
Use the same command if you have an IPv6 Fabric with EBGP IPv6 overlay peering. In the output, look for the IPv6 addresses of the peer device interconnecting interfaces to verify underlay EBGP connectivity. Look for peer device loopback addresses to verify overlay EBGP peering. Ensure that the state is Establ (established).
user@spine-1> show bgp summaryGroups: 5 Peers: 221 Down peers: 0 Table Tot Paths Act Paths Suppressed History Damp State Pending inet.0 9711 182 0 0 0 0 ... Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 192.168.0.2 421000001 28724 31106 0 0 22:40:41 Establ 192.168.0.3 421000001 27424 32106 0 0 22:43:41 Establ 192.168.0.4 421000001 29457 30494 0 0 22:39:04 Establ 192.168.1.1 421000001 3814 75108 0 0 22:43:54 Establ ... 192.168.1.96 421000001 4831 73047 0 0 22:43:41 Establ
Configuring a Bridged Overlay on the Leaf Device
To configure a bridged overlay on a leaf device, perform the following:
The following example shows the configuration for Leaf 1, as shown in Figure 3.
- Configure the IP Fabric underlay and overlay:
For an IP Fabric underlay using IPv4:
Ensure the IP fabric underlay is in place. To configure an IP fabric on a leaf device, see IP Fabric Underlay Network Design and Implementation.
Confirm that your IBGP overlay peering is up and running. To configure IPv4 IBGP overlay peering on your leaf device, see Configure IBGP for the Overlay.
For an IPv6 Fabric underlay with EBGP IPv6 overlay peering:
Ensure the IPv6 underlay is in place and the EBGP overlay peering is up and running. To configure an IPv6 Fabric, see IPv6 Fabric Underlay and Overlay Network Design and Implementation with EBGP.
- Configure the EVPN protocol with VXLAN encapsulation,
and specify the VTEP source interface (in this case, the loopback
interface of the leaf device).
If your configuration uses the default instance, you configure EVPN-VXLAN at the global level. You also specify the VTEP source interface at the [edit switch-options] hierarchy level:
Leaf 1 (Default Instance):
set protocols evpn encapsulation vxlanset protocols evpn extended-vni-list allset switch-options vtep-source-interface lo0.0If your configuration uses MAC-VRF instances, define a routing instance of type mac-vrf. Then configure EVPN-VXLAN and the VTEP source interface at that MAC-VRF routing instance hierarchy level. You also must configure a service type for the MAC-VRF instance. We configure the vlan-aware service type so you can associate multiple VLANs with the MAC-VRF instance. This setting is consistent with the alternative configuration that uses the default instance.
Leaf 1 (MAC-VRF Instance):
set routing-instances MAC-VRF-1 instance-type mac-vrfset routing-instances MAC-VRF-1 service-type vlan-awareset routing-instances MAC-VRF-1 protocols evpn encapsulation vxlanset routing-instances MAC-VRF-1 protocols evpn extended-vni-list allset routing-instances MAC-VRF-1 vtep-source-interface lo0.0If you have an IPv6 Fabric infrastructure (supported only with MAC-VRF instances), in this step you include the inet6 option when you configure the VTEP source interface to use the device loopback address. This option enables IPv6 VXLAN tunneling in the fabric. This is the only difference in the MAC-VRF instance configuration with an IPv6 Fabric as compared to the MAC-VRF instance configuration with an IPv4 Fabric.
Leaf 1 (MAC-VRF Instance with an IPv6 Fabric):
set routing-instances MAC-VRF-1 instance-type mac-vrfset routing-instances MAC-VRF-1 service-type vlan-awareset routing-instances MAC-VRF-1 protocols evpn encapsulation vxlanset routing-instances MAC-VRF-1 protocols evpn extended-vni-list allset routing-instances MAC-VRF-1 vtep-source-interface lo0.0 inet6 - Define an EVPN route target and route distinguisher, and
use the auto option to derive route targets automatically.
Setting these parameters specifies how the routes are imported and
exported. The import and export of routes from a bridging table is
the basis for dynamic overlays. In this case, members of the global
BGP community with a route target of target:64512:1111 participate
in the exchange of EVPN/VXLAN information.
If your configuration uses the default instance, you use statements in the [edit switch-options] hierarchy, as follows:
Leaf 1 (Default Instance):
set switch-options route-distinguisher 192.168.1.1:1set switch-options vrf-target target:64512:1111set switch-options vrf-target autoThe main difference with a MAC-VRF configuration is that you configure these statements in the MAC-VRF instance at the [edit routing-instances mac-vrf-instance-name] hierarchy level, as follows:
Leaf 1 (MAC-VRF Instance):
set routing-instances MAC-VRF-1 route-distinguisher 192.168.1.1:1set routing-instances MAC-VRF-1 vrf-target target:64512:1111set routing-instances MAC-VRF-1 vrf-target autoNote A specific route target processes EVPN Type 1 routes, while an automatic route target processes Type 2 routes. This reference design requires both route targets.
- (MAC-VRF instances only) Enable shared tunnels on devices
in the QFX5000 line running Junos OS.
A device can have problems with VTEP scaling when the configuration uses multiple MAC-VRF instances. As a result, to avoid this problem, we require that you enable the shared tunnels feature on the QFX5000 line of switches running Junos OS with a MAC-VRF instance configuration. When you configure the shared tunnels option, the device minimizes the number of next-hop entries to reach remote VTEPs. This statement is optional on the QFX10000 line of switches running Junos OS because those devices can handle higher VTEP scaling than QFX5000 switches. You also don’t need to configure this option on devices running Junos OS Evolved, where shared tunnels are enabled by default.
Include the following statement to globally enable shared VXLAN tunnels on the device:
set forwarding-options evpn-vxlan shared-tunnelsNote This setting requires you to reboot the device.
- (Required on PTX10000 Series routers only) Enable tunnel termination
globally (in other words, on all interfaces) on the device: set forwarding-options tunnel-termination
- Configure ESI settings. Because the end systems in this
reference design are multihomed to three leaf devices per device type
cluster (such as QFX5100), you must configure the same ESI identifier
and LACP system identifier on all three leaf devices for each unique
end system. Unlike other topologies where you would configure a different
LACP system identifier per leaf device and have VXLAN select a single
designated forwarder, use the same LACP system identifier to allow
the 3 leaf devices to appear as a single LAG to a multihomed end system.
In addition, use the same aggregated Ethernet interface number for
all ports included in the ESI.
The configuration for Leaf 1 is shown below, but you must replicate this configuration on both Leaf 2 and Leaf 3 per the topology shown in Figure 4.
Tip When you create an ESI number, always set the high order octet to 00 to indicate the ESI is manually created. The other 9 octets can be any hexadecimal value from 00 to FF.
Figure 4: ESI Topology for Leaf 1, Leaf 2, and Leaf 3 
Leaf 1:
set interfaces ae11 esi 00:00:00:00:00:00:51:00:00:01set interfaces ae11 esi all-activeset interfaces ae11 aggregated-ether-options lacp activeset interfaces ae11 aggregated-ether-options lacp system-id 00:00:51:00:00:01set interfaces ae11 unit 0 family ethernet-switching interface-mode trunkset interfaces ae11 unit 0 family ethernet-switching vlan members [ 10 20 30 40 ]set interfaces xe-0/0/10 ether-options 802.3ad ae11set interfaces xe-0/0/11 ether-options 802.3ad ae11If your configuration uses MAC-VRF instances, you must also add the configured aggregated Ethernet interface to the MAC-VRF instance:
set routing-instances MAC-VRF-1 interface ae11.0 - Configure VLANs and map them to VNIs. This step enables
the VLANs to participate in VNIs across the EVPN/VXLAN domain.
This step shows the VLAN to VNI mapping either in the default instance or in a MAC-VRF instance configuration.
Leaf 1 (Default Instance):
set vlans VNI_1000 vlan-id 10set vlans VNI_1000 vxlan vni 1000set vlans VNI_2000 vlan-id 20set vlans VNI_2000 vxlan vni 2000set vlans VNI_3000 vlan-id 30set vlans VNI_3000 vxlan vni 3000set vlans VNI_4000 vlan-id 40set vlans VNI_4000 vxlan vni 4000Leaf 1 (MAC-VRF Instance):
The only difference with a MAC-VRF instance configuration is that you configure these statements in the MAC-VRF instance at the [edit routing-instances mac-vrf-instance-name] hierarchy level, as follows:
set routing-instances MAC-VRF-1 vlans VNI_1000 vlan-id 10set routing-instances MAC-VRF-1 vlans VNI_1000 vxlan vni 1000set routing-instances MAC-VRF-1 vlans VNI_2000 vlan-id 20set routing-instances MAC-VRF-1 vlans VNI_2000 vxlan vni 2000set routing-instances MAC-VRF-1 vlans VNI_3000 vlan-id 30set routing-instances MAC-VRF-1 vlans VNI_3000 vxlan vni 3000set routing-instances MAC-VRF-1 vlans VNI_4000 vlan-id 40set routing-instances MAC-VRF-1 vlans VNI_4000 vxlan vni 4000
Verifying the Bridged Overlay on the Leaf Device
Run the following commands to verify that the overlay is working properly on your leaf devices.
The commands here show output for a default instance configuration. With a MAC-VRF instance configuration, you can alternatively use:
show mac-vrf forwarding commands that are aliases for the show ethernet-switching commands in this section.
The show mac-vrf routing instance command, which is an alias for the show evpn instance command in this section.
See MAC-VRF Routing Instance Type Overview for tables of show mac-vrf forwarding and show ethernet-switching command mappings, and show mac-vrf routing command aliases for show evpn commands.
The output with a MAC-VRF instance configuration displays similar information for MAC-VRF routing instances as this section shows for the default instance. One main difference you might see is in the output with MAC-VRF instances on devices where you enable the shared tunnels feature. With shared tunnels enabled, you see VTEP interfaces in the following format:
vtep-index.shared-tunnel-unit
where:
index is the index associated with the MAC-VRF routing instance.
shared-tunnel-unit is the unit number associated with the shared tunnel remote VTEP logical interface.
For example, if a device has a MAC-VRF instance with index 26 and the instance connects to two remote VTEPs, the shared tunnel VTEP logical interfaces might look like this:
vtep-26.32823 vtep-26.32824
If your configuration uses an IPv6 Fabric, you provide IPv6 address parameters where applicable. Output from the commands that display IP addresses reflect the IPv6 device and interface addresses from the underlying fabric. See IPv6 Fabric Underlay and Overlay Network Design and Implementation with EBGP for the fabric parameters reflected in command outputs in this section with an IPv6 Fabric.
- Verify the interfaces are operational. Interfaces xe-0/0/10
and xe-0/0/11 are dual homed to the Ethernet-connected end system
through interface ae11, while interfaces et-0/0/48 through et-0/0/51
are uplinks to the four spine devices.
user@leaf-1> show interfaces terse | match ae.*xe-0/0/10.0 up up aenet --> ae11.0 et-0/0/48.0 up up aenet --> ae1.0 et-0/0/49.0 up up aenet --> ae2.0 et-0/0/50.0 up up aenet --> ae3.0 et-0/0/51.0 up up aenet --> ae4.0 xe-0/0/11.0 up up aenet --> ae11.0 ae1 up up ## To Spine 1 ae1.0 up up inet 172.16.1.1/30 ae2 up up ## To Spine 2 ae2.0 up up inet 172.16.1.5/30 ae3 up up ## To Spine 3 ae3.0 up up inet 172.16.1.9/30 ae4 up up ## To Spine 4 ae4.0 up up inet 172.16.1.13/30 ae11 up up ## To End System ae11.0 up up eth-switch
user@leaf-1> show lacp interfacesAggregated interface: ae1 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity et-0/0/48 Actor No No Yes Yes Yes Yes Fast Active et-0/0/48 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State et-0/0/48 Current Fast periodic Collecting distributing ... Aggregated interface: ae11 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-0/0/10 Actor No No Yes Yes Yes Yes Fast Active xe-0/0/10 Partner No No Yes Yes Yes Yes Fast Active xe-0/0/11 Actor No No Yes Yes Yes Yes Fast Active xe-0/0/11 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-0/0/10 Current Fast periodic Collecting distributing xe-0/0/11 Current Fast periodic Collecting distributinguser@leaf-1> show ethernet-switching interface ae11Routing Instance Name : default-switch Logical Interface flags (DL - disable learning, AD - packet action drop, LH - MAC limit hit, DN - interface down, MMAS - Mac-move action shutdown, AS - Autostate-exclude enabled, SCTL - shutdown by Storm-control, MI - MAC+IP limit hit) Logical Vlan TAG MAC MAC+IP STP Logical Tagging interface members limit limit state interface flags ae11.0 65535 0 tagged VNI_1000 10 65535 0 Forwarding tagged VNI_2000 20 65535 0 Forwarding tagged VNI_3000 30 65535 0 Forwarding tagged VNI_4000 40 65535 0 Forwarding tagged - Verify that the leaf devices have reachability to their
peer leaf devices.
For example, on Leaf 1 with an IPv6 Fabric, view the possible routes to remote Leaf 2 using the show route address command with device IPv6 address 2001:db8::192:168:1:2 for Leaf 2.
user@leaf-1> show route 2001:db8::192:168:1:2inet6.0: 18 destinations, 25 routes (18 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2001:db8::192:168:1:2/128 *[BGP/170] 18:53:41, localpref 100 AS path: 4200000001 4200000012 I, validation-state: unverified > to 2001:db8::173:16:1:1 via ae1.0 [BGP/170] 18:53:41, localpref 100 AS path: 4200000002 4200000012 I, validation-state: unverified > to 2001:db8::173:16:2:1 via ae2.0 :vxlan.inet6.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2001:db8::192:168:1:2/128 *[Static/1] 22:23:04, metric2 0 to 2001:db8::173:16:2:1 via ae2.0 > to 2001:db8::173:16:1:1 via ae1.0 - Verify on Leaf 1 and Leaf 3 that the Ethernet switching
table has installed both the local MAC addresses and the remote MAC
addresses learned through the overlay.
Note To identify end systems learned remotely from the EVPN overlay, look for the MAC address, ESI logical interface, and ESI number. For example, Leaf 1 learns about an end system with the MAC address of 02:0c:10:03:02:02 through esi.1885. This end system has an ESI number of 00:00:00:00:00:00:51:10:00:01. Consequently, this matches the ESI number configured for Leaf 4, 5, and 6 (QFX5110 switches), so we know that this end system is multihomed to these three leaf devices.
user@leaf-1> show ethernet-switching table vlan-id 30MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 10 entries, 10 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source ## Learned locally VNI_3000 02:0c:10:03:02:01 DL ae11.0 ## Learned from the QFX5110 switches - Leaf 4 to 6 VNI_3000 02:0c:10:03:02:02 DR esi.1885 00:00:00:00:00:00:51:10:00:01 ## Learned from the QFX5200 switches - Leaf 7 to 9 VNI_3000 02:0c:10:03:02:03 DR esi.1887 00:00:00:00:00:00:52:00:00:01 ## Learned from the QFX10002 switches - Leaf 10 to 12 VNI_3000 02:0c:10:03:02:04 DR esi.1892 00:00:00:00:00:01:00:00:00:01 ## End System MAC address, connected locally to the leaf device 02:0c:10:03:02:01 ## MAC address learned over the overlay, these end systems are also multihomed 02:0c:10:03:02:02,03,04 - Verify the remote EVPN routes from a specific VNI and
MAC address.
Note The format of the EVPN routes is EVPN-route-type:route-distinguisher:vni:mac-address.
For example, with an IPv4 Fabric, view the remote EVPN routes from VNI 1000 and MAC address 02:0c:10:01:02:02. In this case, the EVPN routes come from Leaf 4 (route distinguisher 192.168.1.4) by way of Spine 1 (192.168.0.1).
user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 1000 evpn-mac-address 02:0c:10:01:02:02bgp.evpn.0: 910 destinations, 3497 routes (904 active, 0 holddown, 24 hidden) + = Active Route, - = Last Active, * = Both 2:192.168.1.4:1::1000::02:0c:10:01:02:02/304 MAC/IP *[BGP/170] 00:11:37, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 1000 evpn-mac-address 02:0c:10:01:02:02 detailbgp.evpn.0: 925 destinations, 3557 routes (919 active, 0 holddown, 24 hidden) 2:192.168.1.4:1::1000::02:0c:10:01:02:02/304 MAC/IP (4 entries, 0 announced) *BGP Preference: 170/-101 Route Distinguisher: 192.168.1.4:1 Next hop type: Indirect, Next hop index: 0 Address: 0xb3a2170 Next-hop reference count: 160 Source: 192.168.0.1 Protocol next hop: 192.168.1.4 Indirect next hop: 0x2 no-forward INH Session ID: 0x0 State: <Active Int Ext> Local AS: 4210000001 Peer AS: 4210000001 Age: 13:42 Metric2: 0 Validation State: unverified Task: BGP_4210000001.192.168.0.1 AS path: I (Originator) Cluster list: 192.168.0.10 Originator ID: 192.168.1.4 Communities: target:62273:268445456 encapsulation:vxlan(0x8) Import Accepted Route Label: 1000 ESI: 00:00:00:00:00:00:51:10:00:01 Localpref: 100 Router ID: 192.168.0.1 Secondary Tables: default-switch.evpn.0 ... ## This output has been abbreviated.Or with an IPv6 Fabric, for example, view the remote EVPN routes from VNI 1000 and MAC address c8:fe:6a:e4:2e:00. In this case, the EVPN routes come from Leaf 2 (route distinguisher 192.168.1.2) by way of Spine 1 (2001:db8::192:168:0:1).
user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 1000 evpn-mac-address c8:fe:6a:e4:2e:00bgp.evpn.0: 43916 destinations, 76851 routes (43916 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2:192.168.1.2:1::1000::c8:fe:6a:e4:2e:00/304 MAC/IP *[BGP/170] 19:46:55, localpref 100, from 2001:db8::192:168:0:2 AS path: 4200000001 4200000012 I, validation-state: unverified > to 2001:db8::173:16:1:1 via ae1.0 [BGP/170] 23:16:47, localpref 100, from 2001:db8::192:168:0:1 AS path: 4200000002 4200000012 I, validation-state: unverified > to 2001:db8::173:16:2:1 via ae2.0 2:192.168.1.2:1::1000::c8:fe:6a:e4:2e:00::77.5.241.242/304 MAC/IP *[BGP/170] 19:46:55, localpref 100, from 2001:db8::192:168:0:2 AS path: 4200000001 4200000012 I, validation-state: unverified > to 2001:db8::173:16:1:1 via ae1.0 [BGP/170] 23:16:47, localpref 100, from 2001:db8::192:168:0:1 AS path: 4200000002 4200000012 I, validation-state: unverified > to 2001:db8::173:16:2:1 via ae2.0 2:192.168.1.2:1::1000::c8:fe:6a:e4:2e:00::2001:db8::77:0:5f1:242/304 MAC/IP *[BGP/170] 19:46:55, localpref 100, from 2001:db8::192:168:0:2 AS path: 4200000001 4200000012 I, validation-state: unverified > to 2001:db8::173:16:1:1 via ae1.0 [BGP/170] 23:16:47, localpref 100, from 2001:db8::192:168:0:1 AS path: 4200000002 4200000012 I, validation-state: unverified > to 2001:db8::173:16:2:1 via ae2.0 2:192.168.1.2:1::1000::c8:fe:6a:e4:2e:00::fe80::cafe:6a05:f1e4:2e00/304 MAC/IP *[BGP/170] 19:46:55, localpref 100, from 2001:db8::192:168:0:2 AS path: 4200000001 4200000012 I, validation-state: unverified > to 2001:db8::173:16:1:1 via ae1.0 [BGP/170] 23:16:47, localpref 100, from 2001:db8::192:168:0:1 AS path: 4200000002 4200000012 I, validation-state: unverified > to 2001:db8::173:16:2:1 via ae2.0user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 1000 evpn-mac-address c8:fe:6a:e4:2e:00 detailbgp.evpn.0: 43916 destinations, 76851 routes (43916 active, 0 holddown, 0 hidden) 2:192.168.1.2:1::1000::c8:fe:6a:e4:2e:00/304 MAC/IP (2 entries, 0 announced) *BGP Preference: 170/-101 Route Distinguisher: 192.168.0.2:1 Next hop type: Indirect, Next hop index: 0 Address: 0x8c0701c Next-hop reference count: 41840 Source: 2001:db8::192:168:0:1 Protocol next hop: 2001:db8::192:168:1:2 Indirect next hop: 0x2 no-forward INH Session ID: 0 State: <Active Ext> Peer AS: 4200000001 Age: 23:16:11 Metric2: 0 Validation State: unverified Task: BGP_4200000001_42000000.2001:db8::192:168:0:1 AS path: 4200000001 4200000012 I Communities: target:2:1000 encapsulation:vxlan(0x8) mac-mobility:0x1:sticky (sequence 0) Import Accepted Route Label: 1000 ESI: 00:00:00:00:00:00:00:00:00:00 Localpref: 100 Router ID: 192.168.0.1 Secondary Tables: MAC-VRF-1.evpn.0 Thread: junos-main Indirect next hops: 1 Protocol next hop: 2001:db8::192:168:1:2 Indirect next hop: 0x2 no-forward INH Session ID: 0 Indirect path forwarding next hops: 2 Next hop type: Router Next hop: 2001:db8::173:16:1:1 via ae1.0 Session Id: 0 Next hop: 2001:db8::173:16:2:1 via ae2.0 Session Id: 0 2001:db8::192:168:1:2/128 Originating RIB: inet6.0 Node path count: 1 Forwarding nexthops: 2 Next hop type: Router Next hop: 2001:db8::173:16:1:1 via ae1.0 Session Id: 0 Next hop: 2001:db8::173:16:2:1 via ae2.0 Session Id: 0 - Verify the source and destination address of each VTEP
interface and view their status. Use the show ethernet-switching
vxlan-tunnel-end-point source and show interfaces vtep commands.
Note A scaled-out reference design can have 96 leaf devices, which corresponds to 96 VTEP interfaces - one VTEP interface per leaf device. The output here is truncated for readability.
The following example shows these commands with an IPv4 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point sourceLogical System Name Id SVTEP-IP IFL L3-Idx <default> 0 192.168.1.1 lo0.0 0 L2-RTT Bridge Domain VNID MC-Group-IP default-switch VNI_1000+10 1000 0.0.0.0 default-switch VNI_2000+20 2000 0.0.0.0 default-switch VNI_3000+30 3000 0.0.0.0 default-switch VNI_4000+40 4000 0.0.0.0user@leaf-1> show interfaces terse vtepInterface Admin Link Proto Local Remote vtep up up vtep.32768 up up vtep.32769 up up eth-switch vtep.32770 up up eth-switch vtep.32771 up up eth-switch vtep.32772 up up eth-switch ... vtep.32869 up up eth-switch
user@leaf-1> show interfaces vtepPhysical interface: vtep, Enabled, Physical link is Up Interface index: 646, SNMP ifIndex: 503 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vtep.32768 (Index 554) (SNMP ifIndex 648) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 192.168.1.1, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 0 ... Logical interface vtep.32814 (Index 613) (SNMP ifIndex 903) Flags: Up SNMP-Traps Encapsulation: ENET2 VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 192.168.1.96, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 6364 Protocol eth-switch, MTU: Unlimited Flags: Trunk-ModeOr the following example shows these commands with an IPv6 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point sourceLogical System Name Id SVTEP-IP IFL L3-Idx SVTEP-Mode ELP-SVTEP-IP <default> 0 2001:db8::192:168:1:1 lo0.0 0 L2-RTT Bridge Domain VNID Translation-VNID MC-Group-IP MAC-VRF-1 VNI_1000+10 1000 :: MAC-VRF-1 VNI_2000+20 2000 :: MAC-VRF-1 VNI_3000+30 3000 :: MAC-VRF-1 VNI_4000+40 4000 ::user@leaf-1> show interfaces vtepPhysical interface: vtep, Enabled, Physical link is Up Interface index: 650, SNMP ifIndex: 506 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vtep.32768 (Index 2031) (SNMP ifIndex 1737) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 2001:db8::192:168:1:1, L2 Routing Instance: MAC-VRF-1, L3 Routing Instance: default Input packets : 0 Output packets: 0 Logical interface vtep.32775 (Index 2038) (SNMP ifIndex 1744) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 2001:db8::192:168:1:1, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 0 Logical interface vtep.32776 (Index 829) (SNMP ifIndex 1775) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 2001:db8::192:168:1:1, L3 Routing Instance: default Input packets : 0 Output packets: 0 Logical interface vtep.32777 (Index 830) (SNMP ifIndex 1776) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Shared Remote, VXLAN Endpoint Address: 2001:db8::192:168:1:2, L3 Routing Instance: default Input packets : 3873955 Output packets: 12766 Protocol eth-switch, MTU: Unlimited Flags: Is-Primary, Trunk-Mode - Verify that each VNI maps to the associated VXLAN tunnel.
For example, with an IPv4 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remote0 192.168.1.1 lo0.0 0 RVTEP-IP IFL-Idx NH-Id 192.168.1.2 586 1782 VNID MC-Group-IP 2000 0.0.0.0 4000 0.0.0.0 1000 0.0.0.0 3000 0.0.0.0 ... RVTEP-IP IFL-Idx NH-Id 192.168.1.96 613 1820 VNID MC-Group-IP 1000 0.0.0.0 2000 0.0.0.0 3000 0.0.0.0 4000 0.0.0.0Or for example, with an IPv6 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remoteLogical System Name Id SVTEP-IP IFL L3-Idx SVTEP-Mode ELP-SVTEP-IP <default> 0 2001:db8::192:168:1:1 lo0.0 0 RVTEP-IP IFL-Idx Interface NH-Id RVTEP-Mode ELP-IP Flags 2001:db8::192:168:1:2 830 vtep.32777 22929 RNVE RVTEP-IP L2-RTT IFL-Idx Interface NH-Id RVTEP-Mode ELP-IP Flags 2001:db8::192:168:1:2 MAC-VRF-1 675430409 vtep-532.32777 22929 RNVE VNID MC-Group-IP 1000 :: 2000 :: 3000 :: 4000 :: - Verify that MAC addresses are learned through the VXLAN
tunnels.
For example, with an IPv4 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remote mac-tableMAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC) Logical system : <default> Routing instance : default-switch Bridging domain : VNI_1000+10, VLAN : 10, VNID : 1000 MAC MAC Logical Remote VTEP address flags interface IP address 02:0c:10:01:02:04 DR esi.1764 192.168.1.11 192.168.1.12 192.168.1.10 02:0c:10:01:02:02 DR esi.1771 192.168.1.6 192.168.1.4 02:0c:10:01:02:03 DR esi.1774 192.168.1.7 ... 0e:ad:10:01:00:60 D vtep.32784 192.168.1.84 0e:ad:10:01:00:30 D vtep.32785 192.168.1.36 0e:ad:10:01:00:48 D vtep.32786 192.168.1.60 0e:ad:10:01:00:4e D vtep.32788 192.168.1.66 0e:ad:10:01:00:4c D vtep.32789 192.168.1.64 0e:ad:10:01:00:36 D vtep.32790 192.168.1.42 ... ---(more)---Or for example, with an IPv6 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remote mac-tableMAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC) Logical system : <default> Routing instance : MAC-VRF-1 Bridging domain : VNI_1000+10, VLAN : 10, VNID : 1000 MAC MAC Logical Remote VTEP address flags interface IP address 00:00:5e:00:00:04 DRP vtep-532.32777 2001:db8::192:168:1:2 00:00:00:00:09:80 S,NM vtep-532.32777 2001:db8::192:168:1:2 c8:fe:6a:e4:2e:00 DRP vtep-532.32777 2001:db8::192:168:1:2 - Verify multihoming information of the gateway and the
aggregated Ethernet interfaces.
For example, with an IPv4 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point esi## Local AE link – QFX5100 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:51:00:00:01 default-switch 1768 131078 esi.1768 ae11.0, 2 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.2 vtep.32780 1782 1 2 192.168.1.3 vtep.32772 1767 0 2 ## Remote AE Link for QFX5110 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:51:10:00:01 default-switch 1771 131081 esi.1771 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.6 vtep.32771 1766 2 2 192.168.1.4 vtep.32770 1765 1 2 192.168.1.5 vtep.32774 1770 0 2 ## Remote AE Link for QFX5200 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:52:00:00:01 default-switch 1774 131084 esi.1774 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.9 vtep.32778 1776 2 2 192.168.1.8 vtep.32777 1775 1 2 192.168.1.7 vtep.32776 1773 0 2 ## Remote AE Link for QFX10002 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:01:00:00:00:01 default-switch 1764 131074 esi.1764 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.11 vtep.32775 1772 2 2 192.168.1.12 vtep.32773 1769 1 2 192.168.1.10 vtep.32769 1759 0 2 ...Or for example, with an IPv6 Fabric:
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point esiESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:ff:00:01:00:03:00:05 MAC-VRF-1 36499 525338 esi.36499 ae5.0, 1 Aliasing RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS MAC-COUNT 2001:db8::191:168:1:2 vtep-532.32777 22929 0 2 0 - Verify that the VXLAN tunnel from one leaf to another
leaf is load balanced with equal cost multipathing (ECMP) over the
underlay.
user@leaf-1> show route forwarding-table table default-switch extensive | find vtep.32770Destination: vtep.32770 Route type: interface Route reference: 0 Route interface-index: 576 Multicast RPF nh index: 0 P2mpidx: 0 Flags: sent to PFE Nexthop: Next-hop type: composite Index: 1765 Reference: 12 Next-hop type: indirect Index: 131076 Reference: 3 Next-hop type: unilist Index: 131193 Reference: 238 Nexthop: 172.16.1.2 Next-hop type: unicast Index: 1791 Reference: 10 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.1.6 Next-hop type: unicast Index: 1794 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.1.10 Next-hop type: unicast Index: 1758 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.1.14 Next-hop type: unicast Index: 1795 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0
- Verify that remote MAC addresses are reachable through
ECMP.
For example, with an IPv4 Fabric:
user@leaf-1> show route forwarding-table table default-switch extensive destination 02:0c:10:01:02:03/48Routing table: default-switch.evpn-vxlan [Index 4] Bridging domain: VNI_1000.evpn-vxlan [Index 3] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 02:0c:10:01:02:03/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 582 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 169 Epoch: 0 Sequence Number: 0 Learn Mask: 0x400000000000000003000000000000000000000 L2 Flags: control_dyn Flags: sent to PFE Nexthop: Next-hop type: composite Index: 1773 Reference: 12 Next-hop type: indirect Index: 131085 Reference: 3 Next-hop type: unilist Index: 131193 Reference: 238 Nexthop: 172.16.1.2 Next-hop type: unicast Index: 1791 Reference: 10 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.1.6 Next-hop type: unicast Index: 1794 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.1.10 Next-hop type: unicast Index: 1758 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.1.14 Next-hop type: unicast Index: 1795 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0Note Though the MAC address is reachable over multiple VTEP interfaces, QFX5100, QFX5110, QFX5120-32C, and QFX5200 switches do not support ECMP across the overlay because of a merchant ASIC limitation. Only the QFX10000 line of switches contain a custom Juniper Networks ASIC that supports ECMP across both the overlay and the underlay.
user@leaf-1> show ethernet-switching table vlan-id 10 | match 02:0c:10:01:02:03VNI_1000 02:0c:10:01:02:03 DR esi.1774 00:00:00:00:00:00:52:00:00:01
user@leaf-1> show route forwarding-table table default-switch extensive destination 02:0c:10:01:02:03/48Routing table: default-switch.evpn-vxlan [Index 9] Bridging domain: VNI_1000.evpn-vxlan [Index 3] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 02:0c:10:01:02:03/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 550 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 0 Epoch: 0 Sequence Number: 0 Learn Mask: 0x400000000000000001000000000000000000000 L2 Flags: control_dyn, esi Flags: sent to PFE Next-hop type: indirect Index: 2097173 Reference: 5 Nexthop: Next-hop type: composite Index: 1947 Reference: 2 Nexthop: Next-hop type: composite Index: 1948 Reference: 8 Next-hop type: indirect Index: 2097174 Reference: 3 Next-hop type: unilist Index: 2097280 Reference: 241 Nexthop: 172.16.10.2 Next-hop type: unicast Index: 1950 Reference: 11 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.10.6 Next-hop type: unicast Index: 1956 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.10.10 Next-hop type: unicast Index: 1861 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.10.14 Next-hop type: unicast Index: 1960 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0Or for example, with an IPv6 Fabric:
user@leaf-1> show route forwarding-table table MAC-VRF-1 extensive | find c8:fe:6a:e4:2e:00Destination: c8:fe:6a:e4:2e:00/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 1641 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 0 Epoch: 0 Sequence Number: 0 Learn Mask: 0x4000000000000000000000000000000000000000 L2 Flags: control_dyn Flags: sent to PFE Nexthop: Next-hop type: composite Index: 22929 Reference: 8358 Next-hop type: indirect Index: 524287 Reference: 3 Next-hop type: unilist Index: 524286 Reference: 531 Nexthop: 2001:db8::173:16:1:1 Next-hop type: unicast Index: 22928 Reference: 6 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 2001:db8::173:16:2:1 Next-hop type: unicast Index: 12290 Reference: 6 Next-hop interface: ae2.0 Weight: 0x0
- Verify which device is the Designated Forwarder (DF) for
broadcast, unknown, and multicast (BUM) traffic coming from the VTEP
tunnel.
For example, with an IPv4 Fabric:
Note Because the DF IP address is listed as 192.168.1.2, Leaf 2 is the DF.
user@leaf-1> show evpn instance esi 00:00:00:00:00:00:51:00:00:01 designated-forwarderInstance: default-switch Number of ethernet segments: 12 ESI: 00:00:00:00:00:00:51:00:00:01 Designated forwarder: 192.168.1.2Or, for example, with an IPv4 Fabric:
Note Because the DF IPv6 address is listed as 2001:db8::192:168:1:1, Leaf 1 is the DF.
user@leaf-1> show evpn instance esi 00:00:00:ff:00:01:00:03:00:05 designated-forwarderInstance: MAC-VRF-1 Number of ethernet segments: 2 ESI: 00:00:00:ff:00:01:00:03:00:05 Designated forwarder: 2001:db8::192:168:1:1
See also
Bridged Overlay — Release History
Table 1 provides a history of all of the features in this section and their support within this reference design.
Table 1: Bridged Overlay in the Cloud Data Center Reference Design– Release History
Release | Description |
|---|---|
19.1R2 | QFX10002-60C and QFX5120-32C switches running Junos OS Release 19.1R2 and later releases in the same release train support all features documented in this section. |
18.4R2 | QFX5120-48Y switches running Junos OS Release 18.4R2 and later releases in the same release train support all features documented in this section. |
18.1R3-S3 | QFX5110 switches running Junos OS Release 18.1R3-S3 and later releases in the same release train support all features documented in this section. |
17.3R3-S2 | All devices in the reference design that support Junos OS Release 17.3R3-S2 and later releases in the same release train also support all features documented in this section. |