Centrally-Routed Bridging Overlay Design and Implementation
A centrally-routed bridging overlay performs routing at a central location in the EVPN network as shown in Figure 1, In this example, IRB interfaces are configured in the overlay at each spine device to route traffic between the VLANs that originate at the leaf devices and end systems. For an overview of centrally-routed bridging overlays, see the Centrally-Routed Bridging Overlay section in Data Center Fabric Blueprint Architecture Components.
The following sections provide the detailed steps of how to implement a centrally-routed bridging overlay:
Configuring a VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance
This basic form of overlay is supported on all platforms included in this reference design. It uses the simplest VLAN-aware method to enable a single, default switching instance that supports up to 4094 VLANs.
As shown in Figure 2, you configure VLANs at the leaf devices, and IRB interfaces for routing at the spine devices. Such configuration is placed in the default switching instance at the [edit vlans], [edit interfaces], [edit protocols evpn], and [edit switch-options] hierarchy levels. Routing instances are not required for this overlay style, but can be implemented as an option depending on the needs of your network.
When you implement this style of overlay on a spine device, you:
Configure IRB interfaces to route traffic between Ethernet virtual network instances.
Set virtual gateway addresses.
Add VXLAN features to optimize traffic paths.
Configure EVPN with VXLAN encapsulation in the default switching instance or in a routing instance.
Set the loopback interface as the VTEP source interface.
Configure route distinguishers and route targets to direct traffic to peers.
Map VLANs to VNIs.
When you implement this style of overlay on a leaf device, you:
Configure Ethernet Segment Identifier (ESI) settings.
Enable EVPN with VXLAN encapsulation in the default switching instance.
Establish route targets and route distinguishers.
Map VLANs to VNIs.
For an overview of VLAN-aware centrally-routed bridging overlays, see the Centrally-Routed Bridging Overlay section in Data Center Fabric Blueprint Architecture Components.
If you need to implement more than 4094 VLANs, you can use a centrally-routed bridging overlay with virtual switches (available on switches in the QFX10000 line) or MAC-VRF instances. See Configuring a VLAN-Aware Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances. With MAC-VRF instances, you expand your options to either isolate traffic between tenant systems or to enable routing and forwarding between tenant systems.
The following sections provide the detailed steps of how to configure and verify the VLAN-aware centrally-routed bridging overlay in the default switching instance:
Configuring a VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance on the Spine Device
To configure a VLAN-aware centrally-routed bridging overlay in the default switching instance on a spine device, perform the following:
The following example shows the configuration for Spine 1, as shown in Figure 3.
- Ensure the IP fabric underlay is in place. To configure an IP fabric on a spine device, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your spine devices, see Configure IBGP for the Overlay.
- Configure the VTEP tunnel endpoint as the loopback address,
and add a route distinguisher and a route target (target:64512:1111).
Also, keep your configuration simple by using the auto route target
option, which uses one target for both import and export.
Spine 1:
set switch-options vtep-source-interface lo0.0set switch-options route-distinguisher 192.168.0.1:1set switch-options vrf-target target:64512:1111set switch-options vrf-target auto - Configure IRB interfaces for each VNI and the corresponding
virtual gateway address (which uses .254 in the 4th octet for each
prefix). Include VXLAN features, such as proxy-macip-advertisement and virtual-gateway-accept-data, to improve performance
and manageability.
Note The proxy-macip-advertisement statement allows MAC address plus IP address information (ARP entries) learned locally for a subnet to be sent by one central gateway (spine device) to the other central gateways. This is referred to as ARP synchronization. We recommend you enable this feature in a centrally-routed bridging overlay fabric when any of the leaf devices in the fabric advertise only the MAC address of the connected hosts in their EVPN Type 2 route advertisements. If all of the leaf devices in the fabric can advertise both MAC and IP addresses of hosts in Type 2 advertisements, this setting is optional. This feature improves convergence times and traffic handling in the EVPN/VXLAN network.
You must configure both the virtual-gateway-accept-data statement and the preferred IPv4 and IPv6 addresses to use the ping operation and verify connectivity to the virtual gateway IP address from the end system.
Spine 1:
set interfaces irb unit 100 family inet address 10.1.0.1/24 virtual-gateway-address 10.1.0.254set interfaces irb unit 100 family inet address 10.1.0.1/24 preferredset interfaces irb unit 100 proxy-macip-advertisementset interfaces irb unit 100 virtual-gateway-accept-dataset interfaces irb unit 100 family inet6 address 2001:db8::10:1:0:1/112 virtual-gateway-address 2001:db8::10:1:0:254set interfaces irb unit 100 family inet6 address fe80::10:1:0:254/112set interfaces irb unit 200 family inet address 10.1.1.1/24 virtual-gateway-address 10.1.1.254set interfaces irb unit 200 family inet address 10.1.1.1/24 preferredset interfaces irb unit 200 proxy-macip-advertisementset interfaces irb unit 200 virtual-gateway-accept-dataset interfaces irb unit 200 family inet6 address 2001:db8::10:1:1:1/112 virtual-gateway-address 2001:db8::10:1:1:254set interfaces irb unit 200 family inet6 address fe80::10:1:1:254/112set interfaces irb unit 300 family inet address 10.1.2.1/24 virtual-gateway-address 10.1.2.254set interfaces irb unit 300 family inet address 10.1.2.1/24 preferredset interfaces irb unit 300 proxy-macip-advertisementset interfaces irb unit 300 virtual-gateway-accept-dataset interfaces irb unit 300 family inet6 address 2001:db8::10:1:2:1/112 virtual-gateway-address 2001:db8::10:1:2:254set interfaces irb unit 300 family inet6 address fe80::10:1:2:254/112set interfaces irb unit 400 family inet address 10.1.3.1/24 virtual-gateway-address 10.1.3.254set interfaces irb unit 400 family inet address 10.1.3.1/24 preferredset interfaces irb unit 400 proxy-macip-advertisementset interfaces irb unit 400 virtual-gateway-accept-dataset interfaces irb unit 400 family inet6 address 2001:db8::10:1:3:1/112 virtual-gateway-address 2001:db8::10:1:3:254set interfaces irb unit 400 family inet6 address fe80::10:1:3:254/112 - Configure a secondary logical unit on the loopback interface
for the default switching instance.
Spine 1:
set interfaces lo0 unit 1 family inet address 192.168.0.101/32 - Configure EVPN with VXLAN encapsulation. Include the no-gateway-community option to advertise the virtual gateway
and IRB MAC addresses to the EVPN peer devices so that Ethernet-only
PE devices can learn these MAC addresses.
Spine 1:
set protocols evpn encapsulation vxlanset protocols evpn default-gateway no-gateway-communityset protocols evpn extended-vni-list all - Configure mapping between VLANs and VXLAN VNIs.
Spine 1:
set vlans VNI_10000 vlan-id 100set vlans VNI_10000 l3-interface irb.100set vlans VNI_10000 vxlan vni 10000set vlans VNI_20000 vlan-id 200set vlans VNI_20000 l3-interface irb.200set vlans VNI_20000 vxlan vni 20000set vlans VNI_30000 vlan-id 300set vlans VNI_30000 l3-interface irb.300set vlans VNI_30000 vxlan vni 30000set vlans VNI_40000 vlan-id 400set vlans VNI_40000 l3-interface irb.400set vlans VNI_40000 vxlan vni 40000 - Configure a routing instance named VRF 1, and map IRB
interfaces irb.100 (VNI 10000) and irb.200 (VNI 20000) to this instance.
Note Because the irb.300 (VNI 30000) and irb.400 (VNI 40000) interfaces are not configured inside a routing instance, they are part of the default switching instance for the spine devices. The end result of your configuration should match the diagram shown in Figure 3.
Spine 1:
set routing-instances VRF_1 instance-type vrfset routing-instances VRF_1 interface irb.100set routing-instances VRF_1 interface irb.200set routing-instances VRF_1 interface lo0.1set routing-instances VRF_1 route-distinguisher 192.168.0.1:100set routing-instances VRF_1 vrf-target target:62273:10000
Verifying the VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance for the Spine Device
Issue the following commands to verify that the overlay is working properly on your spine devices:
- Verify the IRB interfaces are operational for both IPv4
and IPv6.
user@spine-1> show interfaces terse irbInterface Admin Link Proto Local Remote irb up up irb.100 up up inet 10.1.0.1/24 inet6 2001:db8::10:1:0:1/112 fe80::10:1:0:254/112 irb.200 up up inet 10.1.1.1/24 inet6 2001:db8::10:1:1:1/112 fe80::10:1:1:254/112 irb.300 up up inet 10.1.2.1/24 inet6 2001:db8::10:1:2:1/112 fe80::10:1:2:254/112 irb.400 up up inet 10.1.3.1/24 inet6 2001:db8::10:1:3:1/112 fe80::10:1:3:254/112 - Verify that the VTEP interfaces are up.
user@spine-1> show interfaces terse vtepInterface Admin Link Proto Local Remote vtep up up vtep.32768 up up vtep.32769 up up eth-switch vtep.32770 up up eth-switch vtep.32771 up up eth-switch vtep.32772 up up eth-switch ... vtep.32804 up up eth-switch ---(more)---
user@spine-1> show interfaces terse vtep | match eth-switch | countCount: 109 lines
- Verify the endpoint destination IP address for the VTEP
interfaces. The spine devices display their VTEPs as loopback addresses
in the range of 192.168.0.x (1 - 4) and the leaf
devices display their VTEPs as loopback addresses in the range of
192.168.1.x (1 - 96).
user@spine-1> show interfaces vtepPhysical interface: vtep, Enabled, Physical link is Up Interface index: 240, SNMP ifIndex: 504 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vtep.32768 (Index 670) (SNMP ifIndex 505) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 192.168.0.1, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 0 ... Logical interface vtep.32771 (Index 802) (SNMP ifIndex 536) Flags: Up SNMP-Traps Encapsulation: ENET2 VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 192.168.1.4, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 1979 Output packets: 9867 Protocol eth-switch, MTU: Unlimited ---(more)--- - Verify that the spine device has all the routes to the
leaf devices.
user@spine-2> show route 192.168.1.1inet.0: 446 destinations, 19761 routes (446 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 192.168.1.1/32 *[BGP/170] 00:06:29, localpref 100 AS path: 4200000011 I, validation-state: unverified > to 172.16.1.5 via ae1.0 [BGP/170] 00:06:22, localpref 100 AS path: 4200000023 4200000004 4200000011 I, validation-state: unverified > to 172.16.13.5 via ae13.0 ... [BGP/170] 00:06:18, localpref 100 AS path: 4200000032 4200000004 4200000011 I, validation-state: unverified > to 172.16.22.5 via ae22.0 ---(more)--- - Verify that each end system resolves the virtual gateway
MAC address for a subnet using the gateway IRB address on the central
gateways (spine devices).
user@spine-1> show arp no-resolve vpn VRF_1MAC Address Address Interface Flags 06:4b:8c:cd:13:f8 10.1.0.2 irb.100 [vtep.32796] none ## Spine 2 IRB interface 06:4b:8c:cd:c4:38 10.1.0.3 irb.100 [vtep.32878] none ## Spine 3 IRB interface 06:38:e1:6f:30:29 10.1.0.4 irb.100 [vtep.32821] none ## Spine 4 IRB interface02:0c:10:01:02:01 10.1.0.201 irb.100 [.local..11] none ## End system behind the QFX5100s 02:0c:10:01:02:02 10.1.0.202 irb.100 [.local..11] none ## End system behind the QFX5110s 02:0c:10:01:02:03 10.1.0.203 irb.100 [.local..11] none ## End system behind the QFX5200s 02:0c:10:01:02:04 10.1.0.204 irb.100 [.local..11] none ## End system behind the QFX10002s 00:00:5e:00:01:01 10.1.0.254 irb.100 permanent published gateway ## Virtual gateway IP and MAC address 06:4b:8c:cd:13:f8 10.1.1.2 irb.200 [vtep.32796] none 06:4b:8c:cd:c4:38 10.1.1.3 irb.200 [vtep.32878] none 06:38:e1:6f:30:29 10.1.1.4 irb.200 [vtep.32821] none 0e:ad:10:02:00:01 10.1.1.101 irb.200 [vtep.32776] noneuser@spine-1> show ipv6 neighborsIPv6 Address Linklayer Address State Exp Rtr Secure Interface 2001:db8::10:1:0:2 06:4b:8c:cd:13:f8 stale 325 no no irb.100 [vtep.32796] 2001:db8::10:1:0:3 06:4b:8c:cd:c4:38 stale 514 yes no irb.100 [vtep.32878] 2001:db8::10:1:0:4 06:38:e1:6f:30:29 stale 326 no no irb.100 [vtep.32821] 2001:db8::10:1:0:201 02:0c:10:01:02:01 stale 1114 no no irb.100 [.local..11] 2001:db8::10:1:0:202 02:0c:10:01:02:02 stale 443 no no irb.100 [.local..11] 2001:db8::10:1:0:203 02:0c:10:01:02:03 stale 853 no no irb.100 [.local..11] 2001:db8::10:1:0:204 02:0c:10:01:02:04 stale 1181 no no irb.100 [.local..11] 2001:db8::10:1:0:254 00:00:5e:00:02:01 reachable 0 no no irb.100 2001:db8::10:1:1:2 06:4b:8c:cd:13:f8 stale 325 no no irb.200 [vtep.32796] 2001:db8::10:1:1:3 06:4b:8c:cd:c4:38 stale 514 yes no irb.200 [vtep.32878] 2001:db8::10:1:1:4 06:38:e1:6f:30:29 stale 326 no no irb.200 [vtep.32821] 2001:db8::10:1:1:201 02:0c:10:02:02:01 stale 1121 no no irb.200 [.local..11] 2001:db8::10:1:1:202 02:0c:10:02:02:02 stale 423 no no irb.200 [.local..11] 2001:db8::10:1:1:203 02:0c:10:02:02:03 stale 1081 no no irb.200 [.local..11] 2001:db8::10:1:1:204 02:0c:10:02:02:04 stale 1167 no no irb.200 [.local..11]
- Verify the switching table for VNI 10000 to see entries
for end systems and the other spine devices.
user@spine-1> show ethernet-switching table vlan-id 100MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 105 entries, 105 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source VNI_10000 00:00:5e:00:01:01 DR esi.2453 05:19:17:f3:41:00:00:27:10:00## Entries for the spine devices VNI_10000 06:4b:8c:cd:13:f8 D vtep.32796 192.168.0.2 VNI_10000 06:4b:8c:cd:c4:38 D vtep.32878 192.168.0.3 VNI_10000 06:38:e1:6f:30:29 D vtep.32821 192.168.0.4 ## The next four MAC addresses belong to the end systems connected to Leaf 1 - 3 (QFX5100), Leaf 4-6 (QFX5110), Leaf 7-9 (QFX5200), and Leaf 10-12 (QFX10002). VNI_10000 02:0c:10:01:02:01 DR esi.2443 00:00:00:00:00:00:51:00:00:01 VNI_10000 02:0c:10:01:02:02 DR esi.2497 00:00:00:00:00:00:51:10:00:01 VNI_10000 02:0c:10:01:02:03 DR esi.2427 00:00:00:00:00:00:52:00:00:01 VNI_10000 02:0c:10:01:02:04 DR esi.2610 00:00:00:00:00:01:00:00:00:01 ... VNI_10000 0e:ad:10:01:00:02 D vtep.32814 192.168.1.96
- Verify MAC address and ARP information learned from the
leaf devices over the control plane.
user@spine-1> show evpn database mac-address 02:0c:10:01:02:01 extensiveInstance: default-switch VN Identifier: 10000, MAC address:: 02:0c:10:01:02:01 Source: 00:00:00:00:00:00:51:00:00:01, Rank: 1, Status: Active Remote origin: 192.168.1.2 ## Leaf 2 and Leaf 3 advertised this route Remote origin: 192.168.1.3 Timestamp: Jul 13 23:35:37 (0x59686639) State: <Remote-To-Local-Adv-Done> IP address: 10.1.0.201 ## MAC Address + IP Flags: <Proxy> Remote origin: 192.168.1.2 Remote origin: 192.168.1.3 IP address: 2001:db8::10:1:0:201 ## MAC Address + IPv6 Remote origin: 192.168.1.2 Remote origin: 192.168.1.3 History db: Time Event Jul 13 23:35:38 2017 Applying remote state to peer 192.168.1.2 Jul 13 23:35:38 2017 Remote peer 192.168.1.2 updated Jul 13 23:35:38 2017 MAC+IP not updated, source l2ald is not owner (type2) Jul 13 23:35:38 2017 Updated Jul 13 23:35:38 2017 No change to MAC state Jul 13 23:35:38 2017 Applying remote state to peer 192.168.1.3 Jul 13 23:35:38 2017 Remote peer 192.168.1.3 updated Jul 13 23:35:38 2017 MAC+IP not updated, source l2ald is not owner (type2) Jul 13 23:35:38 2017 Updated Jul 13 23:35:38 2017 No change to MAC state - Verify the remote VXLAN tunnel end points.
user@spine-1> show ethernet-switching vxlan-tunnel-end-point remoteLogical System Name Id SVTEP-IP IFL L3-Idx <default> 0 192.168.0.1 lo0.0 0 RVTEP-IP IFL-Idx NH-Id 192.168.1.1 827 2444 VNID MC-Group-IP 10000 0.0.0.0 20000 0.0.0.0 30000 0.0.0.0 40000 0.0.0.0 RVTEP-IP IFL-Idx NH-Id ... RVTEP-IP IFL-Idx NH-Id 192.168.1.96 812 2428 VNID MC-Group-IP 10000 0.0.0.0 20000 0.0.0.0 30000 0.0.0.0 40000 0.0.0.0 - Verify that MAC addresses are learned through the VXLAN
tunnel.
user@spine-1> show ethernet-switching vxlan-tunnel-end-point remote mac-tableMAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC) Logical system : <default> Routing instance : default-switch Bridging domain : VNI_10000+100, VLAN : 100, VNID : 10000 MAC MAC Logical Remote VTEP address flags interface IP address 02:0c:10:01:02:03 DR esi.2427 192.168.1.8 192.168.1.7 192.168.1.9 02:0c:10:01:02:01 DR esi.2443 192.168.1.2 192.168.1.3 ## This next entry shows that the virtual gateway MAC address of 00:00:5e:00:01:01 has been learned by the other spine devices. 00:00:5e:00:01:01 DR esi.2453 192.168.0.3 192.168.0.4 192.168.0.2 02:0c:10:01:02:02 DR esi.2497 192.168.1.6 192.168.1.4 02:0c:10:01:02:04 DR esi.2610 192.168.1.12 192.168.1.10 192.168.1.11 06:4b:8c:cd:13:f8 D vtep.32796 192.168.0.2 ---(more)---
Configuring a VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance on the Leaf Device
To configure a VLAN-aware centrally-routed bridging overlay in the default switching instance on a leaf device, perform the following:
The following example shows the configuration for Leaf 1, as shown in Figure 4.
- Ensure the IP fabric underlay is in place. To configure an IP fabric on a leaf device, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your leaf device, see Configure IBGP for the Overlay.
- Configure the EVPN protocol with VXLAN encapsulation,
and specify the VTEP source interface (in this case, the loopback
interface of the leaf device).
Leaf 1:
set protocols evpn encapsulation vxlanset protocols evpn extended-vni-list allset switch-options vtep-source-interface lo0.0 - Define an EVPN route target and route distinguisher, and
use the auto option to derive route targets automatically.Setting
these parameters specifies how the routes are imported and exported.
The import and export of routes from a routing or bridging table is
the basis for dynamic overlays. In this case, members of the global
BGP community with a route target of target:64512:1111 participate
in the exchange of EVPN/VXLAN information.
Leaf 1:
set switch-options route-distinguisher 192.168.1.1:1set switch-options vrf-target target:64512:1111set switch-options vrf-target auto - Configure ESI settings on all similar leaf devices. Because
the end systems in this reference design are multihomed to three leaf
devices per device type cluster (such as QFX5100), you must configure
the same ESI identifier and LACP system identifier on all three leaf
devices for each unique end system. Unlike other topologies where
you would configure a different LACP system identifier per leaf device
and have VXLAN select a single designated forwarder, use the same
LACP system identifier to allow the 3 leaf devices to appear as a
single LAG to a multihomed end system.
In addition, use the same aggregated Ethernet interface number for
all ports included in the ESI.
The configuration for Leaf 1 is shown below, but you must replicate this configuration on both Leaf 2 and Leaf 3 per the topology shown in Figure 5.
Tip When you create an ESI number, always set the high order octet to 00 to indicate the ESI is manually created. The other 9 octets can be any hexadecimal value from 00 to FF.
Figure 5: ESI Topology for Leaf 1, Leaf 2, and Leaf 3 
Leaf 1:
set interfaces ae11 esi 00:00:00:00:00:00:51:00:00:01set interfaces ae11 esi all-activeset interfaces ae11 aggregated-ether-options lacp system-id 00:00:51:00:00:01set interfaces xe-0/0/10 ether-options 802.3ad ae11set interfaces xe-1/0/10 ether-options 802.3ad ae11 - Configure VLANs and map them to VNIs. This step enables
the VLANs to participate in VNIs across the EVPN/VXLAN domain.
Leaf 1:
set vlans VNI_10000 vlan-id 100set vlans VNI_10000 vxlan vni 10000set vlans VNI_20000 vlan-id 200set vlans VNI_20000 vxlan vni 20000set vlans VNI_30000 vlan-id 300set vlans VNI_30000 vxlan vni 30000set vlans VNI_40000 vlan-id 400set vlans VNI_40000 vxlan vni 40000
Verifying the VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance for the Leaf Device
Issue the following commands to verify that the overlay is working properly on your leaf devices:
- Verify the interfaces are operational.
user@leaf-1> show interfaces terse | match ae.*xe-0/0/10.0 up up aenet --> ae11.0 et-0/0/48.0 up up aenet --> ae1.0 et-0/0/49.0 up up aenet --> ae2.0 et-0/0/50.0 up up aenet --> ae3.0 et-0/0/51.0 up up aenet --> ae4.0 xe-1/0/10.0 up up aenet --> ae11.0 et-1/0/48.0 up up aenet --> ae1.0 et-1/0/49.0 up up aenet --> ae2.0 et-1/0/50.0 up up aenet --> ae3.0 et-1/0/51.0 up up aenet --> ae4.0 ae1 up up ## To Spine 1 ae1.0 up up inet 172.16.1.1/30 ae2 up up ## To Spine 2 ae2.0 up up inet 172.16.1.5/30 ae3 up up ## To Spine 3 ae3.0 up up inet 172.16.1.9/30 ae4 up up ## To Spine 4 ae4.0 up up inet 172.16.1.13/30 ae11 up up ## To End System ae11.0 up up eth-switch
user@leaf-1> show lacp interfacesAggregated interface: ae1 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity et-0/0/48 Actor No No Yes Yes Yes Yes Fast Active et-0/0/48 Partner No No Yes Yes Yes Yes Fast Active et-1/0/48 Actor No No Yes Yes Yes Yes Fast Active et-1/0/48 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State et-0/0/48 Current Fast periodic Collecting distributing et-1/0/48 Current Fast periodic Collecting distributing ... Aggregated interface: ae11 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-0/0/10 Actor No No Yes Yes Yes Yes Fast Active xe-0/0/10 Partner No No Yes Yes Yes Yes Fast Active xe-1/0/10 Actor No No Yes Yes Yes Yes Fast Active xe-1/0/10 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-0/0/10 Current Fast periodic Collecting distributing xe-1/0/10 Current Fast periodic Collecting distributing - Verify that the EVPN routes are being learned through
the overlay.
Note Only selected excerpts of this output are displayed.
The format of the EVPN routes is EVPN-route-type:route-distinguisher:vni:mac-address.
user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 10000bgp.evpn.0: 828 destinations, 3169 routes (827 active, 0 holddown, 4 hidden) + = Active Route, - = Last Active, * = Both ## Spine 1: Virtual Gateway MAC Address for IPv4 2:192.168.0.1:1::10000::00:00:5e:00:01:01/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 1: Virtual Gateway MAC Address for IPv6 2:192.168.0.1:1::10000::00:00:5e:00:02:01/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 1: IRB MAC Address 2:192.168.0.1:1::10000::06:4b:8c:67:0f:f0/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 1: ARP for the virtual gateway 2:192.168.0.1:1::10000::00:00:5e:00:01:01::10.1.0.254/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 1: IRB IPv4 ARP 2:192.168.0.1:1::10000::06:4b:8c:67:0f:f0::10.1.0.1/304 MAC/IP *[BGP/170] 00:04:50, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 00:04:50, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 00:04:50, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 00:04:50, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 2: ARP for the virtual gateway 2:192.168.0.2:1::10000::00:00:5e:00:01:01::10.1.0.254/304 MAC/IP *[BGP/170] 07:55:22, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:31:11, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:29:37, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 ## Spine 2: IRB IPv4 ARP 2:192.168.0.2:1::10000::06:4b:8c:cd:13:f8::10.1.0.2/304 MAC/IP *[BGP/170] 07:55:22, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:31:11, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 [BGP/170] 07:29:37, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.6 via ae2.0 ## Spine 1: IPv6 ARP for the virtual gateway 2:192.168.0.1:1::10000::00:00:5e:00:02:01::2001:db8::10:1:0:254/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ## Spine 1: IRB IPv6 ARP 2:192.168.0.1:1::10000::06:4b:8c:67:0f:f0::2001:db8::10:1:0:1/304 MAC/IP *[BGP/170] 09:12:00, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:33:39, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:31:15, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 [BGP/170] 07:29:41, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.2 via ae1.0 ... - Verify on Leaf 1 and Leaf 3 that the Ethernet switching
table has installed both the local MAC addresses and the remote MAC
addresses learned through the overlay.
Note To identify end systems learned remotely from the EVPN overlay, look for the MAC address, ESI logical interface, and ESI number. For example, Leaf 1 learns about an end system with the MAC address of 02:0c:10:03:02:02 through esi.1885. This end system has an ESI number of 00:00:00:00:00:00:51:10:00:01. Consequently, this matches the ESI number configured for Leaf 4, 5, and 6 (QFX5110 switches), so we know that this end system is multihomed to these three leaf devices.
user@leaf-1> show ethernet-switching table vlan-id 300MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 10 entries, 10 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source VNI_30000 00:00:5e:00:01:01 DR esi.1679 05:19:17:f3:41:00:00:75:30:00 VNI_30000 00:00:5e:00:02:01 DR esi.1679 05:19:17:f3:41:00:00:75:30:00 VNI_30000 06:4b:8c:67:0f:f0 D vtep.32770 192.168.0.1 VNI_30000 06:4b:8c:cd:13:f8 D vtep.32783 192.168.0.2 VNI_30000 06:4b:8c:cd:c4:38 D vtep.32769 192.168.0.3 VNI_30000 06:38:e1:6f:30:29 D vtep.32879 192.168.0.4 ## Learned locally VNI_30000 02:0c:10:03:02:01 DL ae11.0 ## Learned from the QFX5110 switches - Leaf 4 to 6 VNI_30000 02:0c:10:03:02:02 DR esi.1885 00:00:00:00:00:00:51:10:00:01 ## Learned from the QFX5200 switches - Leaf 7 to 9 VNI_30000 02:0c:10:03:02:03 DR esi.1887 00:00:00:00:00:00:52:00:00:01 ## Learned from the QFX10002 switches - Leaf 10 to 12 VNI_30000 02:0c:10:03:02:04 DR esi.1892 00:00:00:00:00:01:00:00:00:01 ## IPv4 virtual gateway MAC address learned over the overlay and distributed to the leaf devices by Spine 1, 2, 3 and 4 00:00:5e:00:01:01 # IPv6 virtual gateway MAC address learned over Overlay 00:00:5e:00:02:01 ## IRB MAC address prefix for Spine 1, 2, and 3 (Physical MAC address) 06:4b:* ## End System MAC address, connected locally to the leaf device 02:0c:10:03:02:01 ## MAC address learned over the overlay, these end systems are also multihomed 02:0c:10:03:02:02,03,04user@leaf-3> show ethernet-switching table vlan-id 100MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 106 entries, 106 learned Routing instance : default-switch Vlan MAC MAC Logical Active name address flags interface source ## 00:00:5e:00:01:01 is the virtual gateway MAC address for the spine devices and is reachable over the dynamically created logical link esi.1679. As a result, you can use this ESI number to filter future command output by using esi.1679 to find the virtual gateway. VNI_10000 00:00:5e:00:01:01 DR esi.1769 05:19:17:f3:41:00:00:27:10:00 VNI_10000 00:00:5e:00:02:01 DR esi.1769 05:19:17:f3:41:00:00:27:10:00 VNI_10000 06:4b:8c:67:0f:f0 D vtep.32781 192.168.0.1 VNI_10000 06:4b:8c:cd:13:f8 D vtep.32782 192.168.0.2 VNI_10000 06:4b:8c:cd:c4:38 D vtep.32775 192.168.0.3 ## Learned locally VNI_10000 02:0c:10:01:02:01 DL ae11.0 ## Learned through the overlay VNI_10000 02:0c:10:01:02:02 DR esi.1760 00:00:00:00:00:00:51:10:00:01 VNI_10000 02:0c:10:01:02:03 DR esi.1782 00:00:00:00:00:00:52:00:00:01 VNI_10000 02:0c:10:01:02:04 DR esi.1758 00:00:00:00:00:01:00:00:00:01 VNI_10000 06:38:e1:6f:30:29 D vtep.32783 192.168.0.4 VNI_10000 0e:ad:10:01:00:01 D vtep.32821 192.168.1.85 - Verify on Leaf 1 that the virtual gateway ESI (esi.1679)
is reachable by all the spine devices.
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point esi | find esi.1679ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 05:19:17:f3:41:00:00:75:30:00 default-switch 1679 131072 esi.1679 4 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.0.4 vtep.32879 1890 3 2 192.168.0.2 vtep.32783 1795 2 2 192.168.0.1 vtep.32770 1682 1 2 192.168.0.3 vtep.32769 1764 0 2 - Verify the remote EVPN routes coming from VNI 10000 and
MAC address 02:0c:10:01:02:02. In this case, they are coming from
Leaf 4 (192.168.1.4) by way of Spine 1 (192.168.0.1).
Note The format of the EVPN routes is EVPN-route-type:route-distinguisher:vni:mac-address.
user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 10000 evpn-mac-address 02:0c:10:01:02:02bgp.evpn.0: 910 destinations, 3497 routes (904 active, 0 holddown, 24 hidden) + = Active Route, - = Last Active, * = Both 2:192.168.1.4:1::10000::02:0c:10:01:02:02/304 MAC/IP *[BGP/170] 00:11:37, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0 [BGP/170] 00:11:37, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.1.10 via ae3.0user@leaf-1> show route table bgp.evpn.0 evpn-ethernet-tag-id 10000 evpn-mac-address 02:0c:10:01:02:02 detailbgp.evpn.0: 925 destinations, 3557 routes (919 active, 0 holddown, 24 hidden) 2:192.168.1.4:1::10000::02:0c:10:01:02:02/304 MAC/IP (4 entries, 0 announced) *BGP Preference: 170/-101 Route Distinguisher: 192.168.1.4:1 Next hop type: Indirect, Next hop index: 0 Address: 0xb3a2170 Next-hop reference count: 160 Source: 192.168.0.1 Protocol next hop: 192.168.1.4 Indirect next hop: 0x2 no-forward INH Session ID: 0x0 State: <Active Int Ext> Local AS: 4210000001 Peer AS: 4210000001 Age: 13:42 Metric2: 0 Validation State: unverified Task: BGP_4210000001.192.168.0.1 AS path: I (Originator) Cluster list: 192.168.0.10 Originator ID: 192.168.1.4 Communities: target:62273:268445456 encapsulation:vxlan(0x8) Import Accepted Route Label: 10000 ESI: 00:00:00:00:00:00:51:10:00:01 Localpref: 100 Router ID: 192.168.0.1 Secondary Tables: default-switch.evpn.0 ... ## This output has been abbreviated. In a full set of output, there should also be routes sourced by Spine 2 (192.168.0.2), Spine 3 (192.168.0.3), and Spine 4 (192.168.0.4). - Verify the source and destination address of each VTEP
interface and view their status.
Note There are 96 leaf devices and four spine devices, so there are 100 VTEP interfaces in this reference design - one VTEP interface per device.
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point sourceLogical System Name Id SVTEP-IP IFL L3-Idx <default> 0 192.168.1.1 lo0.0 0 L2-RTT Bridge Domain VNID MC-Group-IP default-switch VNI_10000+100 10000 0.0.0.0 default-switch VNI_20000+200 20000 0.0.0.0 default-switch VNI_30000+300 30000 0.0.0.0 default-switch VNI_40000+400 40000 0.0.0.0user@leaf-1> show interfaces terse vtepInterface Admin Link Proto Local Remote vtep up up vtep.32768 up up vtep.32769 up up eth-switch vtep.32770 up up eth-switch vtep.32771 up up eth-switch vtep.32772 up up eth-switch ... vtep.32869 up up eth-switch
user@leaf-1> show interfaces vtepPhysical interface: vtep, Enabled, Physical link is Up Interface index: 646, SNMP ifIndex: 503 Type: Software-Pseudo, Link-level type: VxLAN-Tunnel-Endpoint, MTU: Unlimited, Speed: Unlimited Device flags : Present Running Link type : Full-Duplex Link flags : None Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vtep.32768 (Index 554) (SNMP ifIndex 648) Flags: Up SNMP-Traps 0x4000 Encapsulation: ENET2 VXLAN Endpoint Type: Source, VXLAN Endpoint Address: 192.168.1.1, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 0 ... Logical interface vtep.32814 (Index 613) (SNMP ifIndex 903) Flags: Up SNMP-Traps Encapsulation: ENET2 VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 192.168.1.96, L2 Routing Instance: default-switch, L3 Routing Instance: default Input packets : 0 Output packets: 6364 Protocol eth-switch, MTU: Unlimited Flags: Trunk-Mode - Verify that each VNI maps to the associated VXLAN tunnel.
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remote0 192.168.1.1 lo0.0 0 RVTEP-IP IFL-Idx NH-Id 192.168.0.1 587 1792 VNID MC-Group-IP 10000 0.0.0.0 20000 0.0.0.0 30000 0.0.0.0 40000 0.0.0.0 ... RVTEP-IP IFL-Idx NH-Id 192.168.1.96 613 1820 VNID MC-Group-IP 10000 0.0.0.0 20000 0.0.0.0 30000 0.0.0.0 40000 0.0.0.0 - Verify that MAC addresses are learned through the VXLAN
tunnels.
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point remote mac-tableMAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC) Logical system : <default> Routing instance : default-switch Bridging domain : VNI_10000+100, VLAN : 100, VNID : 10000 MAC MAC Logical Remote VTEP address flags interface IP address 02:0c:10:01:02:04 DR esi.1764 192.168.1.11 192.168.1.12 192.168.1.10 02:0c:10:01:02:02 DR esi.1771 192.168.1.6 192.168.1.4 02:0c:10:01:02:03 DR esi.1774 192.168.1.7 00:00:5e:00:01:01 DR esi.1781 192.168.0.4 192.168.0.2 192.168.0.1 192.168.0.3 06:4b:8c:cd:c4:38 D vtep.32779 192.168.0.3 06:4b:8c:67:0f:f0 D vtep.32781 192.168.0.1 06:4b:8c:cd:13:f8 D vtep.32782 192.168.0.2 06:38:e1:6f:30:29 D vtep.32783 192.168.0.4 ---(more)--- - Verify multihoming information of the gateway and the
aggregated Ethernet interfaces.
user@leaf-1> show ethernet-switching vxlan-tunnel-end-point esi## Local AE link – QFX5100 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:51:00:00:01 default-switch 1768 131078 esi.1768 ae11.0, 2 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.2 vtep.32780 1782 1 2 192.168.1.3 vtep.32772 1767 0 2 ## Remote AE Link for QFX5110 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:51:10:00:01 default-switch 1771 131081 esi.1771 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.6 vtep.32771 1766 2 2 192.168.1.4 vtep.32770 1765 1 2 192.168.1.5 vtep.32774 1770 0 2 ## Remote AE Link for QFX5200 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:00:52:00:00:01 default-switch 1774 131084 esi.1774 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.9 vtep.32778 1776 2 2 192.168.1.8 vtep.32777 1775 1 2 192.168.1.7 vtep.32776 1773 0 2 ## Remote AE Link for QFX10002 leaf devices ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 00:00:00:00:00:01:00:00:00:01 default-switch 1764 131074 esi.1764 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.11 vtep.32775 1772 2 2 192.168.1.12 vtep.32773 1769 1 2 192.168.1.10 vtep.32769 1759 0 2 ## ESI multihoming to the VTEP for each segment ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs 05:19:17:f3:41:00:00:27:10:00 default-switch 1781 131091 esi.1781 4 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.0.4 vtep.32783 1796 3 2 192.168.0.2 vtep.32782 1793 2 2 192.168.0.1 vtep.32781 1792 1 2 192.168.0.3 vtep.32779 1777 0 2 ... - Verify that the VXLAN tunnel from one leaf to another
leaf is load balanced with equal cost multipathing (ECMP) over the
underlay.
user@leaf-1> show route forwarding-table table default-switch extensive | find vtep.32770Destination: vtep.32770 Route type: interface Route reference: 0 Route interface-index: 576 Multicast RPF nh index: 0 P2mpidx: 0 Flags: sent to PFE Nexthop: Next-hop type: composite Index: 1765 Reference: 12 Next-hop type: indirect Index: 131076 Reference: 3 Next-hop type: unilist Index: 131193 Reference: 238 Nexthop: 172.16.1.2 Next-hop type: unicast Index: 1791 Reference: 10 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.1.6 Next-hop type: unicast Index: 1794 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.1.10 Next-hop type: unicast Index: 1758 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.1.14 Next-hop type: unicast Index: 1795 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0
- Verify that remote MAC addresses are reachable through
ECMP.
user@leaf-1> show route forwarding-table table default-switch extensive destination 02:0c:10:01:02:03/48Routing table: default-switch.evpn-vxlan [Index 4] Bridging domain: VNI_10000.evpn-vxlan [Index 3] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 02:0c:10:01:02:03/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 582 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 169 Epoch: 0 Sequence Number: 0 Learn Mask: 0x400000000000000003000000000000000000000 L2 Flags: control_dyn Flags: sent to PFE Nexthop: Next-hop type: composite Index: 1773 Reference: 12 Next-hop type: indirect Index: 131085 Reference: 3 Next-hop type: unilist Index: 131193 Reference: 238 Nexthop: 172.16.1.2 Next-hop type: unicast Index: 1791 Reference: 10 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.1.6 Next-hop type: unicast Index: 1794 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.1.10 Next-hop type: unicast Index: 1758 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.1.14 Next-hop type: unicast Index: 1795 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0Note Though the MAC address is reachable over multiple VTEP interfaces, QFX5100, QFX5110, QFX5120-32C, and QFX5200 switches do not support ECMP across the overlay because of a merchant ASIC limitation. Only the QFX10000 line of switches contain a custom Juniper Networks ASIC that supports ECMP across both the overlay and the underlay.
user@leaf-1> show ethernet-switching table vlan-id 100 | match 02:0c:10:01:02:03VNI_10000 02:0c:10:01:02:03 DR esi.1774 00:00:00:00:00:00:52:00:00:01
user@leaf-1> show route forwarding-table table default-switch extensive destination 02:0c:10:01:02:03/48Routing table: default-switch.evpn-vxlan [Index 9] Bridging domain: VNI_10000.evpn-vxlan [Index 3] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 02:0c:10:01:02:03/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 550 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 0 Epoch: 0 Sequence Number: 0 Learn Mask: 0x400000000000000001000000000000000000000 L2 Flags: control_dyn, esi Flags: sent to PFE Next-hop type: indirect Index: 2097173 Reference: 5 Nexthop: Next-hop type: composite Index: 1947 Reference: 2 Nexthop: Next-hop type: composite Index: 1948 Reference: 8 Next-hop type: indirect Index: 2097174 Reference: 3 Next-hop type: unilist Index: 2097280 Reference: 241 Nexthop: 172.16.10.2 Next-hop type: unicast Index: 1950 Reference: 11 Next-hop interface: ae1.0 Weight: 0x0 Nexthop: 172.16.10.6 Next-hop type: unicast Index: 1956 Reference: 10 Next-hop interface: ae2.0 Weight: 0x0 Nexthop: 172.16.10.10 Next-hop type: unicast Index: 1861 Reference: 10 Next-hop interface: ae3.0 Weight: 0x0 Nexthop: 172.16.10.14 Next-hop type: unicast Index: 1960 Reference: 10 Next-hop interface: ae4.0 Weight: 0x0 - Verify which device is the Designated Forwarder (DF) for
broadcast, unknown, and multicast (BUM) traffic coming from the VTEP
tunnel.
Note Because the DF IP address is listed as 192.168.1.2, Leaf 2 is the DF.
user@leaf-1> show evpn instance esi 00:00:00:00:00:00:51:00:00:01 designated-forwarderInstance: default-switch Number of ethernet segments: 12 ESI: 00:00:00:00:00:00:51:00:00:01 Designated forwarder: 192.168.1.2
See also
Configuring a VLAN-Aware Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances
You can configure a VLAN-aware centrally-routed bridging overlay model using virtual switches or MAC-VRF instances. With either of these models, you can configure multiple switching instances where each switching instance can support up to 4094 VLANs per instance.
The configuration method for VLANs (at the leaf devices) and IRB interfaces (at the spine devices) is similar to the default instance method for VLAN-aware centrally-routed bridging overlays. The main difference is that you configure certain elements inside the virtual switching instances or MAC-VRF instances. See Figure 6.
When you implement this style of overlay on a spine device, you:
Configure a virtual switch or MAC-VRF instance with:
The loopback interface as the VTEP source interface.
Route distinguishers and route targets.
EVPN with VXLAN encapsulation.
VLAN to VNI mappings and Layer 3 IRB interface associations.
Configure virtual gateways, virtual MAC addresses, and corresponding IRB interfaces (to provide routing between VLANs).
To implement this overlay style on a leaf device:
Configure a virtual switch or a MAC-VRF instance with:
The loopback interface as the VTEP source interface.
Route distinguishers and route targets.
EVPN with VXLAN encapsulation.
VLAN to VNI mappings.
Set the following end system-facing elements:
An Ethernet segment ID (ESI).
Flexible VLAN tagging and extended VLAN bridge encapsulation.
LACP settings.
VLAN IDs.
For an overview of VLAN-aware centrally-routed bridging overlays, see the Centrally-Routed Bridging Overlay section in Data Center Fabric Blueprint Architecture Components.
For information on MAC-VRF instances, see MAC-VRF Instances for Multitenancy in Network Virtualization Overlays and MAC-VRF Routing Instance Type Overview.
For a simpler method that works on all leaf platforms used in this reference design, see Configuring a VLAN-Aware Centrally-Routed Bridging Overlay in the Default Instance
The following sections provide the detailed steps of how to configure and verify the VLAN-aware centrally-routed bridging overlay with virtual switches or MAC-VRF instances.
Configuring the VLAN-Aware Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances on a Spine Device
To configure a VLAN-aware style of centrally-routed bridging overlay on a spine device, perform the following:
The following example shows the configuration for Spine 1, as shown in Figure 7.
- Ensure the IP fabric underlay is in place. To configure an IP fabric on spine devices, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your spine devices, see Configure IBGP for the Overlay.
- Configure a virtual switch instance (VS1) or a MAC-VRF
instance (MAC-VRF-1) for a VLAN-aware service. With the VLAN-aware
service type, you can configure the instance with one or more VLANs.
Include VTEP information, VXLAN encapsulation, VLAN to VNI mapping,
associated IRB interfaces, and other instance details (such as a route
distinguisher and a route target) as part of the configuration.
For a virtual switch instance, use instance-type virtual-switch. Using the VLAN-aware model, configure VLANs VNI_90000 and VNI_100000 in the virtual switch instance with the associated IRB interfaces.
Spine 1 (Virtual Switch Instance):
set routing-instances VS1 vtep-source-interface lo0.0set routing-instances VS1 instance-type virtual-switchset routing-instances VS1 route-distinguisher 192.168.0.1:900set routing-instances VS1 vrf-target target:62273:90000set routing-instances VS1 vrf-target autoset routing-instances VS1 protocols evpn encapsulation vxlanset routing-instances VS1 protocols evpn extended-vni-list allset routing-instances VS1 protocols evpn default-gateway no-gateway-communityset routing-instances VS1 vlans VNI_90000 vlan-id noneset routing-instances VS1 vlans VNI_90000 l3-interface irb.900set routing-instances VS1 vlans VNI_90000 vxlan vni 90000set routing-instances VS1 vlans VNI_100000 vlan-id noneset routing-instances VS1 vlans VNI_100000 l3-interface irb.1000set routing-instances VS1 vlans VNI_100000 vxlan vni 100000With MAC-VRF instances, use instance-type mac-vrf. You also configure the service type when you create the MAC-VRF instance. Here we configure service-type vlan-aware with the two VLANs VNI_90000 and VNI_100000 and their associated IRB interfaces in the MAC-VRF instance.
Spine 1 (MAC-VRF Instance):
set routing-instances MAC-VRF-1 vtep-source-interface lo0.0set routing-instances MAC-VRF-1 instance-type mac-vrfset routing-instances MAC-VRF-1 service-type vlan-awareset routing-instances MAC-VRF-1 route-distinguisher 192.168.0.1:900set routing-instances MAC-VRF-1 vrf-target target:62273:90000set routing-instances MAC-VRF-1 vrf-target autoset routing-instances MAC-VRF-1 protocols evpn encapsulation vxlanset routing-instances MAC-VRF-1 protocols evpn extended-vni-list allset routing-instances MAC-VRF-1 protocols evpn default-gateway no-gateway-communityset routing-instances MAC-VRF-1 vlans VNI_90000 vlan-id noneset routing-instances MAC-VRF-1 vlans VNI_90000 l3-interface irb.900set routing-instances MAC-VRF-1 vlans VNI_90000 vxlan vni 90000set routing-instances MAC-VRF-1 vlans VNI_100000 vlan-id noneset routing-instances MAC-VRF-1 vlans VNI_100000 l3-interface irb.1000set routing-instances MAC-VRF-1 vlans VNI_100000 vxlan vni 100000 - (MAC-VRF instances only) Enable shared tunnels on the
device.
A device can have problems with VTEP scaling when the configuration uses multiple MAC-VRF instances. As a result, to avoid this problem, we require that you enable the shared tunnels feature on the QFX5000 line of switches with a MAC-VRF instance configuration. When you configure the shared-tunnels option, the device minimizes the number of next-hop entries to reach remote VTEPs. The following statement globally enables shared VXLAN tunnels on the device:
set forwarding-options evpn-vxlan shared-tunnelsThis statement is optional on the QFX10000 line of switches, which can handle higher VTEP scaling than QFX5000 switches.
Note This setting requires you to reboot the device.
- Configure spine devices with one or more VLANs for the
VLAN-aware method. Include settings for the IPv4 and IPv6 virtual
gateways and virtual MAC addresses. This example shows the configuration
for Spine 1 with IRB interfaces and virtual gateways for VLANs VNI_90000
and VNI_100000.
Spine 1:
set interfaces irb unit 900 proxy-macip-advertisementset interfaces irb unit 900 virtual-gateway-accept-dataset interfaces irb unit 900 family inet address 10.1.8.1/24 virtual-gateway-address 10.1.8.254set interfaces irb unit 900 family inet6 address 2001:db8::10:1:8:1/112 virtual-gateway-address 2001:db8::10:1:8:254set interfaces irb unit 900 family inet6 address fe80::10:1:8:1/112set interfaces irb unit 900 family inet6 nd6-stale-time 600set interfaces irb unit 900 virtual-gateway-v4-mac 00:00:5e:90:00:00set interfaces irb unit 900 virtual-gateway-v6-mac 00:00:5e:90:00:00set interfaces irb unit 1000 proxy-macip-advertisementset interfaces irb unit 1000 virtual-gateway-accept-dataset interfaces irb unit 1000 family inet address 10.1.9.1/24 virtual-gateway-address 10.1.9.254set interfaces irb unit 1000 family inet6 address 2001:db8::10:1:9:1/112 virtual-gateway-address 2001:db8::10:1:9:254set interfaces irb unit 1000 family inet6 address fe80::10:1:9:1/112set interfaces irb unit 1000 family inet6 nd6-stale-time 600set interfaces irb unit 1000 virtual-gateway-v4-mac 00:00:5e:a0:00:00set interfaces irb unit 1000 virtual-gateway-v6-mac 00:00:5e:a0:00:00
Verifying the VLAN-Aware Model for a Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances on a Spine Device
To verify this style of overlay on a spine device, run the commands in this section.
Most commands here show output for a virtual switch instance configuration. With a MAC-VRF instance configuration, you can alternatively use:
show mac-vrf forwarding commands that are aliases for the show ethernet-switching commands in this section.
The show mac-vrf routing database command, which is an alias for the show evpn database command in this section.
The show mac-vrf routing instance command, which is an alias for the show evpn instance command in this section.
See MAC-VRF Routing Instance Type Overview for tables of show mac-vrf forwarding and show ethernet-switching command mappings, and show mac-vrf routing command aliases for show evpn commands.
Otherwise, you can use the commands in this section for either virtual switch instances or MAC-VRF instances.
The output with a MAC-VRF instance configuration displays similar information for MAC-VRF routing instances as this section shows for virtual switch instances. One main difference you might see is in the output with MAC-VRF instances on devices where you enable the shared tunnels feature. With shared tunnels enabled, you see VTEP interfaces in the following format:
vtep-index.shared-tunnel-unit
where:
index is the index associated with the MAC-VRF routing instance.
shared-tunnel-unit is the unit number associated with the shared tunnel remote VTEP logical interface.
For example, if a device has a MAC-VRF instance with index 26 and the instance connects to two remote VTEPs, the shared tunnel VTEP logical interfaces might look like this:
vtep-26.32823 vtep-26.32824
- Verify the IRB interfaces for VNIs 90000 and 100000 are
operational for both IPv4 and IPv6.
user@spine-1> show interfaces terse irb | find irb\.900irb.900 up up inet 10.1.8.1/24 10.1.8.254/24 inet6 2001:db8::10:1:8:1/112 2001:db8::10:1:8:254/112 fe80::10:1:8:1/112 irb.1000 up up inet 10.1.9.1/24 10.1.9.254/24 inet6 2001:db8::10:1:9:1/112 2001:db8::10:1:9:254/112 fe80::10:1:9:1/112 - (MAC-VRF instances only) Verify the VLANs you configured
as part of the MAC-VRF instance.
user@spine-1> show mac-vrf forwarding instance MAC-VRF-1Information for routing instance and VLAN: Flags (DL - disable learning, SE - stats enabled, AD - packet action drop, LH - MAC limit hit, MI - mac+ip limit hit) Inst Logical Routing VLAN name Index IRB Flags Tag type system instance index RTT Default MAC-VRF-1 26 vlan Default MAC-VRF-1 VNI-90000 3403 4204 900 vlan Default MAC-VRF-1 VNI-100000 3425 4203 1000 . . .user@spine-1> show vlans VNI-90000Routing instance VLAN name Tag Interfaces MAC-VRF-1 VNI-90000 NA esi.105902* esi.89032* vtep-26.32823* vtep-26.32824* vtep-26.32827* - Verify switching details about the EVPN routing instance.
This output includes information about the route distinguisher (192.168.1.10:900),
VXLAN encapsulation, ESI (00:00:00:00:00:01:00:00:00:02), verification
of the VXLAN tunnels for VLANs 900 and 1000, EVPN neighbors (Spine
2 - 4, and Leaf 10 - 12), and the source VTEP IP address (192.168.0.1).
user@spine-1> show evpn instance VS1 extensiveInstance: VS1 Route Distinguisher: 192.168.0.1:900 Encapsulation type: VXLAN MAC database status Local Remote MAC advertisements: 2 14 MAC+IP advertisements: 14 26 Default gateway MAC advertisements: 4 0 Number of local interfaces: 0 (0 up) Number of IRB interfaces: 2 (2 up) Interface name VLAN VNI Status L3 context irb.1000 100000 Up master irb.900 90000 Up master Number of bridge domains: 2 VLAN Domain ID Intfs / up IRB intf Mode MAC sync IM route label SG sync IM core nexthop 8191 90000 0 0 irb.900 Extended Enabled 90000 Disabled 8191 100000 0 0 irb.1000 Extended Enabled 100000 Disabled Number of neighbors: 6 Address MAC MAC+IP AD IM ES Leaf-label 192.168.0.2 4 10 2 2 0 192.168.0.3 4 10 2 2 0 192.168.0.4 4 10 2 2 0 192.168.1.10 1 2 2 2 0 192.168.1.11 0 0 2 2 0 192.168.1.12 1 2 2 2 0 Number of ethernet segments: 3 ESI: 00:00:00:00:00:01:00:00:00:02 Status: Resolved Number of remote PEs connected: 3 Remote PE MAC label Aliasing label Mode 192.168.1.12 90000 0 all-active 192.168.1.11 90000 0 all-active 192.168.1.10 100000 0 all-active ESI: 05:19:17:f3:41:00:01:5f:90:00 Local interface: irb.900, Status: Up/Forwarding Number of remote PEs connected: 3 Remote PE MAC label Aliasing label Mode 192.168.0.3 90000 0 all-active 192.168.0.2 90000 0 all-active 192.168.0.4 90000 0 all-active ESI: 05:19:17:f3:41:00:01:86:a0:00 Local interface: irb.1000, Status: Up/Forwarding Number of remote PEs connected: 3 Remote PE MAC label Aliasing label Mode 192.168.0.3 100000 0 all-active 192.168.0.2 100000 0 all-active 192.168.0.4 100000 0 all-active Router-ID: 192.168.0.1 Source VTEP interface IP: 192.168.0.1 - Verify the MAC address table on the leaf device.
Note 00:00:5e:90:00:00 and 00:00:5e:a0:00:00 are the IP subnet gateways on the spine device.
02:0c:10:09:02:01 and 02:0c:10:08:02:01 are end systems connected through the leaf device.
user@spine-1> show ethernet-switching table instance VS1MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 5 entries, 5 learned Routing instance : VS1 Vlan MAC MAC Logical Active name address flags interface source VNI_100000 00:00:5e:a0:00:00 DR esi.2454 05:19:17:f3:41:00:01:86:a0:00 VNI_100000 06:4b:8c:cd:13:f8 D vtep.32773 192.168.0.2 VNI_100000 06:4b:8c:cd:c4:38 D vtep.32787 192.168.0.3 VNI_100000 02:0c:10:09:02:01 DR esi.2467 00:00:00:00:00:01:00:00:00:02 VNI_100000 06:38:e1:6f:30:29 D vtep.32796 192.168.0.4 MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 5 entries, 5 learned Routing instance : VS1 Vlan MAC MAC Logical Active name address flags interface source VNI_90000 00:00:5e:90:00:00 DR esi.2455 05:19:17:f3:41:00:01:5f:90:00 VNI_90000 06:4b:8c:cd:13:f8 D vtep.32773 192.168.0.2 VNI_90000 06:4b:8c:cd:c4:38 D vtep.32787 192.168.0.3 VNI_90000 02:0c:10:08:02:01 DR esi.2467 00:00:00:00:00:01:00:00:00:02 VNI_90000 06:38:e1:6f:30:29 D vtep.32796 192.168.0.4 - Verify the end system MAC address is reachable from all
three leaf devices.
user@spine-1> show ethernet-switching vxlan-tunnel-end-point esi | find esi.246700:00:00:00:00:01:00:00:00:02 VS1 2467 2097182 esi.2467 3 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.1.10 vtep.32789 2522 2 2 192.168.1.11 vtep.32782 2475 1 2 192.168.1.12 vtep.32779 2466 0 2 ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPs - Verify the end system is reachable through the forwarding
table.
user@spine-1> show route forwarding-table table VS1 destination 02:0c:10:09:02:01/48 extensiveRouting table: VS1.evpn-vxlan [Index 11] Bridging domain: VNI_100000.evpn-vxlan [Index 9] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 02:0c:10:09:02:01/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 676 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 0 Epoch: 0 Sequence Number: 0 Learn Mask: 0x400000000000000006000000000000000000000 L2 Flags: control_dyn, esi Flags: sent to PFE Next-hop type: indirect Index: 2097182 Reference: 3 Nexthop: Next-hop type: composite Index: 2467 Reference: 2 Nexthop: Next-hop type: composite Index: 2522 Reference: 6 Next-hop type: indirect Index: 2097154 Reference: 5 Nexthop: 172.16.10.1 Next-hop type: unicast Index: 2172 Reference: 11 Next-hop interface: ae10.0 Nexthop: Next-hop type: composite Index: 2475 Reference: 6 Next-hop type: indirect Index: 2097193 Reference: 5 Nexthop: 172.16.11.1 Next-hop type: unicast Index: 2194 Reference: 11 Next-hop interface: ae11.0 Nexthop: Next-hop type: composite Index: 2466 Reference: 6 Next-hop type: indirect Index: 2097195 Reference: 5 Nexthop: 172.16.12.1 Next-hop type: unicast Index: 2916 Reference: 11 Next-hop interface: ae12.0 - Verify end system information (MAC address, IP address,
etc.) has been added to the IPv4 ARP table and IPv6 neighbor table.
user@spine-1> show arp no-resolve expiration-time | match "irb.900|irb.1000"06:4b:8c:cd:13:f8 10.1.8.2 irb.900 [vtep.32773] none 1035 06:4b:8c:cd:c4:38 10.1.8.3 irb.900 [vtep.32787] none 1064 06:38:e1:6f:30:29 10.1.8.4 irb.900 [vtep.32796] none 964 02:0c:10:08:02:01 10.1.8.201 irb.900 [.local..11] none 781 06:4b:8c:cd:13:f8 10.1.9.2 irb.1000 [vtep.32773] none 910 06:4b:8c:cd:c4:38 10.1.9.3 irb.1000 [vtep.32787] none 1344 06:38:e1:6f:30:29 10.1.9.4 irb.1000 [vtep.32796] none 1160 02:0c:10:09:02:01 10.1.9.201 irb.1000 [.local..11] none 1099
user@spine-1> show ipv6 neighbors | match "irb.900|irb.1000"2001:db8::10:1:8:2 06:4b:8c:cd:13:f8 stale 969 yes no irb.900 [vtep.32773] 2001:db8::10:1:8:3 06:4b:8c:cd:c4:38 stale 1001 yes no irb.900 [vtep.32787] 2001:db8::10:1:8:4 06:38:e1:6f:30:29 stale 971 yes no irb.900 [vtep.32796] 2001:db8::10:1:8:201 02:0c:10:08:02:01 stale 1178 no no irb.900 [.local..11] 2001:db8::10:1:9:2 06:4b:8c:cd:13:f8 stale 955 yes no irb.1000 [vtep.32773] 2001:db8::10:1:9:3 06:4b:8c:cd:c4:38 stale 1006 yes no irb.1000 [vtep.32787] 2001:db8::10:1:9:4 06:38:e1:6f:30:29 stale 990 yes no irb.1000 [vtep.32796] 2001:db8::10:1:9:201 02:0c:10:09:02:01 stale 1199 no no irb.1000 [.local..11] fe80::10:1:8:2 06:4b:8c:cd:13:f8 stale 991 yes no irb.900 [vtep.32773] fe80::10:1:8:3 06:4b:8c:cd:c4:38 stale 989 yes no irb.900 [vtep.32787] fe80::10:1:8:4 06:38:e1:6f:30:29 stale 966 yes no irb.900 [vtep.32796] fe80::10:1:9:2 06:4b:8c:cd:13:f8 stale 978 yes no irb.1000 [vtep.32773] fe80::10:1:9:3 06:4b:8c:cd:c4:38 stale 994 yes no irb.1000 [vtep.32787] fe80::10:1:9:4 06:38:e1:6f:30:29 stale 1006 yes no irb.1000 [vtep.32796]
- Verify that the EVPN database contains the MAC address
(02:0c:10:08:02:01) and ARP information learned from an end system
connected to the leaf device.
user@spine-1> show evpn database mac-address 02:0c:10:08:02:01 extensiveInstance: VS1 VN Identifier: 90000, MAC address:: 02:0c:10:08:02:01 Source: 00:00:00:00:00:01:00:00:00:02, Rank: 1, Status: Active Remote origin: 192.168.1.10 Remote origin: 192.168.1.11 Remote origin: 192.168.1.12 Timestamp: Sep 10 23:47:37 (0x59b63189) State: <Remote-To-Local-Adv-Done> IP address: 10.1.8.201 Flags: <Proxy> Remote origin: 192.168.1.10 Remote origin: 192.168.1.11 Remote origin: 192.168.1.12 IP address: 2001:db8::10:1:8:201 Remote origin: 192.168.1.10 Remote origin: 192.168.1.11 Remote origin: 192.168.1.12 History db: Time Event Sep 10 23:47:39 2017 Applying remote state to peer 192.168.1.11 Sep 10 23:47:39 2017 Remote peer 192.168.1.11 updated Sep 10 23:47:39 2017 MAC+IP not updated, source l2ald is not owner (type2) Sep 10 23:47:39 2017 Updated Sep 10 23:47:39 2017 No change to MAC state Sep 10 23:47:39 2017 Applying remote state to peer 192.168.1.12 Sep 10 23:47:39 2017 Remote peer 192.168.1.12 updated Sep 10 23:47:39 2017 MAC+IP not updated, source l2ald is not owner (type2) Sep 10 23:47:39 2017 Updated Sep 10 23:47:39 2017 No change to MAC state
Configuring the VLAN-Aware Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances on a Leaf Device
To configure a VLAN-aware centrally-routed bridging overlay in a virtual switch or a MAC-VRF instance on a leaf device, perform the following:
The following example shows the configuration for Leaf 10, as shown in Figure 8.
- Ensure the IP fabric underlay is in place. To configure an IP fabric on leaf devices, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your leaf devices, see Configure IBGP for the Overlay.
- Configure a virtual switch instance (VS1) or a MAC-VRF
instance (MAC-VRF-1) to enable EVPN/VXLAN. Also, map VLANs 900 and
1000 to VNIs 90000 and 100000 in the instance.
For a virtual switch instance, use instance-type virtual-switch.
Leaf 10 (Virtual Switch Instance):
set routing-instances VS1 vtep-source-interface lo0.0set routing-instances VS1 instance-type virtual-switchset routing-instances VS1 route-distinguisher 192.168.1.10:900set routing-instances VS1 vrf-target target:62273:90000set routing-instances VS1 vrf-target autoset routing-instances VS1 protocols evpn encapsulation vxlanset routing-instances VS1 protocols evpn extended-vni-list allset routing-instances VS1 protocols evpn default-gateway no-gateway-communityset routing-instances VS1 vlans VNI_90000 interface ae12.900set routing-instances VS1 vlans VNI_90000 vxlan vni 90000set routing-instances VS1 vlans VNI_100000 interface ae12.1000set routing-instances VS1 vlans VNI_100000 vxlan vni 100000With MAC-VRF instances, use instance-type mac-vrf. You also configure the service type when you create the MAC-VRF instance. Here we configure service-type vlan-aware with the two VLANs VNI_90000 and VNI_100000, and their VNI mappings.
Leaf 10 (MAC-VRF Instance):
set routing-instances MAC-VRF-1 vtep-source-interface lo0.0set routing-instances MAC-VRF-1 instance-type mac-vrfset routing-instances MAC-VRF-1 service-type vlan-awareset routing-instances MAC-VRF-1 route-distinguisher 192.168.1.10:900set routing-instances MAC-VRF-1 vrf-target target:62273:90000set routing-instances MAC-VRF-1 vrf-target autoset routing-instances MAC-VRF-1 protocols evpn encapsulation vxlanset routing-instances MAC-VRF-1 protocols evpn extended-vni-list allset routing-instances MAC-VRF-1 protocols evpn default-gateway no-gateway-communityset routing-instances MAC-VRF-1 vlans VNI_90000 interface ae12.900set routing-instances MAC-VRF-1 vlans VNI_90000 vxlan vni 90000set routing-instances MAC-VRF-1 vlans VNI_100000 interface ae12.1000set routing-instances MAC-VRF-1 vlans VNI_100000 vxlan vni 100000 - (MAC-VRF instances only) Enable shared tunnels on the
device.
A device can have problems with VTEP scaling when the configuration uses multiple MAC-VRF instances. As a result, to avoid this problem, we require that you enable the shared tunnels feature on the QFX5000 line of switches with a MAC-VRF instance configuration. When you configure the shared-tunnels option, the device minimizes the number of next-hop entries to reach remote VTEPs. The following statement globally enables shared VXLAN tunnels on the device:
set forwarding-options evpn-vxlan shared-tunnelsThis statement is optional on the QFX10000 line of switches, which can handle higher VTEP scaling than QFX5000 switches.
Note This setting requires you to reboot the device.
- Configure the leaf device to communicate with the end
system. In this example, configure an aggregated Ethernet interface
on Leaf 10—in this case, ae12 with two member interfaces. With
the interface definition, include LACP options, an ESI in all-active
mode, and VLANs 900 and 1000 (which this example uses for the VLAN-aware
service type). Figure 9 illustrates the topology.
Figure 9: ESI Topology for Leaf 10, Leaf 11, and Leaf 12 
Leaf 10:
set interfaces ae12 flexible-vlan-taggingset interfaces ae12 encapsulation extended-vlan-bridgeset interfaces ae12 esi 00:00:00:00:00:01:00:00:00:02set interfaces ae12 esi all-activeset interfaces ae12 aggregated-ether-options minimum-links 1set interfaces ae12 aggregated-ether-options lacp activeset interfaces ae12 aggregated-ether-options lacp periodic fastset interfaces ae12 aggregated-ether-options lacp system-id 00:01:00:00:00:02set interfaces ae12 aggregated-ether-options lacp hold-time up 300set interfaces ae12 unit 900 vlan-id 900set interfaces ae12 unit 1000 vlan-id 1000set interfaces et-0/0/23 ether-options 802.3ad ae12set interfaces et-0/0/35 ether-options 802.3ad ae12Note that in this example, you configure the aggregated Ethernet interface to support the service provider configuration style. See Flexible Ethernet Service Encapsulation for more information on the service provider style configuration for switch interfaces.
Verifying the VLAN-Aware Centrally-Routed Bridging Overlay with Virtual Switches or MAC-VRF Instances on a Leaf Device
To verify this style of overlay on a leaf device, run the commands in this section.
Most commands here show output for a virtual switch instance configuration. With a MAC-VRF instance configuration, you can alternatively use:
show mac-vrf forwarding commands that are aliases for the show ethernet-switching commands in this section.
The show mac-vrf routing instance command, which is an alias for the show evpn instance command in this section.
See MAC-VRF Routing Instance Type Overview for tables of show mac-vrf forwarding and show ethernet-switching command mappings, and show mac-vrf routing command aliases for show evpn commands.
Otherwise, you can use the commands in this section for either virtual switch instances or MAC-VRF instances.
The output with a MAC-VRF instance configuration displays similar information for MAC-VRF routing instances as this section shows for virtual switch instances. One main difference you might see is in the output with MAC-VRF instances on devices where you enable the shared tunnels feature. With shared tunnels enabled, you see VTEP interfaces in the following format:
vtep-index.shared-tunnel-unit
where:
index is the index associated with the MAC-VRF routing instance.
shared-tunnel-unit is the unit number associated with the shared tunnel remote VTEP logical interface.
For example, if a device has a MAC-VRF instance with index 26 and the instance connects to two remote VTEPs, the shared tunnel VTEP logical interfaces might look like this:
vtep-26.32823 vtep-26.32824
- Verify that the aggregated Ethernet interface is operational
on the leaf device.
user@leaf-10> show interfaces terse ae12Interface Admin Link Proto Local Remote ae12 up up ae12.900 up up eth-switch ae12.1000 up up eth-switch ae12.32767 up up
- (MAC-VRF instances only) Verify the VLANs you configured
as part of the MAC-VRF instance.
user@leaf-10> show mac-vrf forwarding instance MAC-VRF-1Information for routing instance and VLAN: Flags (DL - disable learning, SE - stats enabled, AD - packet action drop, LH - MAC limit hit, MI - mac+ip limit hit) Inst Logical Routing VLAN name Index IRB Flags Tag type system instance index RTT Default MAC-VRF-1 27 vlan Default MAC-VRF-1 VNI-90000 3424 NA vlan Default MAC-VRF-1 VNI-100000 3425 NA . . .user@leaf-10> show vlans VNI-90000Routing instance VLAN name Tag Interfaces MAC-VRF-1 VNI-90000 NA ae12.900* esi.19955* esi.20938* vtep-27.32820* vtep-27.32821* vtep-27.32822* xe-0/0/18:2.900* - Verify switching details about the EVPN routing instance.
This output includes information about the route distinguisher (192.168.1.10:900),
VXLAN encapsulation, ESI (00:00:00:00:00:01:00:00:00:02), verification
of the VXLAN tunnels for VLANs 900 and 1000, EVPN neighbors (Spine
1 - 4, and Leaf 11 and 12), and the source VTEP IP address (192.168.1.10).
user@leaf-10> show evpn instance VS1 extensiveInstance: VS1 Route Distinguisher: 192.168.1.10:900 Encapsulation type: VXLAN MAC database status Local Remote MAC advertisements: 0 20 MAC+IP advertisements: 0 32 Default gateway MAC advertisements: 0 0 Number of local interfaces: 2 (2 up) Interface name ESI Mode Status AC-Role ae12.1000 00:00:00:00:00:01:00:00:00:02 all-active Up Root ae12.900 00:00:00:00:00:01:00:00:00:02 all-active Up Root Number of IRB interfaces: 0 (0 up) Number of bridge domains: 2 VLAN Domain ID Intfs / up IRB intf Mode MAC sync IM route label SG sync IM core nexthop None 90000 1 1 Extended Enabled 90000 Disabled None 100000 1 1 Extended Enabled 100000 Disabled Number of neighbors: 6 Address MAC MAC+IP AD IM ES Leaf-label 192.168.0.1 4 10 2 2 0 192.168.0.2 4 10 2 2 0 192.168.0.3 4 10 2 2 0 192.168.0.4 4 10 2 2 0 192.168.1.11 2 4 2 2 0 192.168.1.12 2 4 2 2 0 Number of ethernet segments: 3 ESI: 00:00:00:00:00:01:00:00:00:02 Status: Resolved by IFL ae12.900 Local interface: ae12.1000, Status: Up/Forwarding Number of remote PEs connected: 2 Remote PE MAC label Aliasing label Mode 192.168.1.12 100000 0 all-active 192.168.1.11 90000 0 all-active DF Election Algorithm: MOD based Designated forwarder: 192.168.1.10 Backup forwarder: 192.168.1.11 Backup forwarder: 192.168.1.12 Last designated forwarder update: Sep 10 23:22:07 ESI: 05:19:17:f3:41:00:01:5f:90:00 Status: Resolved Number of remote PEs connected: 4 Remote PE MAC label Aliasing label Mode 192.168.0.1 90000 0 all-active 192.168.0.3 90000 0 all-active 192.168.0.2 90000 0 all-active 192.168.0.4 90000 0 all-active ESI: 05:19:17:f3:41:00:01:86:a0:00 Status: Resolved Number of remote PEs connected: 4 Remote PE MAC label Aliasing label Mode 192.168.0.1 100000 0 all-active 192.168.0.3 100000 0 all-active 192.168.0.2 100000 0 all-active 192.168.0.4 100000 0 all-active Router-ID: 192.168.1.10 Source VTEP interface IP: 192.168.1.10 - View the MAC address table on the leaf device to confirm
that spine device and end system MAC addresses appear in the table.
Note 00:00:5e:90:00:00 and 00:00:5e:a0:00:00 are the IP subnet gateways on the spine device.
02:0c:10:09:02:01 and 02:0c:10:08:02:01 are end systems connected through the leaf device.
user@leaf-10> show ethernet-switching table instance VS1MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 6 entries, 6 learned Routing instance : VS1 Vlan MAC MAC Logical Active name address flags interface source VNI_100000 00:00:5e:a0:00:00 DR esi.2139 05:19:17:f3:41:00:01:86:a0:00 VNI_100000 06:4b:8c:67:0f:f0 D vtep.32799 192.168.0.1 VNI_100000 06:4b:8c:cd:13:f8 D vtep.32798 192.168.0.2 VNI_100000 06:4b:8c:cd:c4:38 D vtep.32804 192.168.0.3 VNI_100000 02:0c:10:09:02:01 DR ae12.1000 VNI_100000 06:38:e1:6f:30:29 D vtep.32807 192.168.0.4 MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC) Ethernet switching table : 6 entries, 6 learned Routing instance : VS1 Vlan MAC MAC Logical Active name address flags interface source VNI_90000 00:00:5e:90:00:00 DR esi.2144 05:19:17:f3:41:00:01:5f:90:00 VNI_90000 06:4b:8c:67:0f:f0 D vtep.32799 192.168.0.1 VNI_90000 06:4b:8c:cd:13:f8 D vtep.32798 192.168.0.2 VNI_90000 06:4b:8c:cd:c4:38 D vtep.32804 192.168.0.3 VNI_90000 02:0c:10:08:02:01 DR ae12.900 VNI_90000 06:38:e1:6f:30:29 D vtep.32807 192.168.0.4 - Verify that the IP subnet gateway ESIs discovered in Step
3 (esi.2144 for VNI 90000 and esi.2139 for VNI 100000) are reachable
from all four spine devices.
user@leaf-10> show ethernet-switching vxlan-tunnel-end-point esi | find esi.214405:19:17:f3:41:00:01:5f:90:00 VS1 2144 2097224 esi.2144 4 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.0.4 vtep.32807 2033 3 2 192.168.0.2 vtep.32798 2092 2 2 192.168.0.3 vtep.32804 2174 1 2 192.168.0.1 vtep.32799 2093 0 2 ESI RTT VLNBH INH ESI-IFL LOC-IFL #RVTEPsuser@leaf-10> show ethernet-switching vxlan-tunnel-end-point esi | find esi.213905:19:17:f3:41:00:01:86:a0:00 VS1 2139 2097221 esi.2139 4 RVTEP-IP RVTEP-IFL VENH MASK-ID FLAGS 192.168.0.4 vtep.32807 2033 3 2 192.168.0.2 vtep.32798 2092 2 2 192.168.0.3 vtep.32804 2174 1 2 192.168.0.1 vtep.32799 2093 0 2 - Verify the IP subnet gateway on the spine device (00:00:5e:a0:00:00)
is reachable through the forwarding table.
user@leaf-10> show route forwarding-table table VS1 destination 00:00:5e:a0:00:00/48 extensiveRouting table: VS1.evpn-vxlan [Index 10] Bridging domain: VNI_100000.evpn-vxlan [Index 15] VPLS: Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN, Destination: 00:00:5e:a0:00:00/48 Learn VLAN: 0 Route type: user Route reference: 0 Route interface-index: 571 Multicast RPF nh index: 0 P2mpidx: 0 IFL generation: 0 Epoch: 0 Sequence Number: 0 Learn Mask: 0x40000000000000000f000000000000000000000 L2 Flags: control_dyn, esi Flags: sent to PFE Next-hop type: indirect Index: 2097221 Reference: 2 Nexthop: Next-hop type: composite Index: 2139 Reference: 2 Nexthop: Next-hop type: composite Index: 2033 Reference: 9 Next-hop type: indirect Index: 2097223 Reference: 5 Nexthop: 172.16.10.14 Next-hop type: unicast Index: 2106 Reference: 10 Next-hop interface: ae4.0 Nexthop: Next-hop type: composite Index: 2092 Reference: 9 Next-hop type: indirect Index: 2097172 Reference: 5 Nexthop: 172.16.10.6 Next-hop type: unicast Index: 1951 Reference: 11 Next-hop interface: ae2.0 Nexthop: Next-hop type: composite Index: 2174 Reference: 9 Next-hop type: indirect Index: 2097174 Reference: 5 Nexthop: 172.16.10.10 Next-hop type: unicast Index: 2143 Reference: 11 Next-hop interface: ae3.0 Nexthop: Next-hop type: composite Index: 2093 Reference: 9 Next-hop type: indirect Index: 2097165 Reference: 5 Nexthop: 172.16.10.2 Next-hop type: unicast Index: 2153 Reference: 11 Next-hop interface: ae1.0
See also
Centrally-Routed Bridging Overlay — Release History
Table 1 provides a history of all of the features in this section and their support within this reference design.
Table 1: Centrally-Routed Bridging Overlay in the Data Center Fabric Reference Design– Release History
Release | Description |
|---|---|
19.1R2 | QFX10002-60C and QFX5120-32C switches running Junos OS Release 19.1R2 and later releases in the same release train support all features documented in this section. |
17.3R3-S2 | Adds support for Contrail Enterprise Multicloud, where you can configure centrally-routed bridging overlays from the Contrail Command GUI. |
17.3R3-S1 | All devices in the reference design that support Junos OS Release 17.3R3-S1 and later releases in the same release train also support all features documented in this section |