Routed Overlay Design and Implementation
A third overlay option for this Cloud Data Center reference design is a routed overlay, as shown in Figure 1.

The routed overlay service supports IP-connected end systems that interconnect with leaf devices using IP (not Ethernet). The end systems can use dynamic routing protocols to exchange IP routing information with the leaf devices. The IP addresses and prefixes of the end systems are advertised as EVPN type-5 routes and imported by other EVPN/VXLAN-enabled spine and leaf devices into a corresponding VRF routing instance.
To implement a routed overlay, you must use one of the QFX10000 line of switches as the leaf devices. (The QFX5110 is planned to be verified in a future revision of this guide.)
On the spine devices, create a VRF routing instance to accept the EVPN type-5 routes. Include route distinguishers and route targets, use the advertise direct-nexthop option, and configure the same VNI used by the leaf devices.
To configure the leaf devices, create a policy that matches on the BGP routes received from the end systems, and send these routes into a VRF routing instance enabled for EVPN type-5 routes so they can be shared with other leaf and spine devices in the same IP VNI.
For an overview of routed overlays, see the Routed Overlay section in Data Center Fabric Blueprint Architecture Components.
The following sections show the detailed steps of how to configure and verify the routed overlay:
Configuring the Routed Overlay on a Spine Device
To configure the routed overlay on a spine device, perform the following:
The following example shows the configuration for Spine 1, as shown in Figure 2.

- Ensure the IP fabric underlay is in place. To see the steps required to configure an IP fabric on a spine device, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your spine device, see Configuring IBGP for the Overlay.
- Create a VRF routing instance that enables the EVPN type-5
option (also known as IP prefix routes).
Configure a route distinguisher and route target, provide load balancing
for EVPN type-5 routes with the multipath ECMP option,
enable EVPN to advertise direct next hops, specify VXLAN encapsulation,
and assign the VNI.
Note This VRF can be used for north-south traffic flow and will be explained in a future version of this guide.
Spine 1:
set routing-instances VRF_5 instance-type vrfset routing-instances VRF_5 route-distinguisher 192.168.0.1:1677set routing-instances VRF_5 vrf-target target:62273:16776000set routing-instances VRF_5 routing-options multipathset routing-instances VRF_5 routing-options rib VRF_5.inet6.0 multipathset routing-instances VRF_5 protocols evpn ip-prefix-routes advertise direct-nexthopset routing-instances VRF_5 protocols evpn ip-prefix-routes encapsulation vxlanset routing-instances VRF_5 protocols evpn ip-prefix-routes vni 16776000
Verifying the Routed Overlay on a Spine Device
To verify that the routed overlay on a spine device is working, perform the following:
- Verify that the EVPN type-5 routes are being advertised
and received for IPv4 and IPv6.
user@spine-1> show evpn ip-prefix-database l3-context VRF_5
L3 context: VRF_5
EVPN to IPv4 Imported Prefixes
Prefix Etag 172.16.104.0/30 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.10:10 16776000 06:31:46:e2:f0:2a 192.168.1.10 172.16.104.4/30 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 172.16.104.8/30 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12 192.168.3.4/32 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.10:10 16776000 06:31:46:e2:f0:2a 192.168.1.10 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12
EVPN-->IPv6 Imported Prefixes Prefix Etag 2001:db8::172:19:4:0/126 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.10:10 16776000 06:31:46:e2:f0:2a 192.168.1.10 2001:db8::172:19:4:4/126 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 2001:db8::172:19:4:8/126 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12 2001:db8::192:168:3:4/128 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.10:10 16776000 06:31:46:e2:f0:2a 192.168.1.10 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12
- Verify that the end system is multihomed to all three
QFX10000 leaf devices for both IPv4 and IPv6.
user@spine-1> show route 192.168.3.4 table VRF_5
VRF_5.inet.0: 4 destinations, 7 routes (4 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 192.168.3.4/32 @[EVPN/170] 01:15:09 > to 172.16.10.1 via ae10.0 [EVPN/170] 01:19:53 > to 172.16.11.1 via ae11.0 [EVPN/170] 00:06:21 > to 172.16.12.1 via ae12.0 #[Multipath/255] 00:06:21, metric2 0 to 172.16.10.1 via ae10.0 to 172.16.11.1 via ae11.0 > to 172.16.12.1 via ae12.0
user@spine-1> show route 2001:db8::192:168:3:4 table VRF_5
VRF_5.inet6.0: 4 destinations, 7 routes (4 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 2001:db8::192:168:3:4/128 @[EVPN/170] 01:17:04 > to 172.16.10.1 via ae10.0 [EVPN/170] 01:21:48 > to 172.16.11.1 via ae11.0 [EVPN/170] 00:08:16 > to 172.16.12.1 via ae12.0 #[Multipath/255] 00:00:12, metric2 0 to 172.16.10.1 via ae10.0 > to 172.16.11.1 via ae11.0 to 172.16.12.1 via ae12.0
Configuring the Routed Overlay on a Leaf Device
To configure the routed overlay on a leaf device, perform the following:
The following example shows the configuration for Leaf 10, as shown in Figure 3.

- Ensure the IP fabric underlay is in place. To see the steps required to configure an IP fabric on a leaf device, see IP Fabric Underlay Network Design and Implementation.
- Confirm that your IBGP overlay is up and running. To configure an IBGP overlay on your leaf device, see Configuring IBGP for the Overlay.
- Configure a VRF routing instance to extend EBGP peering
to an end system. Specify a route target, route distinguisher, and
the IP address and ASN (AS 4220000001) of the IP-connected end system
to allow the leaf device and end system to become BGP neighbors. To
learn how to implement IP multihoming, see Multihoming an IP-Connected End System Design and Implementation.
Note Each VRF routing instance in the network should have a unique route distinguisher. In this reference design, the route distinguisher is a combination of the loopback interface IP address of the device combined with a unique identifier to signify the device. For VRF 5, the loopback interface IP address on Leaf 10 is 192.168.1.10 and the ID is 15, making the route distinguisher 192.168.1.10:15.
Leaf 10:
set routing-instances VRF_5 instance-type vrfset routing-instances VRF_5 interface et-0/0/15.0set routing-instances VRF_5 protocols bgp group END_SYSTEM_1 type externalset routing-instances VRF_5 protocols bgp group END_SYSTEM_1 neighbor 172.16.104.2 peer-as 4220000001set routing-instances VRF_5 route-distinguisher 192.168.1.10:15set routing-instances VRF_5 vrf-target target:62273:16776000 - Create a policy to match direct routes and BGP routes.
Leaf 10:
set policy-options policy-statement ADVERTISE_EDGE_ROUTES term TERM_1 from protocol bgpset policy-options policy-statement ADVERTISE_EDGE_ROUTES term TERM_1 then acceptset policy-options policy-statement ADVERTISE_EDGE_ROUTES term TERM_2 from protocol directset policy-options policy-statement ADVERTISE_EDGE_ROUTES term TERM_2 then acceptset policy-options policy-statement ADVERTISE_EDGE_ROUTES term TERM_3 then reject - Complete your configuration of the VRF routing instance
by configuring the EVPN type-5 option. To implement this feature,
configure EVPN to advertise direct next hops such as the IP-connected
end system, specify VXLAN encapsulation, assign the VNI, and export
the BGP policy that accepts the end system routes.
Leaf 10:
set routing-instances VRF_5 protocols evpn ip-prefix-routes advertise direct-nexthopset routing-instances VRF_5 protocols evpn ip-prefix-routes encapsulation vxlanset routing-instances VRF_5 protocols evpn ip-prefix-routes vni 16776000set routing-instances VRF_5 protocols evpn ip-prefix-routes export ADVERTISE_EDGE_ROUTES
Verifying the Routed Overlay on a Leaf Device
The operational mode command output included in the following sections assumes you have configured IP multihoming as explained in Multihoming an IP-Connected End System Design and Implementation.
To verify that the routed overlay on a leaf device is working, perform the following:
- Verify that the IPv4 routes from the VRF are converted
to EVPN type-5 routes are advertised to EVPN peers.
user@leaf-10> show evpn ip-prefix-database l3-context VRF_5
L3 context: VRF_5
IPv4 to EVPN Exported Prefixes
Prefix EVPN route status 172.16.104.0/30 Created 192.168.3.4/32 Created ## Note: this is the Lo0 interface of a end system
IPv6 to EVPN Exported Prefixes
Prefix EVPN route status 2001:db8::172:19:4:0/126 Created 2001:db8::192:168:3:4/128 Created ## Note: this is the Lo0 interface of an end systemEVPN to IPv4 Imported Prefixes. These are received as a type-5 route, and converted back to IPv4. Prefix Etag 172.16.104.4/30 0 ## From Leaf 11 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 172.16.104.8/30 0 ## From Leaf 12 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12 192.168.3.4/32 0 ## The loopback address of the end system Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12
EVPN to IPv6 Imported Prefixes
Prefix Etag 2001:db8::172:19:4:4/126 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 2001:db8::172:19:4:8/126 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12 2001:db8::192:168:3:4/128 0 Route distinguisher VNI/Label Router MAC Nexthop/Overlay GW/ESI 192.168.1.11:10 16776001 06:ac:ac:23:0c:4e 192.168.1.11 192.168.1.12:10 16776002 06:ac:ac:24:18:7e 192.168.1.12
- View the VRF route table for IPv4, IPv6, and EVPN to verify
that the end system routes and spine device routes are being exchanged.
user@leaf-10> show route table VRF_5
This is the IPv4 section
VRF_5.inet.0: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 172.16.104.0/30 *[Direct/0] 01:33:42 > via et-0/0/15.0 172.16.104.1/32 *[Local/0] 01:33:42 Local via et-0/0/15.0 172.16.104.4/30 *[EVPN/170] 01:10:08 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 172.16.104.8/30 *[EVPN/170] 00:01:25 > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 192.168.3.4/32 *[BGP/170] 01:33:39, localpref 100 AS path: 4220000001 I validation-state: unverified, > to 172.16.104.2 via et-0/0/15.0 [EVPN/170] 01:10:08 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [EVPN/170] 00:01:25 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0
This is the IPv6 section
VRF_5.inet6.0: 6 destinations, 8 routes (6 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 2001:db8::172:19:4:0/126 *[Direct/0] 01:33:33 > via et-0/0/15.0 2001:db8::172:19:4:1/128 *[Local/0] 01:33:33 Local via et-0/0/15.0 2001:db8::172:19:4:4/126 *[EVPN/170] 01:10:08 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 2001:db8::172:19:4:8/126 *[EVPN/170] 00:01:25 > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 2001:db8::192:168:3:4/128 *[BGP/170] 01:33:28, localpref 100 AS path: 4220000001 I, validation-state: unverified > to 2001:db8::172:19:4:2 via et-0/0/15.0 [EVPN/170] 01:10:08 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [EVPN/170] 00:01:25 to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 fe80::231:4600:ae2:f06c/128 *[Local/0] 01:33:33 Local via et-0/0/15.0
This is the EVPN section. EVPN routes use the following convention:
Type:Route Distinguisher::0::IP Address::Prefix/NLRI Prefix
VRF_5.evpn.0: 12 destinations, 36 routes (12 active, 0 holddown, 0 hidden) + = Active Route, - = Last Active, * = Both 5:192.168.1.10:10::0::172.16.104.0::30/248 *[EVPN/170] 01:33:42 Indirect 5:192.168.1.10:10::0::192.168.3.4::32/248 *[EVPN/170] 01:33:39 Indirect 5:192.168.1.11:10::0::172.16.104.4::30/248 *[BGP/170] 01:10:08, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 01:09:54, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 01:10:08, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 01:09:39, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 ... 5:192.168.1.12:10::0::192.168.3.4::32/248 *[BGP/170] 00:01:20, localpref 100, from 192.168.0.1 AS path: 4220000001 I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 00:01:21, localpref 100, from 192.168.0.2 AS path: 4220000001 I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 00:01:25, localpref 100, from 192.168.0.3 AS path: 4220000001 I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 00:01:17, localpref 100, from 192.168.0.4 AS path: 4220000001 I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0
5:192.168.1.10:10::0::2001:db8::172:19:4:0::126/248 *[EVPN/170] 01:33:33 Indirect 5:192.168.1.10:10::0::2001:db8::192:168:3:4::128/248 *[EVPN/170] 01:33:28 Indirect 5:192.168.1.11:10::0::2001:db8::172:19:4:4::126/248 *[BGP/170] 01:10:08, localpref 100, from 192.168.0.1 AS path: I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 01:09:54, localpref 100, from 192.168.0.2 AS path: I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 01:10:08, localpref 100, from 192.168.0.3 AS path: I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 [BGP/170] 01:09:39, localpref 100, from 192.168.0.4 AS path: I, validation-state: unverified > to 172.16.10.2 via ae1.0 to 172.16.10.6 via ae2.0 ... 5:192.168.1.12:10::0::2001:db8::192:168:3:4::128/248 *[BGP/170] 00:01:20, localpref 100, from 192.168.0.1 AS path: 4220000001 I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 00:01:21, localpref 100, from 192.168.0.2 AS path: 4220000001 I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 00:01:25, localpref 100, from 192.168.0.3 AS path: 4220000001 I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0 [BGP/170] 00:01:17, localpref 100, from 192.168.0.4 AS path: 4220000001 I, validation-state: unverified to 172.16.10.2 via ae1.0 > to 172.16.10.6 via ae2.0
Routed Overlay — Release History
Table 1 provides a history of all of the features in this section and their support within this reference design.
Table 1: Routed Overlay in the Cloud Data Center Reference Design – Release History
Release | Description |
---|---|
19.1R2 | QFX10002-60C and QFX5120-32C switches running Junos OS Release 19.1R2 and later releases in the same release train support all features documented in this section. |
18.4R2 | QFX5120-48Y switches running Junos OS Release 18.4R2 and later releases in the same release train support all features documented in this section. |
18.1R3-S3 | QFX5110 switches running Junos OS Release 18.1R3-S3 and later releases in the same release train support all features documented in this section. |
17.3R3-S1 | All devices in the reference design that support Junos OS Release 17.3R3-S1 and later releases in the same release train also support all features documented in this section. |