Type 5 EVPN/VXLAN GPU Backend Fabric Implementation – Control Plane

Fabric Underlay Control Plane Implementation Options

There are different options to implement the underlay in an EVPN/VXLAN fabric, depending on design goals, operational preferences, and hardware capabilities.

• IPv4 addresses (/31 subnet masks) numbered interfaces
• IPv6 addresses (/127 subnet masks) numbered interfaces
• IPv6 link-local addresses (unnumbered interfaces), with BGP neighbour auto-discovery based on IPV6 neighbour discovery (RFC 4861) and IPv4 route advertisement via IPv6 next-hops (RFC 5549) for IPv4 overlays.

Table 8. Comparison of Underlay Control Plane Implementation Options in EVPN/VXLAN Fabrics

Implementation Options	IPv4 /31	IPv6 /127	IPv6 Link-Local (RFC 4861) RECOMMENDED
Leaf to Spine Interface Addressing	Statically configured /31 IPv4 addresses	Statically configured routable (non-link local) /127 IPv6 addresses	Automatically assigned Link-local IPv6 (no global addressing needed)
BGP Peer Configuration	Explicit neighbor config per interface using IPv4 addresses of directly connected interface.	Explicit neighbor config per interface using routable IPv6 addresses of directly connected interface.	No explicit neighbor config required Uses interface-scoped link-local discovery Link-local IPv6 + interface-scoped BGP config (fe80::1%et-0/0/0)
Benefits	Simple Widely supported Low config overhead	Avoids IPv4 exhaustion IPv6-native underlay Aligns with modern fabrics	Zero IP allocation needed Ideal for massive fabrics Minimal IPAM
Drawbacks	No IPv6 ready	Needs dual-stack address planning	Traceroute visibility reduced
Use cases / Industry Trend	IPv4 /31 remains the most widely used in enterprise and service provider fabrics. For most enterprise and traditional data center fabrics, IPv4 /31 remains the recommended and most straightforward underlay option.	IPv6 /127 is gaining traction in dual-stack environments and in organizations preparing for IPv6 transitions. It conserves IPv4 space and offers a clean separation between infrastructure and tenant traffic.	IPv6 Link-Local / BGP auto-discovery is trending in hyperscaler, telco, and modern leaf-spine fabrics, especially where address scale or IPAM simplicity is critical. Many cloud providers (Azure, AWS internal fabrics, Meta, etc.) use unnumbered underlays. This option is becoming the preferred option for hyper scaling, IP-scarce, or automation-heavy fabrics with experienced ops teams, where managing IPs is a burden.

Note:

This JVD focuses on the IPv6 link-local underlay, which is the preferred design choice. For details on implementing the underlay using statically configured /31 IPv4 addresses or statically configured routable (non-link-local) /127 IPv6 addresses, refer to Appendix B – IPv4 Overlay over IPv4 Underlay Fabric Implementation and Appendix C – IPv6 Overlay with Static Addresses Over IPv6 Underlay Fabric Implementation, respectively.

In all three options, EBGP sessions are established between the leaf and spine nodes using the addresses assigned to the interfaces connecting the nodes (e.g. et-0/0/0.0). These sessions advertise the addresses assigned to the loopback interfaces, which will be used in the overlay.

Configuring IPv6 with link-local addressing in the underlay simplifies network design and reduces operational complexity, especially in large-scale fabrics where scalability and automation are key. By removing the need to assign and manage global IP addresses on leaf-to-spine links, this approach eliminates a major source of administrative overhead and prevents IPv4 address exhaustion. Each link automatically generates unique addresses, streamlining both deployment and ongoing maintenance. This is a significant advantage in environments with thousands of interfaces, where traditional IP management becomes a scaling bottleneck.

From a design and operational perspective, unnumbered underlays align perfectly with modern data center and AI fabric principles. They reduce configuration touch points, lower the chance of human error, and support dynamic, automation-friendly environments. For customers prioritizing agility, scalability, and simplified management, such as hyperscalers, telcos, and AI-driven infrastructures, IPv6 link-local fabrics not only meet today’s technical requirements but also provide a future-proof foundation that supports growth without requiring continuous IP planning.

Control Plane Implementation with IPv6 Link-Local Underlay

The underlay EBGP sessions are set up using unnumbered links, also referred to as BGP auto-discovery or BGP auto-peering, which allows devices to dynamically discover directly connected neighbors and form BGP sessions using IPv6 link-local addresses. This design leverages Junos OS support for:

• RFC 4861: Neighbour Discovery for IP version 6 (IPv6)
• RFC 2462: IPv6 Stateless Address Autoconfiguration

Traditionally, BGP requires explicit configuration of neighbors, Autonomous System (AS) numbers, and routing policies to control route exchanges. With BGP unnumbered peering, neighbors are discovered dynamically, and BGP sessions are established automatically, eliminating the need for manual configuration and enabling faster, more scalable underlay deployments in EVPN/VXLAN data center fabrics.

The interfaces between leaf and spine nodes do not require explicitly configured IP addresses. It is sufficient to enable IPv6 (e.g. family inet6). Enabling IPv6 on an interface automatically assigns a link-local IPv6 address, which is then advertised through standard router advertisements as part of the IPv6 Neighbor Discovery process. This simplifies configuration and eliminates the need for manual IP addressing on leaf–spine links. All leaf and spine nodes are also configured with IPv6. addresses on the loopback interface (lo0.0).

Neighbor discovery uses standard IPv6 mechanisms to learn the link-local addresses of directly connected neighbors. These addresses are then used to automatically establish EBGP sessions.

The EBGP configuration for this model includes the local Autonomous System (AS) number, a list of accepted remote Autonomous System (AS) numbers, the list of interfaces where dynamic BGP neighbors with be accepted, and the export policy that allows the advertisement of routes to reach all the leaf and spine nodes in the fabric. These routes are standard IPv6 unicast advertising the IPv6 addresses assigned to the loopback interface (lo0.0). Peer-auto-discovery using IPv6-nd must also be enabled for the BGP group.

Although this approach requires some changes in the traditional way of configuring BGP, it offers significant operational advantages in highly scalable environments. By eliminating the need to assign and manage IP addresses on point-to-point links, this model simplifies IP planning and is ideal for large-scale, automated EVPN/VXLAN deployments.

Fabric Overlay Control Plane Implementation Options

Like the underlay, the overlay in an EVPN/VXLAN fabric can be implemented using either IPv4 or IPv6 addresses, depending on design goals, operational preferences, and hardware capabilities as summarized in table 9. Both options can be implemented over the recommended IPv6 link-local addressing with BGP auto-discovery in the underlay.

Table 9. Comparison of Overlay Control Plane Implementation Options in EVPN/VXLAN Fabrics

Implementation Options	IPv4 (RFC5549 with IPv6 underlay)	IPv6 RECOMMENDED
VTEP Tunnel Endpoint Addresses (leaf node loopback interface addresses)	Statically configured /32 IPv4 addresses	Statically configured routable (non-link local) /128 IPv6 addresses
Server to Leaf Nodes link prefix	/31 prefixes	/127 prefixes
Server addresses	Statically configured /31 IPv4 addresses	Statically configured /127 IPv6 addresses Autoconfigured /64 IPv6 addresses using SLAAC (RECOMMENDED)
Header Size (VXLAN + IP)	Lower overhead (UDP+IPv4 = ~28B)	Higher overhead (UDP+IPv6 = ~48B)
BGP Peer Configuration	Explicit neighbor config per interface using IPv4 addresses of directly connected interface. Requires	Explicit neighbor config per interface using routable IPv6 addresses of directly connected interface.
Benefits	Simple Widely supported	Avoids IPv4 exhaustion Scalability and future-proofing, especially pertinent to the demands of AI/ML data centers.

Note: This JVD focuses on the IPv6 overly, which is the preferred design choice.

Configuring IPv6 in the overlay ensures alignment with the IPv6-based underlay and avoids the operational complexity of running BGP sessions over mixed-protocol paths. When overlay loopbacks and EVPN sessions are IPv6-based, the routing model remains consistent across all layers, and control plane reachability does not require translation between IPv6 routes and IPv4 transport sessions. Choosing an IPv6 overlay also eliminates the need for dual-stack configurations, resulting in a cleaner deployment model and providing a foundation for extending IPv6-based services, such as SLAAC and per-tenant prefix assignment, across the fabric.

Using IPv4 in the overlay with IPv6 underlay requires the implementation of RFC5549 (Advertising IPv4 Network Layer Reachability Information with an IPv6 Next Hop) which adds a layer of complexity of the solution. However, if IPv4 between the servers and leaf nodes is required, the details for this are covered in Appendix B – IPv4 Overlay over IPv4 Underlay Fabric Implementation.

Control Plane Implementation with IPv6 Overlay

EBGP sessions between leaf and spine nodes are established using the loopback IPv6 addresses advertised by the underlay, with BGP multihop enabled. These overlay sessions carry the IPv6 prefixes corresponding to the point-to-point links between GPU servers and leaf nodes.

On each leaf node, the interfaces connecting to the GPU servers are placed in tenant-specific IP VRFs. The associated IPv6 prefixes are then advertised as EVPN Type-5 routes, each containing a tenant-specific VNI. These routes provide Layer 3 reachability between GPUs assigned to the same tenant, even when distributed across multiple servers and racks, and are installed in the appropriate VRF routing tables.

EVPN Type 5 routes are used to advertise Layer 3 prefixes across the EVPN/VXLAN fabric without requiring destination MAC learning or IRB interfaces. These routes follow the BGP EVPN IP Prefix Route specifications described in RFC9136, and include the IPv6 prefix, route target, route distinguisher, and the VTEP next-hop information, enabling routed connectivity across the fabric. By decoupling routing from MAC learning, Type 5 routes simplify control plane operations and maintain clean tenant separation through BGP extended communities. Each route includes the information summarized in Table 10.

Table 10. EVPN Type 5 Route Fields Description

Field	Description
Route Type	IP Prefix Route (Type 5)
Route Distinguisher (RD)	RD of advertising PE (e.g., based on loopback IP) to make the route unique across the fabric
Ethernet Tag ID	0 (because it’s not associated with a specific VLAN or MAC-VRF)
IP Prefix	The advertised IP prefix being advertised (e.g., FC00:1:1:1::/64)
Prefix Length	The length of the IP prefix (e.g. 64)
Label	VXLAN VNI (e.g., 1) identifying the virtual routing domain
Next hop	Loopback address of the advertising leaf node
Extended Community	Route-target to identify the associated tenant VRF (e.g., target:65000:1) Router-mac to identify the MAC address of the advertising VTEP,
Other BGP Attributes	BGP attributes like origin, AS-path, local-preference, etc.

Control Plane Summary

Table 11. Connections Summary

Option	GPU server to leaf node links	Leaf to spine node links	Leaf and spine nodes loopback interface addresses
IPv4 underlay and IPv4 overlay	Statically configured IPv4 address	Statically configured IPv4 addresses	Statically configured IPv4 addresses
IPv6 underlay and IPv6 overlay	Statically configured IPv6 address	Statically configured IPv6 addresses	Statically configured IPv6 addresses
IPv6 Link-Local underlay and IPv4 overlay (RFC 5549)	Statically configured IPv4 address.	Automatically assigned IPv6 link local addresses	Statically configured IPv4 addresses
RECOMMENDED IPv6 Link-Local underlay and IPv6 overlay	Dynamically assigned IPv6 address using SLAAC (Stateless Address Autoconfiguration)	Automatically assigned IPv6 link local addresses	Statically configured IPv6 address

Table 12. EVPN/VXLAN options summary

Option	GPU server to leaf node links	Leaf to spine node links	Leaf and spine nodes loopback interface addresses	Underlay BGP sessions	Overlay BGP sessions
IPv4 underlay and IPv4 overlay	Statically configured IPv4 address	Statically configured IPv4 addresses	Statically configured IPv4 addresses	Statically configured IPv4 neighbors	Statically configured IPv4 neighbors using loopback interfaces.
IPv6 underlay and IPv6 overlay	Statically configured IPv6 address	Statically configured IPv6 addresses	Statically configured IPv6 addresses	Statically configured IPv6 neighbors	Statically configured IPv6 neighbors using loopback interfaces.
IPv6 Link-Local underlay and IPv4 overlay	Statically configured IPv4 address.	Automatically assigned IPv6 link local addresses	Statically configured IPv4 addresses	Automatically discovered IPv6 neighbors	Statically configured IPv4 neighbors using loopback interfaces.
RECOMMENDED IPv6 Link-Local underlay and IPv6 overlay	Dynamically assigned IPv6 address using SLAAC (Stateless Address Autoconfiguration)	Automatically assigned IPv6 link local addresses	Statically configured IPv6 address	Automatically discovered IPv6 neighbors using link local addresses.	Statically configured IPv6 neighbors using loopback interfaces.