Data Center Interconnect / EVPN Gateways

Historically, enterprises have leveraged Data Center Interconnect (DCI) technology as a building block for business continuity, disaster recovery (DR), or Continuity of Operations (COOP). These service availability use cases primarily relied on the need to connect geographically separated data centers with Layer-2 connectivity for application availability and performance.

With the rise of highly virtualized Software-Defined Data Centers (SDDC), cloud computing, and more recently, edge computing, additional use cases have emerged:

  • Colocation Expansion: Share compute and storage resources to colocation data center facilities.
  • Resource Pooling: Share and shift applications between data centers to increase efficiency or improved end-user experience.
  • Rapid Scalability: Expand capacity from a resource-limited location to another facility or data center.
  • Legacy Migration: Gracefully move applications and data off older and inefficient equipment and architecture to more efficient, higher-performing, and cost-effective architecture.

Users can use Apstra AOS to deploy and manage a vendor inclusive DCI solution that is simple, flexible, and Intent-Based. AOS utilizes the standards-based MP-BGP EVPN with VXLAN, which has achieved broad software and hardware adoption in the networking industry. Customers can choose from a vast selection of cost-effective commodity hardware from traditional vendors to white-box ODMs and software options ranging from conventional vendor integrated Network Operating Systems (NOS) to disaggregated open source options.

EVPN VXLAN is a standards based (RFC-7432) approach for building modern data centers. It incorporates both data plane encapsulation (VXLAN) and a routing control plane (MP-BGP EVPN Address Family) for extending L2 broadcast domains between hosts as well as Layer-3 (L3) routed domains in spine-leaf networks. Relying on a pure L3 underlay for routing of VXLAN tunneled traffic between VXLAN Tunnel Endpoints (VTEPs), EVPN introduces a new address family to the MP-BGP protocol family and supports the exchange of MAC/IP addresses between VTEPs. The advertisement of endpoint MACs and IPs, as well as “ARP/ND-suppression”, eliminates the need for a great majority of Broadcast/Unknown/Multicast (BUM) traffic and relies upon ECMP unicast routing of VXLAN, from Source VTEP to Destination VTEP. This ensures optimal route selection and efficient load-sharing of forwarding paths for overlay network traffic.

Just as EVPN VXLAN works within a single site for extending Layer-2 (L2) between hosts, the DCI feature enables L2 connectivity between sites. The AOS DCI feature enables the extension of Layer-2 or Layer-3 services between data centers for disaster recovery, load balancing of Active-Active sites, or even for facilitating the migration of services from one DC to another.

AOS DCI Options

Apstra AOS implemented the DCI feature to be open and flexible, with three different use cases.

  1. DCI using Over the Top
  2. DCI using Gateways (GW)
  3. DCI using Autonomous System Border Router (ASBR)

There are two important details to note, regardless of which deployment option you choose.

  1. You can extend AOS DCI to other AOS managed data centers, non-AOS managed data centers, or even to legacy non-Spine-Leaf devices.
  2. AOS implementation and behavior is exactly the same thing in all three cases.

Whether the remote end is another DCI GW or an ASBR, it is transparent to AOS. AOS manages neither the GWs nor ASBRs. Should you have any questions, consult your Apstra Solutions Architect (SA) or Systems Engineer (SE) to determine which option is best for your organization.

DCI Using Over the Top

DCI “Over the Top” is a transparent solution, meaning EVPN routes are encapsulated into standard IP and hidden from the underlying transport. This makes the extension of services simple and flexible and is often chosen because it can be implemented by data center teams with little to no coordination with WAN or Service Provider groups. This reduces the implementation times and internal company friction. However, the tradeoff is scalability and resilience.


DCI Using Gateway

Building upon the AOS Remote EVPN Gateway capability, one can optionally specify the Remote EVPN Gateway to be an external router (GW) in the same site, thus extending the EVPN attributes to said GW. This solution creates a fault domain per site, preventing failures from affecting convergence in remote sites and creating multiple fault domains. IP/MAC endpoint tables for remote sites are processed and held in state on an external router GW. WAN QoS and security can also be implemented, along with optimizations made available by the transport technology (e.g., MPLS TE). However, this solution is more operationally complex, requiring additional hardware and cost.



Using the AOS Remote EVPN Gateway capability, one can optionally specify the Remote EVPN Gateway to be an ASBR WAN Edge Device. It is an end-to-end EVPN, that enables uniform encapsulation, removes the dedicated GW requirement. It is also operationally complex but has greater scalability as compared to both “DCI Using Gateway” and “Over the Top”.


AOS EVPN Gateways DCI Implementation

New in version 3.2: You can extend Security Zones and Virtual Networks (VN) to span across AOS-managed Blueprints (across pods) or to remote networks that are not managed by AOS (across datacenters). This feature introduces the EVPN Gateway (GW) role, which could be a switch that participates in the fabric or RouteServer(s) on a server that is connected to the fabric.

EVPN Gateways Use Cases

  • Span L3 isolation domains (VRFs / Security Zones) to multiple AOS-managed PODs(blueprints) or extend to remote EVPN domains.
  • Provide L2 domain extensions for L2VNI/Virtual networks.
  • Help extend EVPN domain from AOS to AOS-managed and AOS to unmanaged PODs.
  • No VXLAN traffic termination on the spines. This can be achieved by connecting external routers on spines. This is to support IPv4 (underlay) external connectivity. Here Spines don’t need to terminate VXLAN traffic unlike border leafs when connected to external routers. This new concept is fundamentally different from Connectivity Points for External Routers. In a nutshell, using this can exchange IPv4 routes to remote VTEPs (in the default Security Zone/VRF) and only L3 connectivity is required:

Over the Top

When BGP EVPN peering is done “over the top”, the Data Center Gateway (DC-GW) is a pure IP transport function and BGP EVPN peering shall be established between gateways in different DCs.

The next sections describes the procedures for interconnecting two or more BGP based Ethernet VPN (EVPN) sites in a scalable fashion over an IP network. The motivation is to support extension of EVPN sites without having to rely on typical Data Center Interconnect (DCI) technologies like MPLS/VPLS, which are often difficult to configure, sometimes proprietary, and likely legacy in nature.

“Over the Top” is a simple solution that only requires IP routing between DCs and an adjusted MTU to support VXLAN encapsulation between gateway endpoints. In such an implementation, EVPN routes are extended end-to-end via MP-BGP between sites. Multi-hop BGP is enabled with the assumption that there will be multiple L3 hops between sites over a WAN, otherwise the default TTL will decrement to 0 and packets will be discarded and won’t make it to the remote router. AOS automatically renders the needed configuration to address these limitations.


This design merges the separate EVPN-VXLAN domains and VXLAN tunnels between sites. Merging of previously separate EVPN domains in different sites realizes the benefit of extending L2 and L3 (VRF) services between sites, but also renders the sites as a single fault domain. So a failure in one site will necessarily be propagated. Also, anytime one stretches L2 across the WAN between sites, you are also extending the flood domain and along with it, all broadcast traffic over your costly WAN links. At this time, this solution does not offer any filtering or QoS.


When individual sites are managed by separate AOS Blueprints, or only one site is AOS managed, then extended Security Zones (VRFs) and Virtual Networks (L2 and/or L3 defined VLANs/subnets) have to be created and managed independently in each site. This creates administrative overhead and requires manual mapping of VRFs and VNs between sites.

This “Over the Top” solution is the easiest to deploy, requires no additional hardware and introduces no additional WAN config other than increasing the MTU. It is the most flexible and has the lowest barrier to entry. However, the downside is that there is a single EVPN Control plane and a routing anomaly in one site will affect convergence and reachability in the other site(s). As previously mentioned, the extension of L2 flood domains also implies that a broadcast storm in one site would extend to the other site(s).

With any DCI implementation, careful resource planning and coordination is required. Adding more sites requires an exponential increase in such planning and coordination. VTEP loopbacks in the underlay need to be leaked. VNIDs must match between sites and in some cases, additional Route Targets (RTs) have to be imported. This is all covered in detail later in this document.

Data Plane Extension: L3 Extension

VXLAN Network IDs (VNIDs) are a part of the VXLAN header that identify unique VXLAN tunnels, each of which are isolated from the other VXLAN tunnels in an IP network. L3 packets can be encapsulated into a VXLAN packet or L2 MAC frames can be encapsulated directly into a VXLAN packet. In both cases, a unique VNID will be associated with either the L3 subnet, or the L2 domain. When extending either L3 or L2 services between sites, one is essentially stitching VXLAN tunnels between sites. VNIDs therefore need to match between sites.

It is important to understand that a particular VNID will be associated with only one VRF (or SZ in AOS terminology). VNIDs exist within a VRF. They are tied to a VRF. For L3 services, the stitching, or extending, of each VNID is done with the export and import of RTs within a Security Zone (VRF). L3 subnets (routes) are identified via RTs. All VNIDs are exported automatically at the EVPN gateway (edge) towards the WAN. Conversely, RTs of the same value will automatically be imported at the EVPN gateway (edge) coming into the fabric. So if one coordinates the L3 VNIDs at one site to match the other, no additional configuration is needed.


In the image above, no additional export or import is required. Everything is automatically exported (Export All) and because the RTs match, they are automatically imported.

However, if a VNID in DC1 is different from a VNID in DC2, then one must import the RTs respectively. Each respective gateway will still automatically import RTs of the same value. In the example below, an additional step of manually adding the RTs from the other site is required.


Data Plane Extension: L2 Extension

In AOS, when creating a VN, one can create the VN to be a pure L2 service, meaning no L3 anycast gateway is instantiated. This is for either a rack local L2 (VLAN on server facing ports contained within a rack), or one can extend the L2 flood and broadcast domain between racks by selecting VXLAN and then selecting the racks you want the L2 domain extended to. This L2 domain has it’s own VNID, and the MAC frames (as opposed to IP packets) are encapsulated into the VXLAN header with the VNID of the L2 domain.

The same principles apply in that all VNIDs are exported at the EVPN gateway (in this case Type-2 routes/MAC addresses), and matching RTs are automatically imported. However, the location of importing and exporting RTs is not at the Security Zone level, but instead at the VN itself.

AOS Workflow

Control Plane Extension: The EVPN Gateway

AOS uses the concept of an an “EVPN Gateway”. This device can theoretically be a leaf, spine or superspine fabric node, as well as the DCI device. EVPN Gateways separate the fabric-side from the network that interconnects the sites and masks the site-internal VTEPs.

In AOS, an EVPN Gateway is a device that belongs to and resides at the edge of an EVPN fabric which is also attached to an external IP network. In a AOS EVPN blueprint, this will always be a border-leaf device. The EVPN Gateway of one data center, establishes BGP EVPN peering with a reciprocal EVPN gateway, or gateways, in another data center. The “other” EVPN gateway is the “Remote EVPN Gateway” in AOS terminology. The Local EVPN Gateway is assumed to be one of the AOS managed devices in the blueprint, and is selected during the creation of the “Remote EVPN Gateway”. The Local EVPN Gateway will be the border-leaf switch with one or more external routing connections for traffic in and out of the EVPN Clos fabric.

Due to this capability, a Local EVPN Gateway (always an AOS managed switch) can be configured to peer with a non AOS managed, or even a non Spine-Leaf device, in another DC. The EVPN Gateway BGP peering is used to carry all EVPN attributes from inside a pod, to outside the pod. In AOS, each DC is represented by an autonomous blueprint and if two or more sites are under AOS management, you still have to configure each and every site to point to the “Remote EVPN Gateway(s)” in other sites. Apstra recommends that the user creates multiple, redundant EVPN Gateways for each DC. There is also currently a full mesh requirement between EVPN gateways, although in future releases this requirement will be removed.

Underlay VTEP Route Advertisements

It is required that the underlay reachability to VTEP IP addresses or an equivalent summary route, is established reciprocally. Each site must advertise these VTEP loopbacks from within the “Default” Security Zone (SZ) into the exported BGP (IPv4) underlay advertisements.

In AOS, this is done in the user’s blueprint, in the “Staged”, “Virtual”, and “Security Zone” tabs. Open the “default” Security Zone VRF and click the “Edit” icon to edit the External Connectivity Point for the “default” Security Zone VRF.

Under “Routing Policies”, verify the “Export Policy” has “Loopbacks” checked.


Creating AOS EVPN Gateways

Remote EVPN Gateway is a logical function that could be instantiated anywhere and on any device, and requires support for BGP in general, L2VPN/EVPN AFI/SAFI specifically. To establish a BGP session with an EVPN gateway, IP connectivity, as well as connectivity to TCP port 179 (BGP TCP port allocated by IANA), should be available.


For resilience purposes, we recommended having at least two remote gateways for the same remote EVPN domain.

  1. In AOS, EVPN Gateways are defined in the user’s blueprint, in the “Staged”, “Virtual”, and “Remote EVPN Gateways” tabs.

  2. To add an EVPN Gateway, click Create Remote EVPN Gateway.

  3. Fill in the following information for the Remote EVPN Gateway.

    1. Name - An arbitrary name for the Remote EVPN Gateway
    2. IP Address - The IP Address for the Remote EVPN Gateway
    3. ASN - The BGP Autonomous System Number (ASN) for the Remote EVPN Gateway
    4. TTL - The BGP multi-hop “Time to Live” (maximum number of L3 hops) value for the BGP session with the Remote EVPN Gateway. This is optional and if omitted, AOS will not configure a TTL for this BGP session on the device and the device will use its default.
    5. Password - BGP TCP authentication password. This is optional and if omitted, AOS will not configure a password for this BGP session on the device.
    6. Keep-alive Timer - The BGP keep-alive timer. This is optional and if omitted, AOS will not configure a keep-alive for this BGP session on the device and the device will use its default.
    7. Hold-time Timer - The BGP hold-time timer. This is optional and if omitted, AOS will not configure a hold-time for this BGP session on the device and the device will use its default.
  4. Select the Local Gateway Nodes. These are the devices in the AOS blueprint that will be configured with a Local EVPN Gateway. You can select one or more devices here to peer with the configured “Remote EVPN Gateway”.


    Apstra recommends using multiple border-leafs which have direct connections to External Routers.

  5. The new Remote EVPN Gateways will be listed on the page.


    Apstra recommends using multiple Remote EVPN Gateways. To configure additional Remote EVPN Gateways, the user can click Create Remote EVPN Gateway to add another Remote EVPN Gateways.

  6. From the blueprint, “Staging” tab, the use can commit changes to deploy them to the devices in the AOS blueprint.


Once the change is deployed, AOS will monitor the BGP session for the Remote EVPN Gateways. Any anomalies for this will be shown as a “External Routing” service anomaly.


If the user is configuring the Remote EVPN Gateway(s) to another AOS blueprint, the user will need to configure and deploy the Remote EVPN Gateway(s) separately in that AOS blueprint.


Enhanced Security Zone

The installation of EVPN routes is mainly governed by RT (route-target) import/export policies on devices which are part of extended service. As per this feature can add additional import and export route-targets used by AOS for Security Zones/VRFs. Please find below related screenshot:



The default route-target generated by AOS for security zones is <L3 VNI>:1. This can’t be altered.

To make sure correct routes are received at VTEP please make sure L3VNIs and route target are identical between the AOS blueprint and remote EVPN domains.

Enhanced Virtual Networks

You can now add additional import and export route-targets used by AOS for Virtual Networks(VN).


The default route target generated by AOS for AOS for Virtual Networks is <L2 VNI>:1. You can’t alter this.

For Intra-VNI communication L2VNI specific RT is used. The import RT is used to determine which received routes are applicable to a particular VNI. L2 VNIs need to be the same between the blueprint and remote domains for connectivity to be established.

You also need to ensure that the SVI subnets are identical across domains.

Remote gateway Topology representation

Remote EVPN gateways are represented on the topology view as cloud elements with dotted line connections to the blueprint elements with which BGP sessions are established. Please find the related screenshot as below: