Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Solution Architecture

The 5-stage Fabric with Juniper Apstra is an EVPN-VXLAN based validated design that is based on ERB network architecture. It consists of a superspine connected to the Pods. The superspine only performs IP forwarding and relaying of routes just as the spines in the Pods do. Hence the superspines and spines in 5-stage Fabric are called Lean superspines and lean spines.

As mentioned in this JVD, 5-stage Fabric is adopted for large scale datacenter designs that have requirement for large datastores with connecting compute nodes. This JVD validates key features such as RDMA over Converged Ethernet version 2 (RoCEv2), Multicast, and other base features in the 5-stage fabric.

This JVD will walk through the high-level steps required to configure a 5-stage Data center. Refer to Figure: 5-Stage Datacenter Architecture with Apstra for the reference architecture and hardware used to validate the 5-stage Fabric. The switches are considered the baseline design of this JVD, though other switches are qualified for these roles, as documented below.

The provisioning of this datacenter was done using Juniper Apstra’s Datacenter Reference design that provisions native config. Apstra uses pre-defined Intent Based Analytics to provide real-time insight into the network.

Note:

The pods hosting the border leaves can be directly connected to the superspine. The border leaves will then need to be manually configured, as this is not yet supported in Apstra Datacenter Reference design architecture.

Below is the lab setup network design for validating 5-stage Fabric. Refer Figure: Test Bed Configuration for more details about this test bed.

Figure 1: 5-Stage Datacenter Lab Network A screenshot of a computer Description automatically generated

Juniper Hardware and Software Components

For this solution, the Juniper products and software versions are as below.

The design documented in this JVD is considered the baseline representation for the validated solution. As part of a complete solutions suite, we routinely swap hardware devices with other models during iterative use case testing. Each switch platform validated in this document goes through the same rigorous role-based testing using specified versions of Junos OS and Apstra management software.

Juniper Hardware Components

The following switches are tested and validated to work with the 5-Stage Fabric with Juniper Apstra JVD in the following roles:

Table 1: : Platform Positioning and Roles
Platforms and Roles
Solution Server Leaf Switches Border Leaf Switches Spine Super Spine
5-stage EVPN/VXLAN Datacenter design (ERB) QFX5120-48YM QFX5130-48C (EVO) QFX5220-32CD (EVO) QFX5230-64CD (EVO)
QFX5130-32CD (EVO)   QFX5210-64C  
    QFX5120-32C  
Note:

The JVD assumes that the other model variations of the tested device in the hardware series should also work. For instance, QFX5120-48YM covers all the rest of the variations such as QFX5120-48Y since the same chipset is used. However, there are some exceptions such as QFX5130-48C and QFX5130-32CD. Please contact the Juniper account representative for more information.
Table 2: Juniper Software and Version
Juniper Software
Juniper Products Software or Image Version
Juniper Apstra 5.0.0-64
Junos OS 23.4R2-S3
Note:

The Junos OS Release 23.4R2-S3 is used for testing. Some of the decisions with respect to multicast influenced the choice of leaf switch.

Important Note: During JVD testing, Apstra generated several incremental “Check gRPC_Reset count” anomalies. The workaround is to disable gRPC_telemetry.The workaround for this issue is to disable gRPC_enabled flag (gRPC_enabled = 0) in /etc/aos/aos.conf. Then restart Apstra service from cli using command “service aos restart” as root user. This would mean that telemetry will still be collected however it will be collected using Netconf. The 5-stage JVD testing using Apstra 5.0 was carried out after implementing this workaround.

Validated Functionality

The 5-stage Fabric with Juniper Apstra was validated using the following parameters in its configuration:

  • This JVD consists of a 5-stage CLOS with an ERB network architecture using EVPN-VXLAN.
  • Servers will be connected and tested both in single-homed and multi-homed configurations.
  • In the case of multihomed ESI servers, LACP is enabled between the servers and the leaf switches.
  • Configure ESI on aggregated ethernet interfaces for multi-homed devices.
  • ECMP is configured across the fabric to minimize traffic loss.
  • Both the overlay and underlay of the fabric are built using eBGP.
  • Learn and advertise EVPN Type 2 and Type 5 routes.
  • BFD is enabled for both underlay eBGP and overlay eBGP.
  • Symmetric IRB is enabled with anycast IP address on L3-enabled leaf switches for inter-VLAN routing. For more information on the IRB model for inter-subnet forwarding in EVPN, refer to the EVPN VXLAN Guide.
  • Both IPv4 and IPv6 are enabled.
  • OISM multicast Bridge Domain Not Everywhere (BDNE) configured on leaf switches in pods for routing and forwarding traffic within Fabric and externally using border leaf switches to exchange multicast traffic with external source and receivers. For more information on juniper OISM configuration refer juniper OISM.
  • ECN and PCF QOS configuration profiles for supporting RoCEv2.
  • DHCP, loopback firewall filter and Duplicate MAC detection.
  • Inter-VRF connectivity is configured using external router to allow route leaking between VRFs, however, to achieve this configuration Apstra Connectivity templates were used to connect to the external router.

The features below are not considered part of this JVD and are not described. However, the features have been validated:

  • DCI
  • SNMP
  • Management VRF
  • Apply pristine configs to devices