Understanding MetaFabric Architecture 2.0

 

MetaFabric Architecture 2.0 is based on the integration of the Juniper Networks® Virtual Chassis Fabric and VMware NSX for vSphere to create an enterprise private cloud environment. This architecture gives you a single point of control for both the physical and virtual networking components of a network. This topic explains these MetaFabric features and concepts:

High-Performance Network Fabric

Enterprise private clouds require high-speed and high-performance network fabrics with low latency. The network fabric also needs to be easy to manage. Adding more capacity, configuration changes, and software upgrades must be plug-and-play, simple, and seamless.

Virtual Chassis Fabric is a plug-and-play Ethernet fabric technology that allows you to combine a set of switches into a single logical switch. A Virtual Chassis Fabric has the following benefits:

  • Single point of management

  • Supports software-defined networking (SDN) with Virtual Extensible LAN (VXLAN) integration

  • Supports Fast Ethernet, Gigabit Ethernet, 10-Gigabit Ethernet, and 40-Gigabit Ethernet interfaces

  • Full Layer 2, Layer 3, and multicast support

  • Equal-cost multipath (ECMP)

  • Scales up to 20 switches

  • Nonstop software upgrade (NSSU)

Virtual Chassis Fabric allows you to easily manage your data center environment. All devices are—at most—only three hops away, regardless of where they are located in the data center. Virtual Chassis Fabric offers line-rate performance for all types of workloads, whether virtual machines or bare metal servers. One of the key benefits is that Virtual Chassis Fabric supports software-defined data centers (SDDC) and SDN with VMware NSX and Juniper Networks Contrail.

Virtual Chassis Fabric is a next-generation Ethernet fabric that delivers a high-speed and high-performance network fabric through a spine and leaf architecture as shown in Figure 1.

Figure 1: MetaFabric Architecture 2.0
MetaFabric Architecture 2.0

End-to-end over-subscription is user-configurable from 1:1 to 6:1, and the end-to-end latency is less than 3 microseconds. Virtual Chassis Fabric allows the entire network fabric to be managed and operated as a single logical switch. All configuration changes and software upgrades are performed in a single place. Virtual Chassis Fabric also supports NSSU, which allows you to upgrade all member switches on a VCF with minimal network traffic disruption during the upgrade. Adding new capacity to the Virtual Chassis Fabric is as simple as cabling the new Juniper Networks QFX5100 switch and powering it up. The QFX5100 switch is automatically discovered and added to the Virtual Chassis Fabric as a new line card.

Software-Defined Networking

Enterprise private clouds need to quickly deploy applications in a multi-tier network which enables better scale and security segmentation. Each application requires custom network segmentation and security policies. VMware NSX for vSphere is an SDN solution that allows you to quickly create virtual networks that are multi-tier with segmentation and security policies.

VMware NSX for vSphere has full integration with other VMware tools such as vCenter Server, vCenter Operations Manager, and vCloud Director.

Virtualized Hosts

Enterprise private clouds need to quickly deliver virtual machines to their end users. One of the most important things is having server hardware that is listed in the VMware hardware compatibility list (HCL). For the MetaFabric Architecture 2.0 lab, we used Supermicro servers and Intel network cards.

Network-Attached Storage

One of the final tenets of enterprise private cloud is doing more with less. When it comes to storage, we want to converge the storage traffic and application traffic onto the same hardware. This means that we used network-attached storage (NAS). It is also important to choose a NAS device that is listed in the VMware HCL to ensure that everything works properly. For the MetaFabric Architecture 2.0 lab, we used a Synology NAS with iSCSI protocol. Because iSCSI uses IP, we can run both the storage traffic and application traffic across the same Virtual Chassis Fabric network.

MetaFabric Architecture 2.0 Sizing

MetaFabric Architecture 2.0 allows you to build a small enterprise private cloud or use a scale out architecture to build a hyper-scale private cloud with over 1,000,000 VMs as shown in Table 1. Whether you use the small architecture or the scale out architecture, MetaFabric Architecture 2.0 is seamless and uses the same products and technologies: QFX5100 family switches and Virtual Chassis Fabric.

The assumption is that each host is connected to the Virtual Chassis Fabric with two 10-Gigabit Ethernet connections and can support 100 VMs per host.

Table 1: MetaFabric Architecture 2.0 Sizes

Size

Network Ports

Hosts

Virtual Machines

Small

96x10-Gigabit Ethernet

48

4,800

Medium

768x10-Gigabit Ethernet

384

38,400

Large - 4 PODs

3,072x10-Gigabit Ethernet

1,536

153,600

Scale Out - 8 PODs

6,144x10-Gigabit Ethernet

3,072

307,200

Scale Out - 16 PODs

12,288x10-Gigabit Ethernet

6,144

614,400

Scale Out - 32 PODs

24,576x10-Gigabit Ethernet

12,288

1,228,800

Small-Sized Virtual Chassis Fabric

The smallest possible Virtual Chassis Fabric you can create consists of four switches: two spine switches and two leaf switches as shown in Figure 2.

Figure 2: Small-Sized Virtual Chassis Fabric with Four Members
Small-Sized Virtual Chassis Fabric with
Four Members

The small Virtual Chassis Fabric can support up to 96 10-Gigabit Ethernet ports with 48 hosts and 4,800 VMs.

Medium-Sized Virtual Chassis Fabric

Using the same Virtual Chassis Fabric technology, you can add up to 20 members into the Ethernet fabric as shown in Figure 3. Although there are 20 members in the Virtual Chassis Fabric, it is managed as a single, logical switch.

Figure 3: Medium-Sized Virtual Chassis Fabric with 20 Members
Medium-Sized Virtual Chassis Fabric with
20 Members

The medium-sized, 20-member Virtual Chassis Fabric supports 768 10-Gigabit Ethernet ports with hosts and 38,400 VMs.

Scale Out Architecture

When a single Virtual Chassis Fabric is not large enough, you can simply move to a scale out architecture. Each Virtual Chassis Fabric becomes a point of delivery (POD). In a scale out architecture, there are many PODs that are connected through a fabric layer as shown in Figure 4.

Figure 4: Scale Out Architecture with Virtual Chassis Fabric PODs
Scale Out Architecture with Virtual Chassis
Fabric PODs

The fabric switches that are connecting the PODs of Virtual Chassis Fabric are Juniper Networks QFX5100-24Q switches, which support 32 40-Gigabit Ethernet interfaces. A scale-out architecture allows MetaFabric Architecture 2.0 to scale beyond a single Virtual Chassis Fabric and support up to 24,576 10-Gigabit Ethernet ports with 12,288 hosts and over 1,000,000 VMs. Other benefits of a scale-out architecture are high availability and resiliency. Each POD is managed separately and is treated as a separate fault domain. A failure in one POD does not impact the performance and availability of the other 31 PODs in MetaFabric Architecture 2.0.

Virtual Chassis Fabric Platforms and Topology

A Virtual Chassis Fabric is constructed using a set of switches. Virtual Chassis Fabric offers the best performance and features with the QFX5100 series which comes in two models, the QFX5100-24Q switch and the QFX5100-48S switch.

As shown in Figure 5, the QFX5100-24Q switch offers 24 built-in QSFP+ ports and two modules that can support 4 ports of QSFP+; this brings the total number of 40-Gigabit Ethernet ports to 32. One nice feature of the QSFP+ ports is that you can break a single port into four 10-Gigabit Ethernet ports.

Figure 5: Juniper Networks QFX5100-24Q Switch
Juniper Networks QFX5100-24Q Switch

The QFX5100-48S switch offers 48 10-Gigabit Ethernet ports and six 40-Gigabit Ethernet ports as shown in Figure 6.

Figure 6: Juniper Networks QFX5100-48S Switch
Juniper Networks QFX5100-48S Switch

Virtual Chassis Fabric is most often deployed using the QFX5100-24Q and QFX5100-48S switches; these switches complement each other when building a simple 3-stage topology.

The Virtual Chassis Fabric topology is a 3-stage Clos architecture which offers the best latency, performance, and scale. The QFX5100-24Q switch is most commonly used as a spine switch, and the QFX5100-48S is most commonly used as a leaf switch as shown in Figure 7.

Figure 7: Virtual Chassis Fabric Architecture
Virtual Chassis Fabric Architecture

One benefit of a Virtual Chassis Fabric is that there are no physical restrictions on where you can connect devices; you can use both the spine and leaf switches to connect servers, routers, or any other device. When creating a small to medium-sized data center, port flexibility creates a distinct advantage because a single fabric can connect all servers, storage, firewalls, and even the Internet and WAN.

Virtual Chassis Fabric Performance and Scale

Virtual Chassis Fabric is a high-speed Ethernet fabric for every device in the data center. The very nature of the spine and leaf topology enables deterministic traffic patterns and latency, which means that applications are lightning fast. Some of the performance characteristics of Virtual Chassis Fabric are as follows:

  • End-to-end latency of 2.5 microseconds

  • Line-rate performance of 10-Gigabit Ethernet and 40-Gigabit Ethernet

  • 1.28Tbps of forwarding capacity per switch

  • 25.6Tbps of forwarding capacity for the entire fabric

As you can see, there is enough scale and performance in a Virtual Chassis Fabric to support any servers and applications that you are using. Virtual Chassis Fabric uses a new technology called the Unified Forwarding Table to give you the power to adjust the logical scale of the Ethernet fabric. Virtual Chassis Fabric uses next-generation flexible tables as shown in Figure 8.

Figure 8: Unified Forwarding Table
Unified Forwarding Table

There are five fabric profiles that you can choose from—each profile incrementally increases the amount of Layer 3 scale, as shown in Table 2.

Table 2: Unified Forwarding Table - Fabric Profiles

Profile

MAC Addresses

Layer 3 Hosts

Longest Prefix Match

l2-profile-one

288,000

16,000

16,000

l2-profile-two

224,000

56,000

16,000

l2-profile-three

160,000

88,000

16,000

l3-profile

96,000

120,000

16,000

lpm-profile

32,000

16,000

128,000

High Availability

Virtual Chassis Fabric leverages the functionality from carrier-class routers such as the Juniper Networks MX Series and T Series to provide high availability in the Ethernet fabric.

  • Graceful Routing Engine switchover (GRES)—Keeps the operating system state synchronized between the master and backup Routing Engines

  • Nonstop active routing (NSR)—Keeps the routing protocol state synchronized between the master and backup Routing Engines

  • Nonstop bridging (NSB)—Keeps the Layer 2 protocol state synchronized between the master and backup Routing Engines

The spine switches act as the master and backup Routing Engines in a Virtual Chassis Fabric as shown in Figure 9.

Figure 9: Virtual Chassis Fabric Roles
Virtual Chassis Fabric Roles

The other leaf switches in a Virtual Chassis Fabric act as simple line cards. If the master Routing Engine experiences a failure, the backup Routing Engine immediately becomes the new master Routing Engine. Traffic will not be interrupted because GRES, NSR, and NSB keep the two Routing Engines continuously synchronized.

Virtual Chassis Fabric is an easy to manage, high-performance, and highly-available Ethernet fabric that allows you to build a best-of-class data center network.