Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

How to Deploy the QFX5100 as a VTEP Gateway with VMware NSX

 

Requirements

NSX Requirements

System requirements for installing NSX-V are available in the VMware NSX 6.4 Installation guide, and we have used the guide as a reference to set up the configuration example.

This example uses the following:

  • Management and Edge Cluster: Three NSX controllers are deployed on ESXi servers using the installation procedures as described above.

  • Compute Clusters: Two iXsystems iX22X2-STTHDTRF servers (ESXi host OS) per compute cluster orchestrate virtual machines used to demonstrate application functionality.

  • Bare-Metal Servers: One iXsystems iX22X2-STTHDTRF server (Ubuntu host OS) orchestrates a baremetal server that runs no virtual machines.

Juniper Hardware – Qualified Models

Table 1: Qualified Models

Hardware Model

Switch Software Version

NSX Version

Solution Topology

QFX5100

18.1R3-S4

INSX 6.4.4

Standalone

QFX5110

18.1R3-S4

NSX 6.4.4

Standalone

18.4R2

NSX 6.4.5/ NSX6.4.6

Standalone

QFX5110 (Virtual Chassis)

19.4R1

NSX 6.4.5/ NSX6.4.6

High Availability

QFX5120-48Y

18.4R2

NSX 6.4.4/ NSX 6.4.5/ NSX6.4.6

Standalone

QFX5120-48Y (Virtual Chassis)

19.3R1

NSX 6.4.5/ NSX6.4.6

High Availability

Physical Topology

The following figure shows the configuration example. It uses VMware NSX 6.4.5 as the SDN controller, with the QFX5100 line of switches. This example also includes QFX5210 switches acting as spine devices.

The topology includes a 3-stage IP CLOS fabric consisting of:

  • Two leaf devices, using either:

    • QFX5110 running Junos OS release 18.4R2 or later, or

    • QFX5120 running Junos OS release 18.4R2 or later

  • Two QFX5210 switches as spine devices, running Junos OS release 18.1R3-S4 or later

Note

The QFX5100 switch has not been officially validated with NSX 6.4.5.

Figure 1: Physical Topology
Physical Topology

Configuration

This section describes NSX-specific configuration as well as relevant configuration details for the Junos devices.

NSX Configuration

Preparing the NSX environment starts with NSX-V installation, following the steps outlined in the VMware NSX 6.4 Installation and Upgrade guides.

NSX configuration components applicable to an intra-VNI use case are:

  • NSX management and edge cluster

  • Compute clusters

  • VXLAN configuration

  • Segment ID and transport zone

  • Logical switches

  • Service definitions

NSX Management and Edge Cluster

The NSX management and edge cluster hosts the NSX manager and three NSX controllers (covered later in this section).

NSX Manager

System pre-requisites required prior to NSX-V installation are available here.

NSX manager installation, as instructed here, is seen as a virtual appliance in the vCenter server environment. Once NSX manager is installed, the next step is to register with the vCenter server.

After successful registration, NSX manager creates a Networking & Security plugin in the vCenter GUI, as shown below.

This is followed by the license key addition.

NSX Controllers

As per VMware recommendation, no matter the size of the NSX Data Center for vSphere deployment, create three NSX Controller nodes in each NSX Controller cluster. Having a different number of controller nodes is not supported. Once deployed, they appear as virtual machines. IP addresses assigned to the controller pool are from the same subnet as the NSX manager. It takes a few minutes to deploy each controller.

In a cluster, the controllers are assigned primary and backup status on a role basis. One controller might be in a primary role for the api-service, and a backup role for directory-service. However, the states across all the controllers are synchronized so that they can be ready to take the primary role if the current primary fails.

Compute Clusters

We used the Host preparation instructions from the VMware NSX 6.4 Installation and Upgrade guide to set up compute clusters. The compute cluster are up in the data center, and consist of several ESXi hosts.

VXLAN Configuration

We used the VXLAN configuration instructions from the VMware NSX 6.4 Installation and Upgrade guide to set up VXLAN.

Each compute host (ESXi server) functions as a VXLAN tunnel endpoint (VTEP) and has a vmkernel (vmk) interface that has an IP address assigned to it that is used as the VTEP IP address. An IP pool assigned to each VTEP has a QFX device interface and IP address as the gateway IP address.

Segment ID and Transport Zone

Segment IDs are functionally equivalent to VNIDs used by VXLAN tunnels. For the above topology, a segment ID pool from 5000-5999 has been defined. Instructions from the VMware NSX 6.4 Installation and Upgrade guide for adding the segment IP pool have been followed.

A transport zone defines the reach of the VXLAN tunnel. Only VMs on the physical hosts that belong to the same transport zone can form VXLAN tunnels. A transport zone has been created to include all compute hosts across the two different clusters, defining the span for both logical switches. Under the transport zone configuration, the defined VXLAN control plane mode regulates the handling of BUM traffic. In Unicast mode, as selected below, there is no physical network replication. The compute hosts will first query the controller for the MAC-VTEP mapping and use the controller’s response to populate their local cache. We used instructions from the VMware NSX 6.4 Installation and Upgrade guide,adding the transport zone .

Logical Switches

Logical switches create logical broadcast domains or segments to which participating end hosts can be logically wired. Each logical switch is seen as a distributed port group on creation and is mapped to a unique VNI (segment ID is assigned roundrobin from the segment ID pool). Since logical switches are distributed port-groups, they are available across all the hosts that are part of the distributed virtual switch. We recommended that the transport zone associated with the logical switch also spans across the same hosts. Once communication between the NSX controller and the hardware gateway is established, interfaces connected to the BMS on the hardware VTEP are attached to the appropriate logical switch. Also connected to this logical switch are the VMs that need to belong in the same VNI. When a VM is attached to the logical switch, the same is reflected as the distributed port group assigned to the attached VM network adapter.

Based on the physical testbed, logical switch Prod_Logical_Switch (VNI 5002) is created, as shown below. Virtual machines VM-1(web-03a) and VM-2(web-04a) are attached to Prod_Logical_Switch. Instructions from the VMware NSX 6.4 Installation and Upgrade guide for adding logical-switches have been followed.

Service Definitions

A QFX device has been added as a hardware VTEP to the NSX Controller.

Note that adding the switch as a hardware VTEP in NSX requires certificate generation for the respective hardware devices. Steps for installing an SSL key and certificate on Junos devices for connection with SDN controllers are available at Creating and Installing an SSL Key and Certificate on a Juniper Networks Device for Connection with SDN Controllers. An example of this procedure is shown below.

The above steps result in the addition of Junos device as hardware gateway.

Now add the replication cluster hosts, as shown below.

Once added, the cluster hosts appear on the screen as shown below.

After you configure the hardware devices, you need to bind hardware ports that host VMs/BMSs to the logical switches. On all Junos devices, the interface that is connected to the BMS is used to create a hardware binding. Upon configuration in Junos, the OVSDB interface is discovered by NSX and can be used to create a hardware binding.

Additional information on how attaching a hardware port on NSX is reflected on Junos devices is shown in the next section.

Junos Configuration

Configure the Junos devices in the environment as shown below.

Underlay (EBGP – IP CLOS)

The underlay network consists of two leaf devices (QFX5100s) and two spine devices, configured in a 3-stage CLOS, using EBGP. You can use a variety of QFX Series devices at the spine layer, from the QFX10000 line to the QFX5100 and QFX5200 lines; this example uses a QFX5210-64C device.

Step-by-Step Procedure

  1. Configure the leaf device that is attached to the BMS.
  2. Configure the leaf device that is attached to ESXi/VMs.
  3. Configure the Spine device on the left side.
  4. Configure the Spine device on the right side.

Overlay (OVSDB/VXLAN)

In order for each Junos device to be discovered by the NSX controller (so the hardware gateway to be added), you must set up an OVSDB connection between each Junos device and the NSX controller. Details for this step are available at OVSDBVXLAN Feature Guide for QFX Series Switches (VMware NSX).

Before the hardware port is attached to the logical switch, you must configure VXLAN. On the QFX device, you configure this under the [edit switch-options] hierarchy.

For QFX platforms with interfaces attached to ESXI servers (VMs), to support the additional overhead of VXLAN encapsulation the MTU on physical interfaces acting as VTEPs needs to be set to 1600 or higher. Add the MTU under the [edit interfaces] hierarchy.

You do not need to configure devices with interfaces connected to BMSs; once the hardware device is added, and the hardware port is bound to the relevant logical switch, the NSX controller pushes configuration (using NETCONF) to these devices. For this operation to succeed, there should not be any residual configuration on BMS-connected physical interfaces, such as interface description.

On QFX platforms, when a hardware port is attached to a logical switch, the logical switches are operational (see output of show ovsdb logical-switch). On the QFX device, the addition of a hardware port in NSX is reflected in the respective [edit interfaces] and [edit vlans] configuration hierarchies.

In the configuration snippets below, HW-VTEP is provisioned for logical switch Prod_Logical_Switch (UUID: 415585bd389b-3965-9223-807d77a96791, VNI: 5002).

Validation

The validation steps first focus on the verifying that the physical tested and all configuration components for NSX-V and the Junos devices are functioning as desired. Traffic flows for a use case will then be validated.

Validating NSX-V Controller Status

Use the commands below from the NSX-V CLI as checkpoints to validate status of the NSX-V controllers. You can access the NSX-V CLI through SSH or console (controller VM).

Validating VXLAN Transport Network Connectivity

Communication failures between end hosts on the same VNI (connected to the same logical switch) could be due to transport network missconfiguration. A quick validation approach is to verify if the VTEPs (ESXi hosts) can ping each other. A common cause for ping failures could be MTU misconfiguration (for example, not accounting for VXLAN encapsulation) on the underlay devices.

Validating VTEP status

You can validate VTEP status for all ESXi hosts at the compute hosts (using esxcli commands) and at the NSX-V controllers (using nsxcli commands).

Validating OVSDB status

OVSDB functions as the control plane for VXLAN to facilitate MAC learning. All components (Junos devices, NSX-V controller and compute nodes) use the OVSDB schema to exchange MAC addresses and other statistics to facilitate communication between end users (residing on virtualized compute nodes) and bare-metal servers.

See also

Traffic Flows

This section illustrates the Intra-VNI (VNI 5002 <-> VNI 5002) use case.

  • Intra-VNI (VNI 5002 <-> VNI 5002)

For this use case, verify the following flows:

  • VM to BMS

  • VM to VM

Verifying Intra-VNI (VNI 5002 <-> VNI 5002) Traffic Flows

Figure 2: Intra-VNI Traffic Flows – Logical Topology
Intra-VNI Traffic Flows – Logical Topology

Verifying VM to BMS Intra-VNI Traffic Flow (VM-1: 172.16.40.11 <-> BMS: 172.16.40.13)

Orchestration

VM-1 is hosted on VTEP-1 (ESXi) and BMS is a bare-metal server.

VTEP-1 is connected to QFX5100 and BMS is connected to Juniper L2 hardware gateway (QFX5100 line).

Test case

Demonstrate bi-directional communication between VM-1 and BMS using interfaces located in the same IP subnet.

Traffic flow 1 in the diagram above proceeds as follows:

  • VM-1 sends IP packets towards VTEP-1. The packets do not yet have a VXLAN header.

    • On the return trip, VTEP-1 receives and decapsulates the packets from VTEP-3 (removing the VXLAN header), and sends them towards VM-1.

  • VTEP-1 encapsulates VM-1’s packets using VNI 5002, and sends them over the Junos devices (acting as the IP underlay) towards VTEP-3, the QFX device hardware gateway (HW-VTEP).

    • On the return trip, VTEP-3 encapsulates BMS-1’s packets using VNI 5002, and sends them over the Junos devices towards VTEP-1.

  • VTEP-3 receives and decapsulates the packets (removing the VXLAN header), and sends them towards BMS.

    • On the return trip, BMS sends IP packets towards VTEP-3. The packets to not yet have a VXLAN header.

Preliminary check:

VM-1 (web-03a):

BMS:

Check interface connectivity (ping from 172.16.40.11 to 172.16.40.13):

Verifying VM to VM Intra-VNI Traffic Flow (VM-1: 172.16.40.11 <-> VM-2: 172.16.40.12)

Orchestration:

VM-2 (172.16.40.12) in VNI 5002, hosted on VTEP-2 (ESXi). VM-1 (172.16.40.11), hosted on VTEP-1 (ESXi).

Test case:

Demonstrate bi-directional communication between VM-1 and VM-2 using interfaces located in the same IP subnet.

Traffic flow 2 in the diagram above proceeds as follows:

  • VM-1 sends IP packets towards VTEP-1 (ESXi). VTEP-1 encapsulates these packets using VNI 5002, and sends them over the Junos devices (acting as the IP underlay) towards VTEP-2.

    • On the return trip, VTEP-2 encapsulates the VM-2’s packets using VNI 5002, and sends them over the underlay towards VTEP-1. VTEP-1 receives and decapsulates the packets from VTEP-2 (removing the VXLAN header), and sends them towards VM-1.

  • VTEP-2 receives and decapsulates the packets (removing the VXLAN header), and sends them towards the VM-2.

    • On the return trip, the VM-2 sends IP packets towards VTEP-2 (ESXi).

Preliminary check:

VM-1 (web-03a):

VM-2 (web-04a):

Check interface connectivity (ping from 172.16.40.11 to 172.16.40.12):

Conclusion

This paper demonstrated a data center solution integrating the QFX5100 line of switches as a hardware gateway with VMware NSX-V 6.4.5 to enable communication between physical and virtual workloads.

This setup can also be used as pass-through providing simple underlay connectivity between virtual workloads running on ESXi nodes, acting as software VTEPs.