Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Example: How to Set Up SR-IOV 10GbE High Availability on vSRX 3.0 with Ubuntu on a KVM Server

 

This example shows how to set up SR-IOV 10GbE high availability deployment on vSRX 3.0 instances.

Requirements

This example uses the following hardware, software components, and operating systems:

Device

  • vSRX 3.0

Software

  • Junos OS Release 20.4R1

Hardware

  • NIC: Intel Corporation Ethernet Controller X710/X520/82599

  • Driver: i40e version: 2.1.14-k or ixgbe version: 5.1.0-k

  • CPU: Intel (R) Xeon (R) Gold 5120 CPU @ 2.20 GHz

  • 56 CPUs

  • 0- 55 online CPUs list

  • 2 threads per core

  • 14 cores per socket

  • 2 sockets

  • 2 non-uniform memory access (NUMA) nodes

For more information on NICs, Hypervisors, and ports supported with SR-IOV see Hardware Specifications.

Operating System

Table 1: SR-IOV HA Supported KVM OS and Network Adapter Information

KVM OS and Network Adapters

Support

Intel 82599/X520/X540 (82599 ixgb driver based)

Yes

Intel X710/XL710/XXV710/X722 (i40e driver based)

Yes

Mellanox ConnectX-4/ConnectX-4 Lx

No

Ubuntu 18.04 (kernel:4.15.0 + libvirt:4.0.0) and 20.04 (kernel:5.4.0 + libvirt:6.0.0) LTS

Yes

Redhat 8.2 (kernel:4.18.0 + libvirt:4.5.0)

Yes

Operating Systems used in this example are:

  • Ubuntu 18.04.3 LTS on a KVM server

  • Kernel: 4.15.0-64-generic

  • Kernel: 4.18.0-193.1.2.el8_2.x86_64

  • redhat rhel 8.2

Overview

This example shows how to:

  • Set up the 10-Gigabit high availability deployment

  • Build VFs bus information on NIC interfaces and change the XML template

  • Configure basic vSRX 3.0 instances

In a high availability environment, the control link and fabric data links are key communication channels for chassis cluster stability. Both links are part of the same Linux bridge. The host operating system (Ubuntu) shares the CPU allotted for the vSRX 3.0 control plane for routine tasks and with one of the vSRX 3.0 PFE data plane threads for packet processing. This contention for resources coupled with the lack of a dedicated VLAN or NIC for the control link could contribute to heartbeat misses.

Furthermore, interrupt handling on the host can also impact the performance. When packets arrive at the NIC, a hardware interrupt indication and the CPU core that services the vSRX 3.0 control plane must stop and service the interrupt. A large number of packets from the NIC can lead to more hardware interruptions and less CPU resources to service the vSRX 3.0 control plane.

To overcome the design constraints and the CPU resource contention, we recommend the following changes:

  • Allot dedicated CPU to each vSRX 3.0 control plane, vSRX 3.0 data plane, and the host operating system.

  • Allot required memory on the host.

  • Leverage SR-IOV for fabric interface in a high availability deployment.

  • Remove GRE for control link communication and use multicast in high availability deployments.

  • Enable IRQ affinity to avoid the interrupts handled by the CPUs for vSRX 3.0 control plane and data plane.

  • Enlarge the physical NIC descriptor from 512 to 4096 bytes.

We recommend you configure all revenue ports of vSRX 3.0 as SR-IOV. Also, on KVM you can configure SR-IOV high availability on management port -fxp0/ control port- em0 / fabric port-ge-0/0/*.

Note

SR-IOV high availability Layer 2 function is not supported. Also, VMware and Mellanox NIC do not support SR-IOV high availability functionality.

Figure 1 shows the topology used in this example.

Figure 1: High Availability Trust and Untrust Dual NIC Topology
High
Availability Trust and Untrust Dual NIC Topology

Configuration

SR-IOV High Availability Deployment

To configure SRX-IOV high availability deployment, perform the following procedures in Ubuntu:

Step-by-Step Procedure

To configure the SR-IOV high availability deployment:

  1. Enable the SR-IOV port.

    Enter the required inputs for availing ports.

  2. Make the following changes in the grub file:
  3. Execute upgrade grub.
  4. Reboot the host for changes to take effect.
  5. (Optional) Cores 0-3 switch to interrupt context - Interrupt Service Routine (ISR) to handle the coming interrupt. Cores 4-13 on NUMA 0 are used for vSRXs. Run the following script:
  6. Increase tx and rx buffer size to 4096 on all NICs.
  7. Turn off flow control.
  8. Check for the server persistent after reboot.
  9. Set SR-IOV VF trust mode on and spoof checking off.

    Or, you can also add below command to rc.local script:

Build Bus Information of Virtual Functions on NICs

Step-by-Step Procedure

To build bus information of VFs on NICs:

  1. Now that we know the backup interfaces, we need to identify the bus information of all VFs on each NIC.

    For backup interfaces in the trust network, we need bus information on the first three VFs.

    For backup interfaces in the untrust network, we need bus information on the first two VFs.

  2. Table 2 explains the XML to Junos interface-mapping required to build the template.

    Table 2: XML to Junos Interfaces Mapping

    NIC

    VF

    Bus Information

    Interface

    XML Position

    fxp0

    fxp0

    1

    em0

    em0

    2

    eth0

    0

    0000:18:02.0

    ge-0/0/0 fab0

    ge-7/0/0 fab1

    3

    1

    0000:18:02.1

    ge-0/0/1

    ge-7/0/1

    4

    2

    0000:18:02.2

    ge-0/0/5

    ge-7/0/5

    8

    eth1

    0

    0000:18:06.0

    ge-0/0/3

    ge-7/0/3

    6

    eth2

    0

    0000:18:0a.0

    ge-0/0/2

    ge-7/0/2

    5

    eth3

    0

    0000:18:0e.0

    ge-0/0/4

    ge-7/0/4

    7

    The XML to Junos configuration is sequential. The first interface is assigned to fxp0 , second interface is assigned to em0 and the last interface is assigned to ge-0/0/9 as shown in Table 3.

  3. Develop the following Table 3 based on Table 2 in step 3.

    Table 3: Junos Interfaces and Bus Information

    XML Position

    BUS Information

    Junos interfaces

    1

    BR0

    fxp0

    2

    BR1

    em0

    3

    0000:18:02.0

    ge-0/0/0

    4

    0000:18:02.1

    ge-0/0/1

    5

    0000:18:0a.0

    ge-0/0/2

    6

    0000:18:06.0

    ge-0/0/3

    7

    0000:18:0e.0

    ge-0/0/4

    8

    0000:18:02.2

    ge-0/0/5

  4. Modify the interface stanza 2,3,4,8 and 12 in XML template below as per the table in step 4.

Configure vSRX 3.0

CLI Quick Configuration

To quickly configure this example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Note

ge-0/0/3, ge-0/0/4, ge-7/0/3, ge-7/0/4 are not used in this configuration.

Verification

Confirm that the configuration is working properly.

Verifying Chassis Cluster Status

Purpose

Verify the chassis cluster status, statistics, and redundancy group information.

Action

From operational mode, enter the following commands.

{primary:node0}
user@host> show chassis cluster interfaces
{primary:node0}
user@host> show chassis cluster statistics
{primary:node0}
user@host> show chassis cluster control-plane statistics
{primary:node0}
user@host> show chassis cluster data-plane statistics
{primary:node0}
user@host> show chassis cluster status redundancy-group 1

Verification of deployment results

Meaning

The sample output shows that there are no manual failover in chassis cluster status and provides you the spoof checking status and SR-IOV VF trust mode state.

Results

From configurational mode, confirm your configuration by entering the show security zones, and show chassis commands. If the output does not display the intended configuration, repeat the instructions in this example to correct the configuration.