Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 

Example: Enabling SR-IOV on vMX Instances on KVM

 

vMX on KVM supports single-root I/O virtualization (SR-IOV) interface types. Single root I/O virtualization (SR-IOV) allows a physical function to appear as multiple, separate vNICs. SR-IOV allows a device, such as a network adapter to have separate access to its resources among various hardware functions. If you have a physical NIC that supports SR-IOV, you can attach SR-IOV-enabled vNICs or virtual functions (VFs) to the vMX instance to improve performance.

System requirements:

  • Junos OS Release 18.4 or later.

  • SR-IOV on the VMX for KVM requires one of the following Intel NIC drivers:

    • Intel X520 or X540 using 10G ports and ixgbe driver

    • Intel X710 or XL710 using 10G ports and i40e driver

    Starting in Junos OS Release 19.1R1-S1 and in Junos OS Release 19.2R1, support for 40G ports with Intel XL710-QDA2 NICs are available for VMX instances. When using 40G ports, the vMX autodetects the port speed and assigns two I/O vCPUs.

To enable SR-IOV on VMX instances, you must complete the following tasks:

  • Prepare a NIC to use SR-IOV in system (/sys/) mode.

  • Install driver from Intel, you must compile the driver, uninstall old driver, and install new compiled driver

    Note

    The vMX installer provides a modified intel-driver as well. You can either use the native drivers from Intel, or use vMX modified driver.

  • Prepare vmx.conf file

  • Use Junos CLI to configure native driver

  • BIOS requirement to enable SR-IOV- Ensure that Intel VT-d or AMD IOMMU are enabled in the system’s BIOS settings.

Procedure for Identifying PCI-Addresses and Kernel Name for the NIC

  1. To find the PCI address, use the following command:
    lab@ubuntu2:/etc/modprobe.d$ ethtool -i ens8f1 | grep bus
  2. To find the kernel name using PCI, use the following command:
    lab@ubuntu2:~$ cd /sys/bus/pci/devices
    lab@ubuntu2:/sys/bus/pci/devices$ ls 0000\:85\:00.1/net/
  3. To find out the driver in use for the NIC, use the following command:
    lab@ubuntu2:~$ ethtool -i ens8f1 | grep ^driver

Download and Install the Latest Driver Software from Intel

You can download the latest driver software from Intel and replace existing driver software provided by Ubuntu.

In this example, download the software from Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections Under Linux and save it into any directory of your choice and follow the README instructions to proceed next.

To install driver software from Intel:

  1. Install the driver software.
  2. Uninstall the old driver and load the updated driver by using the rmmod/modprobe command.
    Warning

    The command rmmod uninstalls the 10GE driver. If this is the only interface you are connected to, then access to the host will be lost.

  3. Verify if the new driver is installed correctly.

Prepare NIC to Use SR-IOV in System Mode

The host needs to be informed for each dedicated NIC by setting the sriov_numvfs value, how many VFs are going to use SR-IOV for the given NIC. The vmx.sh script have no information of how many VFs will use the shared NIC. Because of this, you must configure the sriov_numvfs accordingly.

This value can be set as a boot-option to be persistent after a reboot and can be changed on-the-fly which would not be persistent after a reboot.

The procedure given in this example is temporary solution for configuring sriov_numvfs using /sys Any setting to /sys/class/net/interface-name/device/sriov_numvfs is non-permanent, hence the configuration does not survive a reboot.

To prepare NIC to use SR-IOV, complete the following steps:

  1. Create a virtual function (VF) using the following command:

    Below command allows 4 VNFs to use shared NIC ens8f1 for SR-IOV. You must either use sudo or need login as root user.

    As sudo user:

    As root user

Note

The sriov_numvfs option only accepts values 0-n, where n is the maximum number of VFs that are supported by the SR-IOV.

Setting SR-IOV at Boot-Time

The following procedures provide some alternate methods for configuring SR-IOV where the configuration persists a reboot of the host.

Following options are available to set the value during the boot-process of the host:

  • Using rc.local

  • Setting modprobe options

  • Setting kernel-paramater using grub

Below example shows a method to configure the sriov_numvfs value by using grub kernel command

  • You must set "intel_iommu=on" and ixgbe.max_vfs= value

    For more information on hugepages, see Preparing the Ubuntu Host to Install vMX.

    Edit the file “/etc/default/grub”:

    After editing, update the following:

    Write new boot-loader to make changes active upon next reboot.

    Reboot the host to make settings active.

Verify sriov_numvfs Settings

Purpose

To verify the sriov_numvfs configuration using the CLI. In this example, the required NIC to use with SR-IOV is ens8f1 at PCI-address 85:00.0. Please note the “Virtual Function” in the output.

Action

lab@ubuntu2:~$ lspci | grep 85

The /sys/class/net/ens8f1/device/sriov_numvfs also contains the desired value of sriov_numvfs.

lab@ubuntu2:~$ cat /sys/class/net/ens8f1/device/sriov_numvfs

For testing, the sriov_numvfs can be changed quickly by directly writing into /sys/class/net/interface-name/device/sriov_numvfs.

Changing the Number of sriov_numvfs

We recommend creating sriov_numvfs in advance (example: by using grub command at boot-time), because, changing the VF’s number is not allowed after deploying 1st vMX instance which uses given NIC with SR-IOV. If you must change the VF’s number, then you must stop the running vMX. We recommend to set the sriov_numvfs option to a higher value to avoid changing the sriov_numvfs number afterwards.

To change an already configured value of VFs, you must first change it's value as 0, and then change it to required integer value.

If there are no VFs assigned, the number of VFs can be changed to any valid value (0 - n, where n is the maximum number of VFs that are supported by the SR-IOV)

You must perform the following steps to modify the number of VFs:

  1. Stop running VNFs using the shared SR-IOV NIC.

  2. Disable the SR-IOV network adapter by setting the number of Virtual Functors (VFs) to 0.

  3. Change the required number of VF (you are using six in this example)

    Note

    If you see the following error message, then first set the value to zero as described in step 2 before performing step 3

  4. Verify your configuration by using the following command:

Before restarting the vMX, adopt the vmx.conf file for SR-IOV usage.

Updating the VMX Configuration File (vmx.conf) Parameters

The parameters required to configure vMX are defined in the startup configuration file. The configuration file is in YAML format. The default file is config/vmx.conf. You can save your configuration file to a different name for different instances.

To configure interfaces for SR-IOV device types, you must specify the interface, the NIC, and the MAC address. Table 1 provides the details of the configuration parameters that we are using to change vmx.conf file.

Table 1: VMX Configuration File Parameters

Components

VM Parameters

Description

vPFE

device-type

Use sriov for all interfaces using the SR-IOV or use mixed to allow mixing of SR-IOV and non-SR-IOV-based interfaces.

use_native_drivers

Set to true to allow using the host’s Intel ixgbe driver (which was downloaded and complied in above steps)

Interfaces

type

If type is set to sriov, then port-speed-mbps and nic must be set.

port-speed-mbps

Set it to 10000 for 10GE NIC.

nic

The kernel-name for the interface to use.

virtual-function

Set to 0 for first vMX instance using this NIC. Ensure to set to 1 for 2nd vMX using this shared NIC (and so on)

mac-address

Ensure that each VF instance using the shared NIC is using a unique or different MAC address.

A sample vmx.conf file:

lab@ubuntu2:~/vmx/config$ cat vmx.conf.sriov

Start vmx-install

Changes Required for Using Intel ixgbe Driver

When you try to move an existing deployment from modified IXGBE driver to unmodified IXGBE driver, enter edit mode in Junos CLI and use the following command when using "native" drivers.

For more information, see Modified and Unmodified IXGBE Driver.