Example: Enabling SR-IOV on vMX Instances on KVM
vMX on KVM supports single-root I/O virtualization (SR-IOV) interface types. Single root I/O virtualization (SR-IOV) allows a physical function to appear as multiple, separate vNICs. SR-IOV allows a device, such as a network adapter to have separate access to its resources among various hardware functions. If you have a physical NIC that supports SR-IOV, you can attach SR-IOV-enabled vNICs or virtual functions (VFs) to the vMX instance to improve performance.
System requirements:
Junos OS Release 18.4 or later.
SR-IOV on the VMX for KVM requires one of the following Intel NIC drivers:
Intel X520 or X540 using 10G ports and ixgbe driver
Intel X710 or XL710 using 10G ports and i40e driver
Starting in Junos OS Release 19.1R1-S1 and in Junos OS Release 19.2R1, support for 40G ports with Intel XL710-QDA2 NICs are available for VMX instances. When using 40G ports, the vMX autodetects the port speed and assigns two I/O vCPUs.
To enable SR-IOV on VMX instances, you must complete the following tasks:
Prepare a NIC to use SR-IOV in system (/sys/) mode.
Install driver from Intel, you must compile the driver, uninstall old driver, and install new compiled driver
Note:The vMX installer provides a modified intel-driver as well. You can either use the native drivers from Intel, or use vMX modified driver.
Prepare vmx.conf file
Use Junos CLI to configure native driver
BIOS requirement to enable SR-IOV- Ensure that Intel VT-d or AMD IOMMU are enabled in the system’s BIOS settings.
Procedure for Identifying PCI-Addresses and Kernel Name for the NIC
Download and Install the Latest Driver Software from Intel
You can download the latest driver software from Intel and replace existing driver software provided by Ubuntu.
In this example, download the software from Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections Under Linux and save it into any directory of your choice and follow the README instructions to proceed next.
To install driver software from Intel:
Prepare NIC to Use SR-IOV in System Mode
The host needs to be informed for each dedicated NIC by setting the sriov_numvfs value, how many VFs are going to use SR-IOV for the given NIC. The vmx.sh script have no information of how many VFs will use the shared NIC. Because of this, you must configure the sriov_numvfs accordingly.
This value can be set as a boot-option to be persistent after a reboot and can be changed on-the-fly which would not be persistent after a reboot.
The procedure given in this example is temporary solution for configuring sriov_numvfs using /sys Any setting to /sys/class/net/interface-name/device/sriov_numvfs is non-permanent, hence the configuration does not survive a reboot.
To prepare NIC to use SR-IOV, complete the following steps:
# echo num_of_vf > /sys/class/net/interface-name/device/sriov_numvfs
Below command allows 4 VNFs to use shared NIC ens8f1 for SR-IOV. You must either use sudo or need login as root user.
As sudo user:
root@ubuntu2:~# echo 4 | sudo tee -a /sys/class/net/ens8f1/device/sriov_numvfs
As root user
root@ubuntu2:~# echo 4 > /sys/class/net/ens8f1/device/sriov_numvfs
The sriov_numvfs option only
accepts values 0-n, where n is the maximum number of VFs that are
supported by the SR-IOV.
Setting SR-IOV at Boot-Time
The following procedures provide some alternate methods for configuring SR-IOV where the configuration persists a reboot of the host.
Following options are available to set the value during the boot-process of the host:
Using rc.local
Setting modprobe options
Setting kernel-paramater using grub
Below example shows a method to configure the sriov_numvfs value by using grub kernel command
For more information on hugepages, see Preparing the Ubuntu Host to Install vMX.
Edit the file “/etc/default/grub”:
lab@ubuntu2:~$ cat /etc/default/grub | grep -i cmdGRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"GRUB_CMDLINE_LINUX="isolcpus=34-41,48-55 default_hugepagesz=1Ghugepagesz=1G hugepages=120 ixgbe.max_vfs=8"
After editing, update the following:
sudo update-grub
Write new boot-loader to make changes active upon next reboot.
sudo grub-install /dev/sda
Reboot the host to make settings active.
sudo reboot
Verify sriov_numvfs Settings
Purpose
To verify the sriov_numvfs configuration using the CLI. In this example, the required NIC to use with SR-IOV is ens8f1 at PCI-address 85:00.0. Please note the “Virtual Function” in the output.
Action
lab@ubuntu2:~$ lspci | grep 85
85:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 85:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 85:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 85:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
The /sys/class/net/ens8f1/device/sriov_numvfs also contains the desired value of sriov_numvfs.
lab@ubuntu2:~$ cat /sys/class/net/ens8f1/device/sriov_numvfs
8
For testing, the sriov_numvfs can be changed quickly by directly
writing into /sys/class/net/interface-name/device/sriov_numvfs.
Changing the Number of sriov_numvfs
We recommend creating sriov_numvfs in advance (example: by using grub command at boot-time), because, changing the VF’s number is not allowed after deploying 1st vMX instance which uses given NIC with SR-IOV. If you must change the VF’s number, then you must stop the running vMX. We recommend to set the sriov_numvfs option to a higher value to avoid changing the sriov_numvfs number afterwards.
To change an already configured value of VFs, you must first change it's value as 0, and then change it to required integer value.
If there are no VFs assigned, the number of VFs can be changed to any valid value (0 - n, where n is the maximum number of VFs that are supported by the SR-IOV)
You must perform the following steps to modify the number of VFs:
Stop running VNFs using the shared SR-IOV NIC.
Disable the SR-IOV network adapter by setting the number of Virtual Functors (VFs) to 0.
As root:
root@ubuntu2:~# echo 0 > /sys/class/net/ens8f1/device/sriov_numvfsAs sudo:
lab@ubuntu2:~$ echo 0 | sudo tee -a /sys/class/net/ens8f1/device/sriov_numvfsChange the required number of VF (you are using six in this example)
root@ubuntu2:~# echo 6 > /sys/class/net/ens8f1/device/sriov_numvfsNote:If you see the following error message, then first set the value to zero as described in step 2 before performing step 3
root@ubuntu2:~# echo 6 > /sys/class/net/ens8f1/device/sriov_numvfs bash: echo: write error: Device or resource busy
Verify your configuration by using the following command:
root@ubuntu2:~# cat /sys/class/net/ens8f1/device/sriov_numvfs
Before restarting the vMX, adopt the vmx.conf file for SR-IOV usage.
Updating the VMX Configuration File (vmx.conf) Parameters
The parameters required to configure vMX are defined in the startup configuration file. The configuration file is in YAML format. The default file is config/vmx.conf. You can save your configuration file to a different name for different instances.
To configure interfaces for SR-IOV device types, you must specify the interface, the NIC, and the MAC address. Table 1 provides the details of the configuration parameters that we are using to change vmx.conf file.
Components |
VM Parameters |
Description |
|---|---|---|
vPFE |
|
Use |
|
Set to |
|
Interfaces |
|
If type is set to |
|
Set it to |
|
|
The kernel-name for the interface to use. |
|
|
Set to |
|
|
Ensure that each VF instance using the shared NIC is using a unique or different MAC address. |
A sample vmx.conf file:
lab@ubuntu2:~/vmx/config$ cat vmx.conf.sriov
##############################################################
#
# vmx.conf
# Config file for vmx on the hypervisor.
# Uses YAML syntax.
# Leave a space after ":" to specify the parameter value.
#
##############################################################
---
#Configuration on the host side - management interface, VM images etc.
HOST:
identifier : vmx1 # Maximum 6 characters
host-management-interface : ens4f0
routing-engine-image : "/home/lab/vmx/images/junos-vmx-x86-64-18.1R3-S2.5.qcow2"
routing-engine-hdd : "/home/lab/vmx/images/vmxhdd.img"
forwarding-engine-image : "/home/lab/vmx/images/vFPC-20181023.img"
---
#External bridge configuration
BRIDGES:
- type : external
name : br-ext # Max 10 characters
---
#vRE VM parameters
CONTROL_PLANE:
vcpus : 1
memory-mb : 1024
console_port: 8601
interfaces :
- type : static
ipaddr : 10.102.144.94
macaddr : "0A:00:DD:C0:DE:0E"
---
#vPFE VM parameters
FORWARDING_PLANE:
memory-mb : 8192
vcpus : 4
console_port: 8602
device-type : mixed <<<< sriov or mixed
use_native_drivers : true <<<< use drivers as provided by the host
interfaces :
- type : static
ipaddr : 10.102.144.98
macaddr : "0A:00:DD:C0:DE:10"
---
#Interfaces
JUNOS_DEVICES:
- interface : ge-0/0/0
type : virtio <<< required
mac-address : "02:06:0A:0E:FF:F0"
description : "ge-0/0/0 interface"
- interface : ge-0/0/1
type : sriov <<< required
mtu : 9192 <<< required
port-speed-mbps : 10000 <<< required
nic : ens8f1 <<< required
virtual-function : 0 <<< use consecutive / unique numbers for each vMX instance
mac-address : "02:06:0A:0E:FF:F1" <<< make sure that each vNF is using a DIFFERENT MAC-address
description : "ge-0/0/1 interface"Start vmx-install
lab@ubuntu2:~/vmx$ sudo ./vmx.sh --install --cfg ./config/vmx.conf.sriov
Changes Required for Using Intel ixgbe Driver
When you try to move an existing deployment from modified IXGBE driver to unmodified IXGBE driver, enter edit mode in Junos CLI and use the following command when using "native" drivers.
user@host# set interfaces vlan-offload
For more information, see Modified and Unmodified IXGBE Driver.