System Requirements for Tanzu Deployment

RedHat Enterprise Linux

Version 8.4, 8.5, 8.6

Rocky Linux: 4.18.X

The tested kernel version for Rocky Linux is 4.18.0-372.19.1.rt7.176.el8_6.x86_64 and 4.18.0-372.32.1.rt7.189.el8_6.x86_64

• Intel E810 CVL with Firmware 4.22 0x8001a1cf 1.3346.0

• Intel E810 CPK with Firmware 2.20 0x80015dc1 1.3083.0

• Intel E810-CQDA2 with Firmware 4.20 0x80017785 1.3346.0

• Intel XL710 with Firmware 9.20 0x8000e0e9 0.0.0

• Mellanox ConnectX-6

• Mellanox ConnectX-7

When using Mellanox NICs, ensure your interface names do not exceed 11 characters in length.

JCNR supports an all-in-one or multinode Kubernetes cluster, with control plane and worker nodes running on virtual machines (VMs) or bare metal servers (BMS).

cat /etc/modules-load.d/vfio.conf
vfio
vfio-pci

Update grub and reboot the host. For example:

grub2-mkconfig -o /boot/grub2/grub.cfg

Verify the hugepage is set by executing the following commands:

cat /proc/cmdline

grep -i hugepages /proc/meminfo

Enable the host with SR-IOV and VT-d in the system's BIOS.

Depends on BIOS.

Enable VLAN driver at system boot.

Configure /etc/modules-load.d/vlan.conf as follows:

cat /etc/modules-load.d/vlan.conf
8021q

Reboot and verify by executing the command:

lsmod | grep 8021q

Enable VFIO-PCI driver at system boot.

Configure /etc/modules-load.d/vfio.conf as follows:

cat /etc/modules-load.d/vfio.conf
vfio
vfio-pci

Reboot and verify by executing the command:

lsmod | grep vfio

Set IOMMU and IOMMU-PT in GRUB.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt"

Update grub and reboot.

grub2-mkconfig -o /boot/grub2/grub.cfg 

reboot

Additional kernel modules need to be loaded on the host before deploying JCNR in L3 mode. These modules are usually available in linux-modules-extra or kernel-modules-extra packages.

Create a /etc/modules-load.d/crpd.conf file and add the following kernel modules to it:

tun
fou
fou6
ipip
ip_tunnel
ip6_tunnel
mpls_gso
mpls_router
mpls_iptunnel
vrf
vxlan

Enable kernel-based forwarding on the Linux host.

ip fou add port 6635 ipproto 137

Exclude JCNR interfaces from NetworkManager control.

NetworkManager is a tool in some operating systems to make the management of network interfaces easier. NetworkManager may make the operation and configuration of the default interfaces easier. However, it can interfere with Kubernetes management and create problems.

To avoid NetworkManager from interfering with JCNR interface configuration, exclude JCNR interfaces from NetworkManager control. Here's an example on how to do this in some Linux distributions:

1. Create the /etc/NetworkManager/conf.d/crpd.conf file and list the interfaces that you don't want NetworkManager to manage.

   For example:

   [keyfile]
    unmanaged-devices+=interface-name:enp*;interface-name:ens*

   where enp* and ens* refer to your JCNR interfaces.
   Note: enp* indicates all interfaces starting with enp. For specific interface names, provided a comma-separated list.
2. Restart the NetworkManager service:

   sudo systemctl restart NetworkManager

   .
3. Edit the /etc/sysctl.conf file on the host and paste the following content in it:

   net.ipv6.conf.default.addr_gen_mode=0
   net.ipv6.conf.all.addr_gen_mode=0
   net.ipv6.conf.default.autoconf=0
   net.ipv6.conf.all.autoconf=0

4. Run the command sysctl -p /etc/sysctl.conf to load the new sysctl.conf values on the host.

5. Create the bond interface manually. For example:

   ifconfig ens2f0 down
   ifconfig ens2f1 down
   ip link add bond0 type bond mode 802.3ad
   ip link set ens2f0 master bond0
   ip link set ens2f1 master bond0
   ifconfig ens2f0 up ; ifconfig ens2f1 up; ifconfig bond0 up

Verify the core_pattern value is set on the host before deploying JCNR.

sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

You can update the core_pattern in /etc/sysctl.conf. For example:

kernel.core_pattern=/var/crash/core_%e_%p_%i_%s_%h_%t.gz

Enable iommu unsafe interrupts and unsafe noiommu mode.

echo Y > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

echo Y > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Configure iptables to accept specified traffic.

 iptables -I INPUT -p tcp --dport 830 -j ACCEPT
 iptables -I INPUT -p tcp --dport 24 -j ACCEPT
 iptables -I INPUT -p tcp --dport 8085 -j ACCEPT
 iptables -I INPUT -p tcp --dport 8070 -j ACCEPT

 iptables -I INPUT -p tcp --dport 8072 -j ACCEPT
 iptables -I INPUT -p tcp --dport 50053 -j ACCEPT

iptables -A INPUT -p icmp -j ACCEPT
iptables -A OUTPUT -p icmp -j ACCEPT

iptables -A INPUT   -s 224.0.0.0/4 -j ACCEPT
iptables -A FORWARD -s 224.0.0.0/4 -d 224.0.0.0/4 -j ACCEPT
iptables -A OUTPUT  -d 224.0.0.0/4 -j ACCEPT

On the ESXi Hypervisor, enable 16 queues.

set esxcli system module parameters set -m icen -p NumQPsPerVF=16,16,16,16

On the ESXi Hypervisor, enable trust and disable spoofcheck:

esxcli intnet sriovnic vf set -s false -t true  -v 0  -n vmnic2

Check the settings:

esxcli intnet sriovnic vf get -n vmnic2

VF ID           Trusted         Spoof Check
0               true           false