System Requirements for OpenShift Deployment

• Intel E810 with Firmware 4.00 0x80014411 1.3236.0

• Intel E810-CQDA2 with Firmware 4.000x800144111.3236.0

• Intel XL710 with Firmware 9.00 0x8000cead 1.3179.0

Create a Butane config file, 100-worker-vfiopci.bu, binding the PCI device to the VFIO driver.

variant: openshift
version: 4.8.0
metadata:
  name: 100-worker-vfiopci
  labels:
    machineconfiguration.openshift.io/role: worker 
storage:
  files:
  - path: /etc/modprobe.d/vfio.conf
    mode: 0644
    overwrite: true
    contents:
      inline: |
        options vfio-pci ids=10de:1eb8 
  - path: /etc/modules-load.d/vfio-pci.conf 
    mode: 0644
    overwrite: true
    contents:
      inline: vfio-pci

Create and apply the machine config:

$ butane 100-worker-vfiopci.bu -o 100-worker-vfiopci.yaml

$ oc apply -f 100-worker-vfiopci.yaml

oc create -f hugepages-tuned-boottime.yaml 

# cat hugepages-tuned-boottime.yaml  
apiVersion: tuned.openshift.io/v1 
kind: Tuned 
metadata: 
  name: hugepages  
  namespace: openshift-cluster-node-tuning-operator 
spec: 
  profile:  
  - data: | 
      [main] 
      summary=Boot time configuration for hugepages 
      include=openshift-node 
      [bootloader] 
      cmdline_openshift_node_hugepages=hugepagesz=1G hugepages=8 
    name: openshift-node-hugepages 
  recommend: 
  - machineConfigLabels:  
      machineconfiguration.openshift.io/role: "worker-hp" 
    priority: 30 
    profile: openshift-node-hugepages 

oc create -f hugepages-mcp.yaml  

# cat hugepages-mcp.yaml  
apiVersion: machineconfiguration.openshift.io/v1 
kind: MachineConfigPool 
metadata: 
  name: worker-hp 
  labels: 
    worker-hp: "" 
spec: 
  machineConfigSelector: 
    matchExpressions: 
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]} 
  nodeSelector: 
    matchLabels: 
      node-role.kubernetes.io/worker-hp: "" 

cat /etc/modules-load.d/vlan.conf
8021q

lsmod | grep 8021q

Create a MachineConfig object that defined a kernel argument:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker 
  name: 100-worker-iommu 
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
      - intel_iommu=on iommmu=pt

$ oc create -f 100-worker-kernel-arg-iommu.yaml

Additional kernel modules need to be loaded on the host before deploying JCNR in L3 mode. These modules are usually available in linux-modules-extra or kernel-modules-extra packages. Run the following commands to add the kernel modules:

cat /etc/modules-load.d/crpd.conf
tun
fou
fou6
ipip
ip_tunnel
ip6_tunnel
mpls_gso
mpls_router
mpls_iptunnel
vrf
vxlan

Run the ip fou add port 6635 ipproto 137 command on the Linux host to enable kernel based forwarding.

NetworkManager is a tool in some operating systems to make the management of network interfaces easier. NetworkManager may make the operation and configuration of the default interfaces easier. However, it can interfere with the Kubernetes management and create problems.

To avoid the NetworkManager from interfering with the interface configurations, perform the following steps:

1. Create the file, /etc/NetworkManager/conf.d/crpd.conf.
2. Add the following content in the file.

   [keyfile]
    unmanaged-devices+=interface-name:enp*;interface-name:ens*

   Note: enp* indicates all interfaces starting with enp. For specific interface names, provided a comma-separated list.
3. Restart the NetworkManager service by running the command, sudo systemctl restart NetworkManager.
4. Edit the sysctl file on the host and paste the following content in it:

   net.ipv6.conf.default.addr_gen_mode=0
   net.ipv6.conf.all.addr_gen_mode=0
   net.ipv6.conf.default.autoconf=0
   net.ipv6.conf.all.autoconf=0

5. Run the command sysctl -p /etc/sysctl.conf to load the new sysctl.conf values on the host.

6. Create the bond interface manually. For example:

   ifconfig ens2f0 down
   ifconfig ens2f1 down
   ip link add bond0 type bond mode 802.3ad
   ip link set ens2f0 master bond0
   ip link set ens2f1 master bond0
   ifconfig ens2f0 up ; ifconfig ens2f1 up; ifconfig bond0 up

sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

You can update the core_pattern in /etc/sysctl.conf. For example:

kernel.core_pattern=/var/crash/core_%e_%p_%i_%s_%h_%t.gz