System Requirements for GCP Deployment

Rocky Linux 4.18.X

Red Hat Enterprise Linux (RHEL): 6.12.x

gVNIC provides higher bandwidth and lower latency than VirtIO NIC. gVNIC is supported starting with Cloud-Native Router Release 26.2. See Miscellaneous Requirements for GCP Deployment for gVNIC-specific configuration requirements.

The K8s version for Google Cloud Marketplace Cloud-Native Router subscription is v1.27.5.

To enable VFIO-PCI for VirtIO NIC deployments, follow the steps below:

cat /etc/modules-load.d/vfio.conf
vfio
vfio-pci

Enable Unsafe IOMMU mode:

echo Y > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
echo Y > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

By default, the Cloud-Native Router vRouter uses 1G hugepages with a minimum allocation of 6 Gi. Starting with Cloud-Native Router Release 26.2, you can configure the vRouter to use 2MB hugepages with a minimum allocation of 3 Gi.

To configure the hugepage size, set the hugepage_sz parameter in the values.yaml file. For more information, see Configure Huge Pages.

For 1G hugepages, see Configure the Number of Huge Pages Available on a Node.

For 2MB hugepages, see Configure 2MB Hugepages for Low Memory Deployments.

Set IOMMU and IOMMU-PT in GRUB.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 default_hugepagesz=1G hugepagesz=1G hugepages=64 intel_iommu=on iommu=pt"

Update grub and reboot.

grub2-mkconfig -o /boot/grub2/grub.cfg 

reboot

Additional kernel modules need to be loaded on the host before deploying Cloud-Native Router in L3 mode. These modules are usually available in linux-modules-extra or kernel-modules-extra packages.

Create a /etc/modules-load.d/crpd.conf file and add the following kernel modules to it:

tun
ipip
ip_tunnel
ip6_tunnel
mpls_router
mpls_iptunnel
vrf
vxlan

(Optional) Enable kernel-based forwarding on the Linux host. This step is only required if you are using DHCP relay.

ip fou add port 6635 ipproto 137

Enable IP Forwarding for VMs in GCP.

1. Specify it as an option while creating the VM. For example:

   gcloud compute instances create instance-name --can-ip-forward

2. For an exisiting VM, enable IP forwarding by updating the compute instance via a file. For example:

   gcloud compute instances export transit-jcnr01 --project jcnr-ci-admin --zone us-west1-a --destination=instance_file_1

   Edit the instance file to set the value canIpForward=true.

   Update the compute instance from the file:

   gcloud compute instances update-from-file transit-jcnr01 --project jcnr-ci-admin --zone us-west1-a --source=instance_file_1 --most-disruptive-allowed-action ALLOWED_ACTION

Enable Multi-IP subnet on Guest OS.

gcloud compute images create debian-9-multi-ip-subnet \
     --source-disk debian-9-disk \
     --source-disk-zone us-west1-a \
     --guest-os-features MULTI_IP_SUBNET 

Add firewall rules for loopback address for VPC.

Configure the VPC firewall rule to allow ingress traffic with source filters set to the subnet range to which Cloud-Native Router is attached, along with the IP ranges or addresses for the loopback addresses.

For example:

Navigate to Firewall policies on the GCP console and create a firewall rule with the following attributes:

1. Name: Name of the firewall rule

2. Network: Choose the VPC network

3. Priority: 1000

4. Direction: Ingress

5. Action on Match: Allow

6. Source filters: 10.2.0.0/24, 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, 10.13.3.3/32

7. Protocols: all

8. Enforcement: Enabled

where 10.2.0.0/24 is the subnet to which Cloud-Native Router is attached and 10.51.2.0/24, 10.51.1.0/24, 10.12.2.2/32, and 10.13.3.3/32 are loopback IP ranges.

Exclude Cloud-Native Router interfaces from NetworkManager control.

NetworkManager is a tool in some operating systems to make the management of network interfaces easier. NetworkManager may make the operation and configuration of the default interfaces easier. However, it can interfere with Kubernetes management and create problems.

To avoid NetworkManager from interfering with Cloud-Native Router interface configuration, exclude Cloud-Native Router interfaces from NetworkManager control. Here's an example on how to do this in some Linux distributions:

1. Create the /etc/NetworkManager/conf.d/crpd.conf file and list the interfaces that you don't want NetworkManager to manage.

   For example:

   [keyfile]
    unmanaged-devices+=interface-name:enp*;interface-name:ens*

   where enp* and ens* refer to your Cloud-Native Router interfaces.
   Note: enp* indicates all interfaces starting with enp. For specific interface names, provided a comma-separated list.
2. Restart the NetworkManager service:

   sudo systemctl restart NetworkManager

3. Edit the /etc/sysctl.conf file on the host and paste the following content in it:

   net.ipv6.conf.default.addr_gen_mode=0
   net.ipv6.conf.all.addr_gen_mode=0
   net.ipv6.conf.default.autoconf=0
   net.ipv6.conf.all.autoconf=0

4. Run the command sysctl -p /etc/sysctl.conf to load the new sysctl.conf values on the host.

Verify the core_pattern value is set on the host before deploying JCNR.

sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

You can update the core_pattern in /etc/sysctl.conf. For example:

kernel.core_pattern=/var/crash/core_%e_%p_%i_%s_%h_%t.gz

sysctl -w net.core.rmem_default=67108864
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_default=67108864
sysctl -w net.core.wmem_max=6710886

Configure Cloud-Native Router for gVNIC.

Starting with Cloud-Native Router Release 26.2, gVNIC is supported as a high-performance NIC option on GCP. gVNIC requires the DPDK driver net_gve and the VFIO driver built into the GCP kernel. No module loading is required.

The following configuration values are recommended for gVNIC deployments. None of these values are mandatory. If not set, Cloud-Native Router uses defaults.

1. MTU: The maximum supported MTU for gVNIC is 8896. The recommended MTU for gVNIC deployments is 8400. Set mtu: "8400" in the jcnr-vrouter section of values.yaml for gVNIC deployments.

2. RX/TX descriptors: The recommended descriptor size is 2048 for gVNIC. To configure, set the following in dpdkCommandAdditionalArgs:

   --dpdk_rxd_sz 2048 --dpdk_txd_sz 2048

   The minimum required by gVNIC is 512 descriptors. Cloud-Native Router automatically adjusts the default value of 256 to the NIC minimum (512) before RX queue setup. If not set, the automatic adjustment ensures gVNIC starts correctly.