Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Create the Cluster Nodes

This topic describes the steps you must perform to create the Routing Director deployment cluster nodes on the supported deployment platforms.

Creating cluster nodes involves setting up the virtual machines that will serve as the Routing Director cluster nodes, ensuring that they meet the required hardware, storage, and networking specifications before deployment. Perform the steps detailed in this topic, depending on your deployment platform.

On KVM

Perform the following steps to create the node VMs on KVM hypervisors with Ubuntu and RHEL host OS.

In this example, we are deploying the cluster on separate hypervisor servers with the following location and naming parameters:

  • The VMs are in the following data locations (SSDs) on separate hypervisors.

    • VM1 in /data/routing-director1/

    • VM2 in /data/routing-director2/

    • VM3 in /data/routing-director3/

    • VM4 in /data/routing-director4/

    Where VM1, VM2, VM3, and VM4 are the four cluster node VMs.

    If you are creating a cluster with three nodes, skip all steps corresponding to configuring VM4. If you are creating a single node cluster, skip all steps corresponding to configuring VM2, VM3, and VM4.

  • VM1 is named as routing-director1, VM2 is named as routing-director2, VM3 is named as routing-director3, and VM4 is named as routing-director4.

  • The two disk images for each VM are named routing-director-disk1.img and routing-director-disk2.img. Both the disk images for each VM are located in the corresponding routing-directorx directory. Where x is the VM number (1 through 4).

  • For all VMs, in a four-node cluster: CPU = 16, RAM = 32-GB, and Mode = host cpu

    For all VMs, in a three-node cluster: CPU = 24, RAM = 48-GB, and Mode = host cpu

    For the VM in a single node deployment: CPU = 32, RAM = 64-GB, and Mode = host cpu

    These are the bare minimum hardware resources that are required on each node VM.

  • VMs are attached to the br-ex bridged network.

Perform the following steps to create the node VMs.

  1. Log in to the KVM hypervisor CLI.

  2. Ensure that you have the required libvirt, libvirt-daemon-kvm, qemu-kvm, bridge-utils, and qemu-kvm packages installed.

    Use the dnf update and dnf install libvirt libvirt-daemon-kvm qemu-kvm bridge-utils commands to install the packages.

  3. Convert the .vmdk files to the raw format.

    Here, routing-director-disk1.img is the main disk and routing-director-disk2.img is the Ceph disk.

  4. Adjust and expand the disk size.

  5. Create the folder where you want the VM files to be located.

    For example, /data/routing-director1/ for VM1.

  6. Repeat steps 1 through 5 for the remaining node VMs of your cluster by logging into different hypervisors. Create appropriate folders for each VM. For example:

    • /data/routing-director2/ for VM2

    • /data/routing-director3/ for VM3

    • /data/routing-director4/ for VM4

      If you are creating a three-node cluster, skip this folder.

  7. Copy both the routing-director-disk1.img and routing-director-disk2.img raw disk images to the location of each VM using the cp copy command. For example, root@kvm1:~/routing-director# cp routing-director-disk1.img /data/routing-director1/.

    Once all files are copied, the folder and file structure will be similar to:

  8. Generate and customize an XML file for configuring each VM. You must create different XML files for Ubuntu and RHEL. The notable differences in the XML files based on the host OS are:

    • For Ubuntu—machine type is pc-q35-jammy and emulator binary is /usr/bin/qemu-system-x86_64

    • For RHEL—machine type is q35 and emulator binary is /usr/libexec/qemu-kvm

    A sample XML file for Ubuntu for a node in a four-node cluster is described here:

    Save this file as /root/routing-director/routing-director1.xml. In this example:

    • VM1 corresponds to routing-director1.

    • VM1 CPU = 16, VM1 RAM = 32-GB, VM1 Mode = host cpu, since this is a four-node cluster. Change these values to CPU = 24, RAM = 48-GB for a three-node cluster. Change these values to CPU = 32, RAM = 64-GB for a single node deployment.

    • VM1 has two image disk images at /data/routing-director1/routing-director-disk1.img and /data/routing-director1/routing-director-disk2.img

    • The VM is attached to bridged network with name br-ex.

    • The VNC port for graphical console is listening on port 5911. If you want to dynamically assign ports, then set autoport='yes'.

    A sample XML file for RHEL for a node in a four-node cluster is described here:

    Save this file as /root/routing-director/routing-director1.xml. In this example:

    • VM1 corresponds to routing-director1.

    • VM1 CPU = 16, VM1 RAM = 32-GB, VM1 Mode = host cpu, since this is a four-node cluster. Change these values to CPU = 24, RAM = 48-GB for a three-node cluster. Change these values to CPU = 32, RAM = 64-GB for a single node deployment.

    • VM1 has two image disk images at /data/routing-director1/routing-director-disk1.img and /data/routing-director1/routing-director-disk2.img

    • The VM is attached to bridged network with name br-ex

    • The VNC port for graphical console is listening on port 5911. If you want to dynamically assign ports, then set autoport='yes'.

  9. Define the VM using the XML file.

  10. Verify that the VM is registered.

  11. Set the VM to autostart if the KVM is rebooted.

  12. Power on the VM.

  13. Connect to the VM console in one of the following ways:

    • Using the serial console.

      Connect to the VM using the serial console.

    • Using a VNC client.

      Use any VNC compatible client and connect to kvm-ip::5911.

      Ensure that the firewall allows port 5911 for communication between your computer and the VM.

  14. Repeat steps 8 through 13 for the remaining VMs.

    When you create XML files for the remaining VMs, change the VM name and disk locations as appropriate. Also, change the graphical console port numbers for each VM.

  15. (Optional) Once the cluster is successfully deployed, you can delete the .vmdk and .ova files from the hypervisors to free up space.

Best Practices

The best practices for deploying on KVM are reiterated here:

  • Use cache=‘none’ to bypass host cache.

  • Use pc-q35-jammy or newer emulators for Ubuntu KVM or q35 for RHEL KVM​.

  • Ensure that NAT and IP masquerading are disabled on the hypervisor for all the VM IP addresses and VIP addresses.

You have completed the node preparation steps and created all the VMs. You are ready to configure and deploy the cluster. Go to Configure the node VMs.

On Proxmox VE

Perform the following steps to create the node VMs on Proxmox VE hypervisors.

In this example, we are installing Routing Director on a single server with three provisioned datastores, data0, data1, and data2. data0 is used to save the OVA file, data1 to save the first disk image, and data2 to save the second disk image. The VMs are named, VM1, VM2, VM3, and VM4. We are also configuring the bare minimum hardware resources required to deploy a cluster.

If you have only a single disk on your system and you use an LVM rather than the standard recommended file system/directory such as data0, data1, data2, and so on. The importdisk commands as well as the qm set commands on the disk will look different from the ones listed below due to the differences in storage.

For example:

  • qm importdisk 100 routing-director-2.10.0-disk1.vmdk local-lvm

  • qm set 100 -virtio0 local-lvm:vm-404-disk-0,backup=0,replicate=no,discard=on,iothread=on

  • qm set 100 -virtio0 local-lvm:vm-404-disk-1,backup=0,replicate=no,discard=on,iothread=on

Perform the following steps to create the node VMs.

  1. Log in to the Proxmox VE server CLI.

  2. Create the data0, data1, and data2 datastores using the directory method.

    You can use either XFS or EXT4. We recommend using EXT4 since it performs slightly better and survives unplanned reboots better.

  3. Copy the OVA to the data0 datastore and extract the .vmdk files.

    In this example, we have created a folder called ova in data0, copied the routing-director-2.10.0-build.ova file to the ova folder, and used the tar -xvf routing-director-2.10.0-build.ova command to extract the files.

  4. Create the first VM with a VirtIO network interface (net0), and configure the VM name, ID, and memory.

    Here, the VM ID is 100, VM name is VM1, and VM memory is 32-GB for a four-node cluster. Change memory to 48-GB for a three-node cluster. Change memory to 64-GB for a single node deployment.

    Ensure that the VM ID is a unique identifier and is not shared by any other VM on the same server or in the same Proxmox cluster.

  5. Configure the total number of vCPUs as 16 for a four-node cluster. Configure 24 vCPUs for a three-node cluster. Configure 32 vCPUs for a single node deployment.

  6. Configure storage for the VM.

  7. Configure the CPU type as host.

  8. Navigate to the data0/ova folder.

  9. Import disk 1 to the VM as a raw image. Here, import the routing-director-2.10.0-build-disk1.vmdk file to the data1 datastore.

  10. Import disk 2 to the VM as a raw image. Here, import the routing-director-2.10.0-disk2.vmdk file to the data2 datastore.

  11. Assign the raw disks to virtio0 and virtio1 and configure the IO thread, backup, discard, and replication options and as shown.

    If your disks are on the same datastore they will be disk-0 and disk-1 if you are using multiple data stores like data1 and data2 both disks will be disk-0 as they would be the first disk assigned to ID 100 on that particular datastore.

  12. Assign a boot device for the VMs.

    Disk 1 is the primary boot device and Disk 2 is used for Ceph storage.

  13. (Optional) Configure the VM to be a non-ballooning device.

  14. (Optional) If you are not using a GUI for the OS, set tablet as 0 to save on CPU and memory.

  15. (Optional) Set the display option to Standard VGA to save on CPU and memory.

  16. (Optional) If you want to connect to the VM console using the CLI, configure a serial terminal using a socket.

  17. Configure the disk size for both primary boot and Ceph disks.

    1. Log in to the Proxmox VE GUI.

    2. Select VM1, and click Hardware.

    3. Select Hard Disk (virtio0) and click Disk Action > Resize.

    4. Resize the disk to 400-GB. Currently, the disk size must be 250-GB (size=250G). To increment the size to total 400-GB, enter 150 in Size Increment (GiB) and click Resize disk.

    5. Select Hard Disk (virtio1) and click Disk Action > Resize.

    6. Resize the disk to 100-GB. Currently, the disk size must be 50-GB (size=50G). To increment the size to total 100-GB, enter 50 in Size Increment (GiB) and click Resize disk.

  18. Power on the VM.

  19. Launch the VM console.

    • Using CLI—If you have configured a serial terminal, you can access the VM console through the CLI using the following command.

      The VM console appears.

    • Using the GUI—Log in to the Proxmox VE GUI. Select the powered-on VM1, and click >_ Console. The VM console appears.

  20. Repeat steps 4 through 19 for the remaining VMs. Enter appropriate unique VM IDs and names.

Best Practices

The best practices for deploying on Proxmox VE are reiterated here:

  • Use XFS/EXT4 filesystem instead of LVM for hypervisor disk storage.​

  • Configure the following on the VMs:

    • Use VirtIO drivers to boost VM performance and lower I/O latency.

    • Set VGA to std​.

    • Set tablet to 0.

    • Set memory balloon to 0.​

    • Set CPU type to host.

    • Set MACHINE to q35.

  • Ensure that NAT and IP masquerading are disabled on the hypervisor for all the VM IP addresses and VIP addresses.

You have completed the node preparation steps and created all the VMs. You are ready to configure and deploy the cluster. Go to Configure the node VMs.

On VMware ESXi

Perform the following steps to create the node VMs on VMware ESXi hypervisors.

  1. From your Web browser, connect and log in to the VMware ESXi server where you will install Routing Director.

    If you are using a local installer VM, use the browser in the VM to connect to the VMware ESXi server.

  2. Create the node VMs.

    Perform the following steps to create the VMs.

    1. Right-click the Host icon and select Create/Register VM.

      The New virtual machine wizard appears.

    2. On the Select creation type page, select Deploy a virtual machine from an OVF or OVA file.

      Click Next.

    3. On the Select OVF and VMDK files page, enter a name for the node VM.

      Click to upload or drag and drop the OVA file (or the OVF and .vmdk files).

      Review the list of files to be uploaded and click Next.

    4. On the Select storage page, select the appropriate datastore that can accommodate 512-GB SSD for the node VM. Note that SSD is mandatory.

      Click Next. The extraction of files takes a few minutes.

    5. On the Deployment options page:

      • Select the virtual network to which the node VM will be connected.

      • Select the Thick disk provisioning option.

      • Enable the VM to power on automatically.

      Click Next.

    6. On the Ready to complete page, review the VM settings.

      Click Finish to create the node VM.

    7. Increase the sizes for the primary disk and the Ceph storage disk. Right-click the newly created VM on the inventory page, and click Edit Settings > Virtual Hardware. Enter 400-GB for Hard disk 1 which is the primary disk, and 100-GB for Hard disk 2 which is the Ceph storage disk.

    8. Verify that CPU and memory sizes are correct for the size of the cluster deployment or edit them if required.

      For a four-node cluster, the CPU and Memory must be at least 16-vCPU and 32-GB respectively. For a three-node cluster, the CPU and Memory must be at least 24-vCPU and 48-GB respectively. For a single-node cluster, the CPU and Memory must be at least 32-vCPU and 64-GB respectively.

      These are the bare minimum hardware resources that are required on each node VM.

    9. Click OK to save the updates.

    10. To power on the VM, right-click the newly created VM on the Inventory page, and click Power > Power on.

    11. Repeat steps 2.a through 2.j for the remaining node VMs.

      Alternatively, if you are using VMware vCenter, you can right-click the VM, and click the Clone > Clone to Virtual Machine option to clone the newly created VM. Clone the VM three times to create the remaining node VMs.

      Enter appropriate VM names when prompted.

    12. (Optional) Verify the progress of the VM creation in the Recent tasks section at the bottom of the page. When a VM is created, it appears in the VMware Host Client inventory under Virtual Machines.

    13. When all the VMs have been created, verify that the VMs have the correct specifications and are powered on.

Best Practices

The best practices for deploying on ESXi are listed here:

  • Enable Reserve all guest memory to ensure that no swap file is created​.

  • Avoid using a datastore that consists of multiple physical disks unless the disks are part of the hardware-based RAID.

  • Ensure that NAT and IP masquerading are disabled on the hypervisor for all the VM IP addresses and VIP addresses.

You have completed the node preparation steps and created all the VMs. You are ready to configure and deploy the cluster. Go to Configure the node VMs.

On AWS

To create the node VMs on AWS, you must extract the VMDK files and import them to Amazon Machine Image (AMI).

Perform the following steps to import the Routing Director VMDK files to AMI:

  1. Upload the VMDK files to the S3 bucket.
  2. Create a disk specification containers.json file to convert the VMDK to AMI. For example:

    In this example, there are two VMDK files inside the 320310554640-bucket-west2 S3 bucket as the source images.

  3. Convert the VMDK files to AMI.

    Note down the ImportTaskId from the output.

  4. Check the status of the conversion job using the ImportTaskId until the status is Completed.
    Note: Note down the ImageId and ImportTaskId. You need this information when you create EC2 instances.

    For more information on importing VMDK files to AMI, see https://docs.aws.amazon.com/vm-import/latest/userguide/import-vm-image.html.

    To create the security groups for EC2 instances and load balancers, go to Create the Security Groups.

What's Next

To configure the node VMs, go to Configure the node VMs on KVM, Proxmox VE, or VMware ESXi or Create the Security Groups for AWS depending on your VM deployment platform.