Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Configuring a NorthStar Cluster for High Availability

 

Configuring a cluster for high availability (HA) is an optional process. If you are not planning to use the HA feature, you can skip this topic.

The following sections describe the steps for configuring, testing, deploying, and maintaining an HA cluster.

Before You Begin

  • Download the NorthStar Controller and install it on each server that will be part of the cluster. Each server must be completely enabled as a single node implementation before it can become part of a cluster.

    This includes:

    • Creating passwords

    • License verification steps

    • Connecting to the network for various protocol establishments such as PCEP or BGP-LS

    Note

    All of the servers must be configured with the same cassandra password.

    See NorthStar Controller 3.0.0 RPM Bundle Installation in the NorthStar Controller Getting Started Guide for instructions.

  • Run the net_setup.py utility to complete the required elements of the host and JunosVM configurations as described in NorthStar Controller 3.0.0 RPM Bundle Installation in the NorthStar Controller Getting Started Guide, and keep that configuration information available.

    Note

    If you are using an OpenStack environment, you will have one JunosVM that corresponds to each NorthStar Controller VM.

  • Confirm that all servers that will be in the cluster are part of the same subnet if virtual IP is required for that network.

  • Decide on the priority that each node will have for active node candidacy upon failover. The default value for all nodes is 0, the highest priority. If you want all nodes to have equal priority for becoming the active node, you can just accept the default value for all nodes. If you want to rank the nodes in terms of their active node candidacy, you can change the priority values accordingly—the lower the number, the higher the priority.

  • Know the virtual IPv4 address you want to use for Java Planner client and Web UI access to NorthStar Controller (required). This virtual IP address is configured for the router-facing network for single interface configurations, and for the user-facing network for dual interface configurations. This address is always associated with the active node, even if failover causes the active node to change.

Set Up SSH Keys

Set up SSH keys between the selected node and each of the other nodes in the cluster, and each JunosVM.

  1. Obtain the public SSH key from the selected node. You will need the ssh-rsa string from the output:
  2. Copy the public SSH key from the selected node to each of the other nodes
  3. Copy the public SSH key from the selected node to each remote JunosVM (JunosVM hosted on each other node). To do this, log in to each of the other nodes and connect to its JunosVM.

Access the HA Setup Main Menu

The /opt/northstar/utils/net_setup.py utility (the same utility you use to configure NorthStar Controller) includes an option for configuring high availability (HA) for a node cluster. Run the /opt/northstar/utils/net_setup.py utility on one of the servers in the cluster to set up the entire cluster.

  1. Select one of the nodes in the cluster on which to run the setup utility to configure all the nodes in the cluster.
  2. On the selected node, launch the NorthStar setup utility to display the NorthStar Controller Setup Main Menu.

    Figure 1 shows the NorthStar Controller Setup Main Menu.

    Figure 1: NorthStar Controller Setup Main Menu
    NorthStar Controller
Setup Main Menu
  3. Type E and press Enter to display the HA Setup main menu.

    Figure 2 shows the top portion of the HA Setup main menu in which the current configuration is listed. It includes the five supported interfaces for each node, the virtual IP addresses, and the ping interval and timeout values. In this figure, only the first of the nodes is included, but you would see the corresponding information for all three of the nodes in the cluster configuration template. HA functionality requires an odd number of nodes in a cluster, and a minimum of three.

    Figure 2: HA Setup Main Menu, Top Portion
    HA Setup Main Menu,
Top Portion
    Note

    If you are configuring a cluster for the first time, the IP addresses are blank and other fields contain default values. If you are modifying an existing configuration, the current cluster configuration is displayed, and you have the opportunity to change the values.

    Figure 3 shows the lower portion of the HA Setup main menu. To complete the configuration, you type the number or letter of an option and provide the requested information. After each option is complete, you are returned to the HA Setup main menu so you can select another option.

    Figure 3: HA Setup Main Menu, Lower Portion
    HA Setup Main Menu,
Lower Portion

Configure the Three Default Nodes and Their Interfaces

The HA Setup main menu initially offers three nodes for configuration because a cluster must have a minimum of three nodes. You can add more nodes as needed.

For each node, the menu offers five interfaces. Configure as many of those as you need.

  1. Type 5 and press Enter to modify the first node.
  2. When prompted, enter the number of the node to be modified, the hostname, and the priority, pressing Enter between entries.Note

    The NorthStar Controller uses root as a username to access other nodes.

    The default priority is 0. You can just press Enter to accept the default or you can type a new value.

    For each interface, enter the interface name, IPv4 address, and switchover (yes/no), pressing Enter between entries.

    Note

    For each node, interface #1 is reserved for the cluster communication interface which is used to facilitate communication between nodes. For this interface, it is required that switchover be set to Yes, and you cannot change that parameter.

    When finished, you are returned to the HA Setup main menu.

    The following example configures Node #1 and two of its available five interfaces.

  3. Type 5 and press Enter again to repeat the data entry for each of the other two nodes.

Configure the JunosVM for Each Node

To complete the node-specific setup, configure the JunosVM for each node in the cluster.

  1. From the HA Setup main menu, type 7 and press Enter to modify the JunosVM for a node.
  2. When prompted, enter the node number, the JunosVM hostname, and the JunosVM IPv4 address, pressing Enter between entries.

    Figure 4 shows these JunosVM setup fields.

    Figure 4: Node 1 JunosVM Setup Fields
    Node 1 JunosVM Setup
Fields

    When finished, you are returned to the HA Setup main menu.

  3. Type 7 and press Enter again to repeat the JunosVM data entry for each of the other two nodes.

(Optional) Add More Nodes to the Cluster

If you want to add additional nodes, type 1 and press Enter. Then configure the node and the node’s JunosVM using the same procedures previously described. Repeat the procedures for each additional node.

Note

HA functionality requires an odd number of nodes and a minimum of three nodes per cluster.

The following example shows adding an additional node, node #4, with two interfaces.

The following example shows configuring the JunosVM that corresponds to node #4.

Configure Cluster Settings

The remaining settings apply to the cluster as a whole.

  1. From the HA Setup main menu, type 8 and press Enter to configure the virtual IP address for the external (router-facing) network for single interface configurations. Skip this step if you are configuring a separate user-facing network interface. This is the virtual IP address that is always associated with the active node, even if failover causes the active node to change.Note

    Make a note of this IP address. If failover occurs while you are working in the NorthStar Controller Network Planner UI, the client is disconnected and you must re-launch it using this virtual IP address. For the Web UI, you would be disconnected and would need to log back in.

    The following example shows configuring the virtual IP address for the external network.

  2. Type 8 and press Enter to configure the virtual IP address for the user-facing network for dual interface configurations. If you do not configure this IP address, the router-facing virtual IP address also functions as the user-facing virtual IP address.
  3. Type B and press Enter to configure the setup mode as cluster.
  4. Type C and press Enter to configure the PCEP session. The default is physical_ip. If you are using the cluster virtual IP (VIP) for your PCEP session, configure the PCEP session as vip.Note

    All of your PCC sessions must use either physical IP or VIP, and that must also be reflected in the PCEP configuration on the router.

Test and Deploy the HA Configuration

You can test and deploy the HA configuration from within the HA Setup main menu.

  1. Type E to test the HA connectivity for all the interfaces. You must verify that all interfaces are up before you deploy the HA cluster.
  2. Type F and press Enter to launch a script that connects to and deploys all the servers and all the JunosVMs in the cluster. The process takes approximately 15 minutes, after which the display is returned to the HA Setup menu. You can view the log of the progress at /opt/northstar/logs/net_setup.log.Note

    If the process has not completed within 30 minutes, a process might be stuck. This is sometimes evident upon examining the log available at /opt/northstar/logs/net_setup.log. You can press Ctrl-C to cancel the script, and then restart it.

  3. When the script execution is complete, view cluster information and check the cluster status by typing I and pressing Enter. In addition to providing general cluster information, this option launches the ns_check_cluster.sh script. You can also run this script outside of the setup utility by executing the following commands:
  4. (Optional) Examine the processes running on a specific node by logging into that node and executing the supervisorctl status script.

    For an active node, all processes should be listed as RUNNING. Figure 5 shows example output for an active node. Your actual list of processes could be longer.

    Figure 5: Sample of Processes Running on an Active Node
    Sample of Processes
Running on an Active Node

    For a standby node, NorthStar processes are listed as STOPPED, while other processes remain running to preserve connectivity. Figure 6 shows example output for a standby node. Your actual list of processes could be longer.

    Figure 6: Sample of Processes Running on a Standby Node
    Sample of Processes
Running on a Standby Node

Replace a Failed Node if Necessary

On the HA Setup menu, options G and H can be used when physically replacing a failed node. They allow you to replace a node without having to redeploy the entire cluster which would wipe out all the data in the database.

Warning

While a node is being replaced in a three-node cluster, HA is not guaranteed.

  1. Replace the physical node in the network and install NorthStar Controller on the replacement node.
  2. Run the NorthStar setup utility to configure the replaced node with the necessary IP addresses. Be sure you duplicate the previous node setup, including:
    • IP address and hostname

    • Initialization of credentials

    • Licensing

    • Network connectivity

  3. Go to one of the existing cluster member nodes (preferably the same node that was used to configure the HA cluster initially). Going forward, we will refer to this node as the anchor node.
  4. Set up the SSH key from the anchor node to the replacement node and JunosVM.

    Copy the public SSH key from the selected node to each of the replacement nodes.

    Copy the public SSH key from the anchor node to the replacement node’s JunosVM (the JunosVM hosted on each of the other nodes). To do this, log in to each of the replacement nodes and connect to its JunosVM.

  5. From the anchor node, remove the failed node from the Cassandra database. Run the command nodetool removenode host-id. To check the status, run the command nodetool status.

    The following example shows removing the failed node with IP address 10.25.153.10.

    [root@node-1 ~]# . /opt/northstar/northstar.env
    [root@node-1 ~]# nodetool status
    [root@node-1 ~]# nodetool removenode b985bc84-e55d-401f-83e8-5befde50fe96
    [root@node-1 ~]# nodetool status
  6. From the HA Setup menu on the anchor node, select option G to copy the HA configuration to the replacement node.
  7. From the HA Setup menu on the anchor node, select option H to deploy the HA configuration, only on the replacement node.

Configure Fast Failure Detection Between JunosVM and PCC

You can use Bidirectional Forward Detection (BFD) in deploying the NorthStar application to provide faster failure detection as compared to BGP or IGP keepalive and hold timers. The BFD feature is supported in PCC and JunosVM.

To utilize this feature, configure bfd-liveness-detection minimum-interval milliseconds on the PCC, and mirror this configuration on the JunosVM. We recommend a value of 1000 ms or higher for each cluster node. Ultimately, the appropriate BFD value depends on your requirements and environment.