Use Juniper BNG CUPS with Multiple Geography Redundancy

In a disaggregated BNG model, the control plane provides services to many user planes and their associated subscribers. The control plane’s role in the disaggregated model enables new use cases, services, and levels of network redundancy. The use cases require that the control plane has new levels of redundancy. Moving the control plane into a cloud backed by a Kubernetes cluster enables these redundancies.

Kubernetes brings scalability, operational efficiency, and reliability to the solution. The modularity of a Kubernetes cloud enables cluster architectures to have unparalleled redundancy. But, even the most redundant cluster architectures are susceptible to events such as natural disasters or cyberattacks which might target a specific location or geography. A multiple geography setup mitigates these susceptibilities.

Figure 1 shows an example of a multiple geography setup.

In a multiple geography two cluster deployment, a Kubernetes cluster exists in each geography. Each cluster acts as a workload cluster capable of scheduling application workloads. The cluster internal/overlay networks of each workload cluster is connected by an IPSec tunnel through the use of Submariner. Interconnecting the overlay networks of the workload clusters enables service discovery across geographies, allowing the application to view the communication domain of the two workload clusters as one large communication domain. This eliminates the complexities of exposing externally routable addresses and securing connections between the application's microservices.

The BNG CUPS Controller deploys to the two workload clusters in microservice pairs. Each microservice on one workload cluster has a twin on the other workload cluster. Microservice pairs that are involved in management, state replication, or cluster monitoring and control, deploy as active/active pairs (state-cache, management, and dbng-operator microservices). All other microservices deploy as active/dormant pairs (CPi and bngd microservices). By deploying the microservices to each workload cluster, the cluster resources are guaranteed to be reserved, and the backup microservice has everything it needs to transition to an active state. Active/dormant pairs are configured with a preferred cluster designation that determines on which workload cluster they initially become active. You issue a CLI command from either instance of the management microservice to switchover an active/backup microservice pair. During the switchover, the system attempts to transition the active twin to a dormant state before activating the dormant twin.

In an active/dormant microservice pair, only the active microservice appears as an eligible target for configuration and operational commands of the CLI.

The CPi and bngd microservices archive their state to the local state cache microservice. Using the secure tunnel, the local state cache microservice replicates its state to its peer state cache service in the remote geography.

Figure 2 Shows a BNG CUPS Controller in a multiple geography setup.

In the event of a workload cluster failure, you can run the CLI command to switchover any microservices that were active on the failed cluster. Since the management microservice on the failed workload cluster is no longer accessible, the CLI commands must be issued from the management microservice on the running workload cluster. In any workload cluster failure scenarios, you must ensure that the workload cluster has indeed failed and that none of the BNG CUPS Controller's microservices are running on that cluster, before activating the backup/dormant microservices on the running workload cluster. There are scenarios where a control plane instance microservice pair is active on both workload clusters. To mitigate these scenarios, a generation number is maintained for each control plane instance. The generation number increments each time a microservice transitions from dormant to active.

The bngd microservice coordinates all switchover activities in the system and maintains the generation numbers for itself and for all the control plane instance microservices. All control plane instance microservices initialize in a dormant state. The bngd microservice is the final arbiter in elevating a control plane instance to an active role. The bngd microservice uses the generation number to resolve any ambiguities about which twin in the microservice pair to activate.

Consider the scenario where a workload cluster fails (for example, a data center power loss) and you activate the bngd microservice twin and the control plane instances on the surviving workload cluster. Then, some time later, power is restored to the failed data center and the microservice are reinitialized to a dormant state. The bngd microservice pair exchange their generation numbers to confirm that the current BNGD microservice twin should remain active. The generation numbers of the CPi microservice pairs are also exchanged and again the currently active CPi microservice remains active by virtue of its higher generation number.

When a cluster switchover occurs, the set of external IP addresses used to contact the BNG CUPS Controller change to the address space used in the new workload cluster. Some back-office applications are more sensitive to IP address changes. For instance, RADIUS wants the value of the NAS-IP-Address to remain constant across cluster switchovers.

The BNG CUPS Controller supports a consistent value for the NAS-IP-Address through a microservice called the Multi-geo IP Route Prioritization Operator (MIRP). The Multi-geo IP Route Prioritization Operator is a microservice that is deployed to each workload cluster.

When a CPi is added, you can provide a persistent external address to use for the CPi’s RADIUS listener address by adding the ip-aaa external-radius-address option to the cpi add command.

When the CPi microservice is deployed, a custom resource is created for the MetalLB IP address pool containing the configured external address. A separate load balancer service is created for the RADIUS listener port with MetalLB annotations to reference the IP address pool. This ensures that the service is assigned the listed external address.

The Multi-geo IP Route Prioritization Operator in the workload cluster, where the CPi’s RADIUS load balancer service is created, generates a BGP advertisement custom resource. This resource contains a LOCAL_PREF value derived from the CPi’s generation number as obtained by the bngd (the generation number is incremented when a cluster switchover event occurs). The BGP advertisement is exchanged with the network load balancer’s IBGP peers. The back-office systems (RADIUS in this case) sit behind the IBGP peers. When the CPi switches over to the other workload cluster, its generation number increments. The associated LOCAL_PREF value of the Multi-geo IP Route Prioritization Operator-generated BGP advertisement is also incremented. Traffic bound for the service’s external IP address is routed to the cluster with the higher LOCAL_PREF value which is also the active workload cluster for the CPi.