Introduction to IP PoolBot
IP PoolBot Overview
IP PoolBot is a cloud-native, container-based Linux application that manages IP address pools in a network. It automatically provisions BNGs with prefixes from its centralized address pool before the BNG’s address pools are depleted. IP PoolBot works with Contrail HealthBot to monitor the state of address pools on BNGs in the network. When the number of free addresses in a pool domain drops below a specified threshold on a managed BNG, IP PoolBot adds unused prefixes to the appropriate address pool in that pool domain on the BNG.
IP PoolBot provides an address management solution that helps operators achieve a major goal of network operations, the efficient allocation of IPv4 addresses. Typical address allocation schemes are complex and not as efficient as network operators need. Addresses are typically preprovisioned on network devices to handle the worst-case load in an attempt to prevent the device from running out of addresses. This means that the devices are over-provisioned for most of their operating time.
IP PoolBot does not preprovision addresses, because committing addresses before they are needed wastes addresses that might never be needed and that could be used elsewhere. Instead of preprovisioning, IP PoolBot monitors BNGs and automatically replenishes their address pools before they are exhausted. This efficient allocation mechanism makes addresses available only when they are needed by the BNGs. Timely and efficient allocation of addresses is affected by specific network considerations such as the number of network devices that consume addresses, the presence of VPNs, system redundancy schemes, and the geographic distribution of network elements.
IP PoolBot uses address reclamation to continually adjust the distribution of prefixes to maximize address space utilization and efficiency. Address reclamation involves draining a pool of all subscribers so that all addresses are free, deprovisioning the pool from the router, and recovering the pool’s unused addresses back to the IP PoolBot centralized pool. IP PoolBot can then reallocate these addresses as needed to other pools among the managed routers that need addresses because they are nearing address exhaustion.
Benefits of IP PoolBot
Efficiency—Improves the efficiency of address utilization by centralizing and automating address allocation for multiple BNGs in the network. IP PoolBot uses just-in-time address allocation, so that it provisions addresses when a BNG needs them. IP PoolBot provisions only as many addresses as the BNG needs. After you partition the IP PoolBot global pool into groups of prefixes, IP PoolBot further subdivides the prefixes to match the BNG’s request. This subdivision enables IP PoolBot to optimize the size of the prefixes that it allocates.
Simplicity—Avoids the overhead and complexity of manual monitoring and provisioning each BNG individually.
Deployability—Installs and operates on any hardware that meets the requirements.
Reclaimability—Reclaims unused addresses from pools that aren’t using many addresses and redistributes these addresses to pools that need them.
You should have a good understanding of IP addressing, classless inter-domain routing (CIDR), variable-length subnet masks (VLSM), and how IP prefixes are subdivided into subnetworks (subnets). When you devise your addressing strategy (outside the scope of this documentation) or use manual address reclamation, you may find it helpful to refer to an IP subnet calculator. You can find many such calculators online.
We use the following terminology in this documentation:
Prefix—A 32-bit IPv4 network address and prefix length expressed using CIDR notation; for example, 198.51.100.0/24. A prefix defines the network portion of an IP address. A prefix represents a subnetwork.
Prefix length—The number of bits that determines the length of the prefix and the size of the network portion of an IP address. A /24 prefix length means that the network portion of the address is 24 bits long. The remaining bits (out of 32) represent the host portion of a network address. For a prefix with a /24 prefix length, the host portion is 8 bits: 32 - 24 = 8.
Network size—This term is sometimes used to mean several different things depending on the context, which can lead to ambiguity. We try to improve clarity by instead describing the prefix length and how it corresponds to the number of host addresses in subnetworks.
A longer prefix, determined by a longer prefix length, corresponds to more subnetworks with fewer host addresses to allocate per subnetwork.
A shorter prefix, determined by a shorter prefix length, corresponds to fewer subnetworks with more host addresses to allocate per subnetwork.
Addresses that have been allocated from a pool domain or pool for a subscriber are referred to as used, allocated, or unavailable addresses.
Addresses that are not currently allocated are referred to as free, unallocated, or available addresses.
How IP PoolBot Works
IP PoolBot manages a centralized collection of IP prefixes for a group of BNGs in the network. The IP PoolBot CLI refers to the managed BNGs as entities. This document generally uses the term BNG, but in some cases uses the term entity.
IP PoolBot provisions and monitors one or more pool domains for each BNG under management. Each pool domain corresponds to a linked address pool for a given logical system:routing instance combination on the BNG. The pool domain configuration specifies a set of network prefixes in the global address pool (a partition) and determines how network prefixes are allocated to the pools in the domain.
IP PoolBot continuously monitors the address utilization of each pool domain and its constituent pools. When the utilization reaches a configurable threshold, IP PoolBot allocates new prefixes to the BNG so that the domain does not run out of available addresses.
Figure 1 shows a high-level view of IP PoolBot operations to monitor BNGs and provision them with the addresses they need, when they need them.
IP PoolBot uses plug-ins to monitor the managed BNGs, to determine when the free address space of their pool domains is nearing depletion, and to provision them with the new prefixes. Figure 2 shows the internal NETCONF plug-in being used for provisioning. The internal HealthBot plug-in interacts with an external instance of HealthBot to monitor the status of managed BNGs.
The following steps explain the general operation of IP PoolBot.
IP PoolBot uses the NETCONF XML management protocol to send messages to discover the initial linked address pool configuration of the BNGs under management. IP PoolBot then updates its database tables accordingly. IP PoolBot runs the discovery process whenever a BNG becomes reachable. Discovery enables IP PoolBot to assume control of existing prefixes on the BNG so that it can manage the pool domains.
IP PoolBot uses HealthBot’s Representational State Transfer (REST) API to program HealthBot to monitor pool domains.
HealthBot uses telemetry to collect information from the managed BNGs about address utilization and state.
IP PoolBot collects that information in the form of alarms sent by HealthBot to IP PoolBot’s message broker interface.
IP PoolBot then evaluates the information from HealthBot. If a BNG is running out of addresses, then IP PoolBot selects an available prefix to allocate. It uses NETCONF messages to provision the BNG with the prefixes.
Figure 3 shows a more detailed view of the relationships among the various functional components of a single IP PoolBot instance. Each manager block shows the database tables it uses.
You can run multiple instances of IP PoolBot simultaneously on multiple different clusters in the network. These instances of IP PoolBot are independent and unaware of each other. They do not share state or configuration.
Each IP PoolBot instance includes the following microservices as functional components of the application:
Entity manager (EM)—Orchestrates the activities of pool management for the BNGs under management. These activities include allocating prefixes, reclaiming addresses, and provisioning the BNGs.
Address manager (AM)—Subdivides the configured prefixes into smaller or equal-size prefixes. The address manager intelligently allocates these new prefixes according to the configured criteria.
Provisioning manager (PM)—Manages the BNG’s configuration related to address pools. For example, the provisioning manager discovers the structure of the existing pools in a linked address pool (the pool domain). It also provisions and deprovisions address pools. The provisioning manager uses the internal jnprNETCONF plug-in and the optional extensible provisioning dictionary to interact with the BNG’s native provisioning mechanism, Junos OS.
Threshold manager (TM)—Monitors the pool domains on the BNG. The threshold manager uses the internal jnprHealthBot plug-in to interact with an external HealthBot server, which uses telemetry to monitor address pool utilization. The plug-in coordinates the publication of threshold notifications when the utilization of pool domains or pool domain elements cross the configured thresholds.
Configuration manager (CM)—Serves as an intermediary between cMGD and the IP PoolBot microservices. It records the configuration into the database and notifies the IP PoolBot microservices of configuration changes. The configuration manager also requests status and statistical information for CLI show and request commands.
cMGD provides a text-based configuration schema and the command line interface (CLI) so that you can configure the global prefix pool, the managed BNGs, and their associated pool domains. The CLI enables you to display statistics and state for various functional components. The output provides information about system load, efficiency, utilization, and errors or abnormal conditions.
Database instance (DB)—Provides shared access to database tables that are used by each of IP PoolBot’s functional components. The database includes tables for address, BNG, and pool domain information. Each database instance is associated with a particular IP PoolBot instance. The database provides persistent storage for configuration information and operational states.
IP PoolBot uses Redis to provide the database structure. Redis provides database redundancy by having a master instance and a replica instance on different worker nodes. If the master instance becomes unavailable, the replica instance becomes the new master. Kubernetes then starts a new replica instance to duplicate the master.
Redundancy requires three worker nodes in addition to the master node. The worker nodes must all be on separate physical servers. However, the nodes can be either physical or virtual machines.
Redis also provides a message broker for communication among IP PoolBot services. In addition, HealthBot uses the message broker as a path to raise alarms to IP PoolBot.
Only the provisioning manager and the threshold manager interact with the managed BNGs in the network through their plug-ins. A given plug-in can interact either directly with a managed BNG or indirectly through an external device manager. Do not confuse the external device manager with the IP PoolBot internal entity manager function. An external device manager is a separate management platform that manages multiple network devices.
The IP PoolBot provisioning manager uses the internal jnprNETCONF plug-in to directly provision and deprovision the BNGs. The BNGs process the received NETCONF messages to implement the configuration changes (new prefixes) on the BNG.
The IP PoolBot threshold manager uses the internal jnprHealthBot plug-in that interacts with HealthBot. HealthBot acts as an external entity manager to monitor the linked address pools of the MX Series routers acting as BNGs.
IP PoolBot Functional Components
The configuration manager is part of the configuration management subsystem. The configuration management subsystem consists of two components: the configuration management service and the configuration manager. The subsystem parses and validates any configuration you perform on IP PoolBot. It then distributes validated configuration information to the other functional components.
The configuration management service (cmgd) is a containerized version of the Junos OS management process, enabling the same CLI structure as Junos OS for configuration and monitoring. It also provides the cMGD NETCONF interface that enables you to remotely manage IP PoolBot.
The configuration manager performs the following tasks:
Loads the initial IP PoolBot configuration from the cmgd service into the database before other IP PoolBot components can proceed to their run-time state.
Translates commands and configurations into actions and parameters that are understood by the IP PoolBot microservices.
Records the initial configuration and subsequent changes in the database for persistence. It notifies IP PoolBot components about any changes.
The entity manager coordinates the operations of other functional components that affect entity state: the provisioning manager and the threshold manager.
For each BNG under management, the entity manager tracks the following information:
The BNG address, which is the transport address of the BNG that hosts managed pools.
A list of the pool domains being managed.
A pool domain represents a linked address pool on the router. For each pool domain, the entity manager tracks the following information:
Pool domain name—A user-defined string that identifies the managed pool for the BNG. Each pool domain name must be unique for that BNG. That means that the pool domain name effectively acts as a key; it is sometimes referred to as the pool domain key.
Pool base name—The base name of the managed address pool. The base name is also called the pool head because it identifies the first pool in the chain of linked address pools. IP PoolBot uses the base name that you configure to determine where a linked pool starts so it can provision the BNG by adding new pools with allocated prefixes to the end of the chain.
If you do not configure the pool base name, then when IP PoolBot creates its first pool in this pool domain, that pool serves as the pool head. All subsequently created pools link from that head.
IP PoolBot uses the format jnpr-ipb-pool-domain-name-sequence-number to name the pools that it creates. The sequence-number is a three-digit increasing value, starting with 000. For example, if the name of a pool domain is test-pd, then IP PoolBot names the first pool jnpr-ipb-test-pd-000. It names the second pool jnpr-ipb-test-pd-001, and so on.
Address bias—A partition name and an IP address to guide the allocation for a pool domain. The partition must match. IP PoolBot tries to allocate a prefix from a root prefix for which the address hint is a valid host address. If no prefixes are available in that root prefix tree, then IP PoolBot tries to allocate a prefix from a different root prefix in the partition. If this is not possible, then IP PoolBot does not allocate any prefix.
Allocation behavior—The desired prefix length when IP PoolBot extends a pool domain and how many successive allocations it can make per address utilization alarm.
Prefixes—An ordered list of the prefixes that make up the pool domain.
The entity manager requests the threshold manager to monitor the BNG for a pool domain. The entity manager provides information so that the address utilization of pools in the pool domain can be collected and reported. The request includes the following:
Pool domain details, such as the name of the pool head and the logical system:routing instance for the domain
Configured used-address threshold and free-address threshold
The entity manager collects a number of volatile statistics for each pool domain, such as the number of critical or nominal alarms, the number of allocated addresses, and timestamps for various IP PoolBot activities. You can display these statistics with IP PoolBot show commands, such as show ip-poolbot entity.
The entity manager requests a new prefix for a pool domain from the address manager when:
You configure a pool domain without specifying the pool head. In this case, the entity manager assigns a name for the first pool, which will serve as the pool head, and requests a prefix for the pool domain.
The entity manager receives a critical pool domain utilization threshold event from the external manager, HealthBot.
The prefix request includes the following information:
Address family—IPv4 is currently supported
Allocation key—The IP address and pool domain of the managed BNG
Address bias—The partition name and IP address that you have configured to guide the allocation for that domain
Requested prefix length—The size of the prefix that you want to be allocated from a partition to a pool domain.
The entity manager subsequently attempts to provision the allocated prefix to the BNG’s address pool. It also provisions an associated static discard route if that is configured for the pool domain.
The entity manager begins a process to discover and reconcile the BNG pool domains under management when either of the following occurs:
The configuration manager notifies the entity manager about a newly configured pool domain.
The provisioning manager notifies the entity manager that a BNG is reachable. The provisioning manager sends a reachability report to the entity manager whenever the reachability state changes. The entity manager will request discovery of all pool domains managed for that BNG.
The discovery process uses the provisioning interface to find the pools in a linked address pool, the structure of the pools in the chain, and the network prefixes that are associated with each pool. IP PoolBot always aligns any concept of the linked address pool, for example from a previous discovery process, with the most recent discovery information.
If the discovered information does not match the existing information, then IP PoolBot attempts to update its databases with the BNG configuration. It reserves or returns to the address manager any prefixes as necessary.
If IP PoolBot discovers a conflict during the update, it flags this as a warning in the log.
The address manager uses a VLSM algorithm to subdivide the root prefixes in the address pool partitions into smaller sub-prefixes up to the max-prefix-len that you have configured for each root prefix. During apportionment, the address manager matches a request for an appropriately sized prefix to a partition and a root prefix. A free sub-prefix from the root prefix is allocated and used to satisfy the apportioning event.
The address manager logs a warning message if the percentage of free addresses within a partition drops below the free-prefix-utilization threshold. That crossing indicates that the partition is in danger of running out of addresses because it has allocated so many addresses.
When you configure IP PoolBot, you assign root prefixes to a partition. The address manager apportions prefixes from only a single partition for any pool domain. Each partition represents an allocation context. The address manager uses the bias that you configure for the pool domain to select the partition from which it subdivides prefixes for allocation to the pool domain.
When you add a root prefix to a partition, make sure it fits within the specified minimum and maximum prefix length limits for that partition:
The min-prefix-len is the shortest prefix length that is valid for a root prefix.
The max-prefix-len is the longest prefix length that is valid for a root prefix.
min-prefix-len <= root prefix length <= max-prefix-len
For example, suppose the min-prefix-len is 20 and the max-prefix-len is 24. That means that you can add a root prefix that has any of the following prefix lengths: /20, /21, /22, /23, or /24.
The smaller the prefix length, the more individual host addresses available in the subnet. The larger the prefix length, the fewer individual host addresses available in the subnet. For example:
A prefix length of /20 provides 4,094 usable host addresses.
A prefix length of /24 provides 254 usable host addresses.
If you configure a root prefix that is outside the specified limits, IP PoolBot does not add it to the partition.
The goals of subdivision are to enable IP PoolBot to share a root prefix among several pool domains and to allow pool domains to grow in smaller increments. The address manager uses a VLSM algorithm to subdivide root prefixes in a partition at configuration. Each subdivision is a subnetwork (subnet).
You can control how deeply the address manager subdivides a root prefix by specifying the maximum allowed prefix length. The max-prefix-length is the longest prefix allowed for a subnet. Consequently it determines the minimum number of host addresses that an allocated prefix must provide.
The address manager can allocate any particular prefix to only one pool domain. Prefix allocation depends on the pool domain’s bias information and the requested prefix size:
Bias partition—The bias partition must be present in the IP PoolBot address pool. The address manager can allocate a prefix only from that partition. If that partition is not present, allocation fails and IP PoolBot logs an error.
Bias address hint—The address manager tries to allocate a prefix from a root prefix for which the IP address hint is a valid host address. If that prefix tree is exhausted, the address manager tries to allocate a prefix from a different root prefix in the partition even though the address hint is not a valid host address for that prefix. If the address manager finds no available prefix, then allocation fails and IP PoolBot logs an error.
Preferred prefix length—The address manager makes a best-effort attempt to match the requested prefix size (preferred-prefix-len) when it allocates a prefix. The partition might not have any prefixes left that match the requested length. For example, when the address manager allocates a superior prefix to a pool, that obviously means that all of its subordinate prefixes are also allocated to the pool.
VLSM creates a hierarchy of subnets from a root prefix. It subdivides the root prefix by adding bits to the prefix length. Each bit added to the prefix length creates another subordinate level of subnets.
Each level has twice as many subnets as the next higher level.
Each level has only half as many host addresses per subnet as the next higher level.
Each root prefix and its associated subnet hierarchies is called a prefix tree. A partition therefore consists of a collection of prefix trees. The address manager can allocate only a prefix that fits somewhere within one of these prefix trees.
Prefixes may be in one of the following states:
Available—The prefix is available for allocation to a pool-domain.
Allocated—The prefix is already allocated to a pool domain and entity.
Reserved—The prefix is administratively reserved with the reserved-prefix statement. IP PoolBot cannot allocate the prefix to a pool domain that does not match the requesting pool domain.
Figure 4 shows a hierarchy for the root prefix 192.0.2.0/24 in partition test-1. You can see that the number of subnets doubles for each bit added to the prefix length, from 1 subnet for /24 to 8 subnets for /27. The number of usable addresses per subnet is halved for each additional prefix length bit, from 254 addresses for /24 to 30 addresses for /27.
Each prefix block in the diagram shows usable addresses:
(total number of addresses) - 2
The two excluded addresses correspond to the lowest address (the network address) and the highest address (the multicast address).
Consider the following scenario with this root prefix tree:
The address manager receives an allocation request with the following information:
Bias partition: test-1
Bias address hint: 192.0.2.121
Preferred prefix length: 25
The address manager looks in partition test-1 for a root prefix to match that includes the address hint. It finds 192.0.2.0/24.
The address manager looks for a /25 prefix that includes the address. 192.0.2.0/25 matches and is selected if it is available.
What happens if 192.0.2.0/25 is not available? That means that 192.0.2.0/24 is also unavailable. The address manager looks for another /25 prefix, even though it won’t match the address hint.
The address manager selects 192.0.2.128/25 if it is available. If that is not available, then the address manager tries to allocate a /25 prefix from a different root prefix.
Now consider a slightly more complex scenario for the same root prefix.
The address manager receives an allocation request with the following information:
Bias partition: test-1
Bias address hint: 192.0.2.200
Preferred prefix length: 27
The address manager checks the 192.0.2.128/25 branch first, because it includes the hinted address. The prefix 192.0.2.192/27 includes the hinted address and it is the preferred length. The address manager selects this prefix if it is available.
If it is not available, then the address manager checks for an available /27 prefix.
If all the /27 prefixes are unavailable in this scenario, then all of the higher-level prefixes are also unavailable. IP PoolBot then checks the other root prefixes in the partition to find an available prefix to allocate for the request.
The provisioning manager uses the internal jnprNETCONF plug-in to perform the following functions:
Discovery—Determines the set of address pools, their relationships, and the prefixes in an existing linked address pool. This determination is based on the name of a routing instance and the name of the pool head in the pool domain. The plug-in can only discover prefixes that match the provisioning schema. For example, that includes a network prefix, a range that excludes both the host address (0) and the subnet broadcast address, and optionally two excluded addresses. (For DHCP clients, the .1 address is automatically excluded and you can optionally exclude one additional address.)
IP PoolBot always reconciles the current information about the linked address pool with what it discovers on the BNG. If IP PoolBot discovers an existing prefix that is a subnetwork of a prefix that IP PoolBot manages, then it cannot subsequently allocate that prefix to another pool domain. If IP PoolBot has already allocated that prefix before the discovery, it flags a duplicate address warning.
You can reserve prefixes for a particular BNG and pool domain to prevent IP PoolBot from allocating them to a different BNG. Prefix reservation is useful when you know that a BNG has already been provisioned with certain prefixes. You cannot be sure that the BNG will be reachable for discovery, which is how IP PoolBot would find these prefixes. If the BNG is unreachable, then IP PoolBot does not learn that the prefixes are already assigned. It might allocate those same prefixes to another BNG. However, if you reserve these prefixes for that preprovisioned BNG before discovery, then IP PoolBot cannot allocate them somewhere else.
The provisioning manager performs discovery whenever:
You configure a new pool domain.
IP PoolBot establishes a connection with the managed BNG. This includes a connection that is reestablished after a connection failure. While the connection is down, an administrator could change the configuration on the BNG. The subsequent discovery can detect this so IP PoolBot can adjust accordingly.
The provisioning manager notifies the entity manager that a BNG is reachable. The provisioning manager sends a reachability report to the entity manager whenever the reachability changes. This can be the initial connection to the BNG or when the BNG becomes reachable after a connection failure. While the connection is down, an administrator could change the configuration on the BNG. The subsequent discovery can detect this so IP PoolBot can adjust accordingly.
A reachable BNG becomes unreachable when both of the following have occurred:
The connection with the BNG is broken.
The provisioning manager has attempted to reconnect for the configured maximum number of retry attempts.
Provisioning—Translates requests to provision and deprovision prefixes on a managed BNG. The plug-in confirms whether the BNG is running a compatible Junos OS release. If the software is compatible, the provisioning manager compiles a set of configuration commands based on the IP PoolBot configuration. The manager then configures the BNG by mapping changes to the Junos configuration syntax on the BNG. The provisioning manager uses the internal, nonconfigurable provisioning dictionary to create the Junos OS syntax. It can also use an optional extensible provisioning dictionary that enables jnprNETCONF to send customized configuration statements based on parameterized variables. The configuration sent by jnprNETCONF becomes part of the Junos OS configuration on the BNG.
The provisioning manager also monitors the connection between IP PoolBot and managed BNGs. When it detects a connection failure, the provisioning manager attempts to reconnect after waiting a configured interval. If that fails, it continues trying to reconnect. The wait interval doubles for each attempt. For example, if the first reconnection attempt fails after an initial wait interval of 10 seconds, then the provisioning manager waits 20 seconds before making a second attempt.
If the provisioning manager reaches the maximum number of retries without reconnecting, it notifies the entity manager that the BNG is unreachable. If this happens, the provisioning manager does not stop trying to reconnect. It tries to reestablish the connection forever, making each attempt at the last computed retry interval.
If the connection is restored on any attempt, the provisioning manager notifies the entity manager that the BNG is reachable. The entity manager requests the provisioning manager to run the discovery process.
During the process of reconnection, the provisioning manager ignores any provisioning actions triggered by HealthBot alerts until it has reestablished the connection and rediscovered the BNG’s pool domains.
Extensible Provisioning Dictionary
You can configure an extensible provisioning dictionary that enables the jnprNETCONF provisioning plug-in to send customized configuration statements to the BNG. This dictionary is an XML file that contains parameterized variables to provision and deprovision the BNGs.
The following requirements apply when you create the dictionary file:
You must name the file
You must place the file in the operational configuration persistent volume by copying the file to the directory where the persistent volume is mapped. (In the documentation we use jnpr-ipb-opcfg-pv as an example name for this volume.)
When the file meets those requirements, jnprNETCONF loads the dictionary during plug-in initialization. The plug-in then validates the file contents. The dictionary must consist of parsable XML with correct semantics. When jnprNETCONF successfully loads the dictionary, it generates an information log message to that effect.
If jnprNETCONF cannot load or parse the file, it generates a warning log and ignores the dictionary. If you correct the file, you must reload the extensible provisioning dictionary. You can do this in either of the following ways:
Restart only the provisioning manager service. This is the preferred method.
$ poolbot kill --services prov-man
Killing service prov-man (pod: jnpr-ipb-provman-5545477c79-9vgs4)...ok New pod for service prov-man is jnpr-ipb-provman-5545477c79-zpbwq $
Restart IP PoolBot, which restarts all the services.
$ poolbot stop Stopping prov-man service... ok Stopping thresh-man service... ok Stopping addr-man service... ok Stopping ent-man service... ok Stopping cfg-man service... ok Stopping haproxy service... ok Stopping dbHaMon service... ok Stopping db service... ok Stopping cmgd service... ok
$ poolbot start Starting service db... ......0 .................1 ok Starting service cmgd... ok Starting service dbHaMon... .........0 ..............1 ............2 ok Starting service haproxy... ........ ok Starting service cfg-man... ...................... ok Starting service addr-man... ok Starting service ent-man... ............. ok Starting service prov-man... ok Starting service thresh-man... ok
The extensible provisioning dictionary must have the following structure:
<?xml version="1.0"?> <cfg_primitives> <device_type name="JUNOS"> <ndriPoolAdd> <command ordinal=”1”> “cfgNdriStatement1” </command> <command ordinal=”2”> “cfgNdriStatement2” </command> </ndriPoolAdd> <ndriPoolDel> <command ordinal=”1”> “cfgNdriStatement1” </command> <command ordinal=”2”> “cfgNdriStatement2” </command> </ndriPoolDel> <poolAdd> <command ordinal=”1”> “cfgNdriStatement1” </command> <command ordinal=”2”> “cfgNdriStatement2” </command> </poolAdd> <poolDel> <command ordinal=”1”> “cfgNdriStatement1” </command> <command ordinal=”2”> “cfgNdriStatement2” </command> </poolDel> </device_type> </cfg_primitives>
The dictionary file has the following components:
The XML declaration must be the first line of the file and it must state version 1.0.
The cfg_primitives element is the top-level element in the file. The next-level element is the device_type element, which specifies the operating system on the managed device. Because IP PoolBot manages only BNGs running Junos OS, you must specify JUNOS as the device_type name.
The device_type element includes two kinds of command group elements, one for default routing instances and one for nondefault instances. Some Junos OS syntax varies depending on whether the default or nondefault routing instance is being configured, so you need different command groups to define your provisioning statements.
The default routing instance command groups are poolAdd and poolDel.
The nondefault routing instance command groups are ndriPoolAdd and ndriPoolDel.
Each command group element includes the specific provisioning or deprovisioning statements that are sent to the BNG.
ndriPoolAdd—The configuration commands in this group are committed whenever a pool is added in a nondefault routing instance.
ndriPoolDel—The configuration commands in this group are committed whenever a pool is deleted from a nondefault routing instance.
poolAdd—The configuration commands in this group are committed whenever a pool is added in the default routing instance.
poolDel—The configuration commands in this group are committed whenever a pool is deleted from the default routing instance.
When statement order matters according to Junos CLI syntax, you must list statements in the correct order within a given command group.
You must ensure that the syntax of the provisioning statements is valid for the targeted BNG. Invalid statements result in a commit failure, which causes the provisioning or deprovisioning operation to fail.
There is no limit to the number of configuration commands you can have in a command group.
You can use the following parameterized variables when you create provisioning statements in the command groups. IP PoolBot resolves each variable to a value that it derives from the current provisioning state of pool domains and their associated pools. After resolving the variables, IP PoolBot sends the statement to the BNG. All variables in a statement must be resolvable to be committed to the BNG.
$EADDR—Address to exclude from the pool being provisioned; configured for the pool domain on the provisioned BNG.
$FAM—Address family of the prefix being provisioned.
$LOCALGW—Local address on the BNG for the address pool that is being provisioned.
$LOOPDEV—Local loopback interface configured for the pool domain on the provisioned BNG; may also be referred to as a loopback device.
$LOOPUNIT—Logical unit number for the loopback interface for the pool domain.
$POOL—Name of the pool being provisioned.
$PREFIX—Prefix being provisioned into the linked address pool.
$RLOW—Low address for the address pool range.
$RHIGH—High address for the address pool range.
$RI—Name of the routing instance where the linked address pool is being provisioned.
$POOLDOMTOTADDR—Total number of available addresses associated with the pool domain. This count includes a pool being added to the domain. The count does not include any of the following: a pool being removed from the domain, excluded addresses, loopback addresses, or addresses outside the default range.
$TAG—Tag to add to the discard route on the BNG.
If you specify a variable that does not apply for a statement, IP PoolBot cannot resolve the statement. When that happens, the unresolved statement is omitted from the set of provisioning statements sent to the BNG.
For example, suppose you did not include a loopback interface when you configured the pool domain for the BNG. If you subsequently include the $LOOPDEV variable in a provisioning statement in the dictionary, IP PoolBot will not include that statement in the set of provisioning commands for the BNG.
The following extensible provisioning dictionary is a simple example of what you can do. In this example, the goal is to provide rudimentary call admission control for L2TP sessions. You want to base this control on the number of addresses present in the pool domain. You specify the $POOLDOMTOTADDR variable in your statements.
Because the value varies as you provision and deprovision pools in the domain, you include the statement each time you add and delete a pool. The maximum-sessions statement is available for both default and nondefault routing instances, so you include it in all the command groups.
<?xml version="1.0"?> <cfg_primatives> <device_type name="JUNOS"> <ndriPoolAdd> <command ordinal=”1”> “set services l2tp maximum-sessions $POOLDOMTOTADDR” </command> </ndriPoolAdd> <ndriPoolDel> <command ordinal=”1”> “set services l2tp maximum-sessions $POOLDOMTOTADDR” </ndriPoolDel> <poolAdd> <command ordinal=”1”> “set services l2tp maximum-sessions $POOLDOMTOTADDR” </command> </poolAdd> <poolDel> <command ordinal=”1”> “set services l2tp maximum-sessions $POOLDOMTOTADDR” </command> </poolDel> </device_type> </cfg_primitives>
The threshold manager programs HealthBot to monitor address pool telemetry from the managed BNGs and to raise events when the telemetry data cross the following configured thresholds:
The free-address threshold is a per-domain count. It is compared to the total number of available addresses in the pool domain; that is, across all the individual pools in the domain. Provisioning of new addresses for the domain occurs when the number of free addresses drops below the threshold.
The used-address threshold is a per-pool count. It is compared to the number of addresses that are in use by an individual pool. When a pool drops below its used-address threshold, it might trigger the auto reclamation procedure if you configured auto-reclamation for the pool domain.
The threshold manager uses the internal jnprHealthBot plug-in to interact with HealthBot for monitoring pool domains as requested by the entity manager. A HealthBot playbook guides the specific monitoring and evaluation that HealthBot performs for the BNGs. HealthBot relays event information directly to the entity manager through a user-defined action.
The jnprHealthBot plug-in performs the following actions:
It instantiates the playbook and associated topic rules with HealthBot for the managed BNGs.
It instantiates a device with HealthBot.
It instantiates a device group for each monitored pool domain. Each device group includes variables such as the logical-system:routing instance, the name of the pool head for the pool domain, utilization threshold values, and other information needed to interact with IP PoolBot.
It modifies device group instances if pool domain attributes change, such as the pool head.
The playbook topic has the following rules:
Collect associated pools for a given logical-system:routing instance and pool head.
Compute the aggregate pool utilization for all pools in the pool domain.
HealthBot user-defined actions notify IP PoolBot about the following:
Whether the pool domain is critical or nominal
Whether the address utilization for an individual pool in the pool domain is
At or above the used-address threshold (nominal)
Below the used-address threshold (critical)
How Address Reclamation Works
Address reclamation recovers underutilized provisioned prefixes from router address pools to IP PoolBot’s centralized pool. IP PoolBot can then reallocate these prefixes as needed to other pools that are nearing address exhaustion. This means that the distribution of prefixes is continually adjusted to maximize address space utilization and efficiency.
Reclamation is based on the used-address threshold that you configure for a pool domain. This threshold applies to each pool in the domain. When the number of used addresses in a pool drops below the used-address threshold, the associated prefix is considered to be underutilized. HealthBot sends a critical pool alarm to IP PoolBot. This alarm indicates that the pool is eligible for reclamation.
Reclamation consists of draining a pool and then recovering the prefixes.
When IP PoolBot initiates a drain, it sends a configuration to the router to begin actively draining the pool. This means that the BNG gracefully shifts subscribers from this address pool to an alternative pool for which active drain is not configured.
When the pool is completely drained, no subscribers are using an address in that pool. All addresses in that pool are free. HealthBot sends an idle pool alarm to IP PoolBot, which then deprovisions the pool and recovers all the addresses to its centralized pool for allocation to other pools.
IP PoolBot can perform address reclamation by either of the following methods:
Manual—You use show commands to monitor BNGs for critical alarms and address utilization by individual pools. You then issue request ip-poolbot commands to drain addresses from a pool and then deprovision the drained pool and recover its addresses.
Automatic—You can configure reclamation to be an entirely automatic process that occurs when IP PoolBot receives a critical or idle alarm for address utilization. You can specify the process to begin immediately or to take place only during a specific time window.
Manual Reclamation of Addresses
Manual reclamation requires you to closely monitor the pool domains and address pools on your managed BNGs. Use the show ip-poolbot critical-pools command to display only pools that HealthBot has flagged as critical. The output lists the BNG IP address, the pool domain and the address pool. It indicates how long the pool has been critical. The output also shows the eligibility state of the pools to indicate which reclamation operation is appropriate for each pool:
The drain eligibility state means that the number of addresses allocated from the pool has dropped below the configured used-address threshold. The pool is eligible to be drained. You can issue the request ip-poolbot drain command to begin draining that pool.
The eligibility is a hint as to which pools you might want to consider draining based on the used-address threshold that you configured. Typically, you will drain only a pool with the drain eligibility state. However, you can choose to drain a different pool if you prefer to keep this pool active.
The reclaim eligibility state means that the pool is not using any addresses. The number of used addresses is 0 and the pool is idle. This does not mean that the pool has an active drain set. If a drain is not in place on the pool, the pool could still allocate addresses.
If you want to reclaim the addresses from this pool, check the pool reclamation status with the show ip-poolbot reclaim-events command:
If the pool is listed in the drain-set reclamation state, the drain is active and the pool will remain idle. You can issue the request ip-poolbot reclaim command to begin reclamation.
If the pool is not listed, issue the request ip-poolbot drain command. When the reclamation state transitions to drain-set, you can issue the request ip-poolbot reclaim command to begin reclamation.
When you issue the request ip-poolbot reclaim command, IP PoolBot deprovisions the pool and returns the addresses drained from the pool to the source partition for reallocation.
You can also issue the show ip-poolbot entity command for a specific pool domain on a managed BNG to get the latest values from HealthBot regarding the pool domain free address count and individual pool used address count. The output also shows the alarm state for the pool domain and its constituent pools.
After you have initiated a drain or reclamation action, the show ip-poolbot reclaim-events command enables you to monitor the reclamation state for each affected pool. You can remove a drain that you have initiated by issuing the request ip-poolbot activate command.
When you choose manual reclamation instead of automatic reclamation, you have to be careful which pool you choose to reclaim. What are some of the considerations for choosing which pool to reclaim?
When you drain a pool, the subscribers using those addresses have to be accommodated in other pools in the pool domain. There must be enough free addresses in the other pools (in the domain) to absorb these subscribers. That means that the number of free addresses in the pool domain must be greater than the number of used addresses in the pool that you drain:
(pool domain free addresses) - (drain pool free addresses) > (drain pool used addresses)
When you drain a pool, you must not leave the pool domain in immediate danger of running out of free addresses. If the domain free address count falls below the free-address threshold, it triggers a critical alarm that results in IP PoolBot provisioning more addresses for the pool. In other words, try not to initiate a drain on a pool unless the following inequality holds true:
(pool domain free addresses) - (drain pool total addresses) > free-address threshold)
The following examples assume that you do not want to reclaim the pool head, Pool 1. They also assume that subscriber numbers and address use are static. However, even as you drain a pool, or consider which pool to drain, subscribers may be getting added to or removed from any of the pools. Further, you may have many more pools in a pool domain. This means that your real-life situation will be more complex, making automatic reclamation a more attractive choice than manual reclamation.
Consider Pool Domain A in Figure 5. The pool domain numbers for total, used, and free addresses are the totals across all four pools. Pool 3 is in a critical state because it is using only 119 addresses, which is less than the used-address threshold of 120.
What happens when you choose Pool 2, Pool 3, or Pool 4?
After the pool is drained, does the pool domain have enough free addresses to absorb the drained subscribers? The answer is yes, for all three pools, so you could choose any of these pools to drain.
531 - 100 = 431; 431 > 300
531 - 281 = 250; 250 > 119
531 - 250 = 281; 281 > 150
After the subscribers are absorbed from the drained pool, does the pool domain have enough free addresses to stay above the free-address threshold? The answer is yes, for all three pools.
531 - 400 = 131; 131 > 50
531 - 400 = 131; 131 > 50
531 - 400 = 131; 131 > 50
Figure 6 shows the same pool domain and pools as before, but the address utilization for Pool 4 is different.
After the pool is drained, does the pool domain have enough free addresses to absorb the drained subscribers? The answer is no, for all three pools.
Pool 2 causes a deficit of 6 addresses.
396 - 100 = 296; 296 < 300
Pool 3 causes a deficit of 4 addresses.
396 - 281 = 115; 115 < 119
Pool 4 causes a deficit of 4 addresses.
396 - 15 = 381; 381 < 385
After the subscribers are absorbed from the drained pool, does the pool domain have enough free addresses to stay above the free-address threshold? The answer is no, for all three pools, because the pool domain has a deficit in each case.
396 - 400 = -4; 131 > 50
396 - 400 = -4; 131 > 50
396 - 400 = -4; 131 > 50
In this situation, pool draining is not appropriate to deal with the under-utilization of addresses by Pool 3. The 119 addresses in use by Pool 3 cannot be fully compensated by the free addresses in Pool 2 (100) and pool 4 (15). If Pool 4 subsequently uses its remaining 15 free addresses, it seems likely that Pool 3 will use more addresses and cross the used-address threshold.
How to use IP PoolBot request commands for Manual Reclamation
Use the following CLI commands for manual reclamation operations. You must specify the router address, the name of the pool domain, and the name of the pool.
To initiate a drain action on a pool:
root@jnpr-ipb-cmgd> request ip-poolbot drain entity ip-address pool-domain name pool pool-name
To recover addresses from a drained pool:
root@jnpr-ipb-cmgd> request ip-poolbot reclaim entity ip-address pool-domain name pool pool-name
To remove the active drain currently in process on a pool:
root@jnpr-ipb-cmgd> request ip-poolbot activate entity ip-address pool-domain name pool pool-name
For example, suppose router 10.4.4.108 has pools in various states of reclamation:
root@jnpr-ipb-cmgd> show ip-poolbot reclaim-events entity 10.4.4.108
Entity Pool Domain Pool State Ref 10.4.4.108 vs009afd jnpr-ipb-vs009afd-000 ineligible 10.4.4.108 vs009afd jnpr-ipb-vs009afd-003 draining jnpr-ipb-vs009afd-000 10.4.4.108 uk001bnf jnpr-ipb-uk001bnf-001 drain-set
Pool jnpr-ipb-vs009afd-000 is in the ineligible reclamation state. This means that you cannot reclaim addresses from this pool.
Pool jnpr-ipb-uk001bnf-001 is in the drain-set state, meaning it is in the process of being drained. You can initiate address reclamation for the pool, but you should first check whether the pool is idle. In the following output snippet, the asterisk indicates that a drain is in place on jnpr-ipb-uk001bnf-001. The pool is not yet idle, because it has 20 addresses in use.
root@jnpr-ipb-cmgd> show ip-poolbot entity address 10.4.4.108 pool-domain uk001bnf
... Pool Prefix Critical Alrm Nominal Alrm Used jnpr-ipb-uk001bnf-000 10.4.4.108/24 0 0 55 jnpr-ipb-uk001bnf-001* 10.4.4.108/24 0 0 20
If you try to manually reclaim the addresses for pool jnpr-ipb-uk001bnf-001 before it is idle, the command displays an error response and also logs an error message with the same information.
root@jnpr-ipb-cmgd> request ip-poolbot reclaim entity 10.4.4.108 pool-domain uk001bnf pool jnpr-ipb-uk001bnf-001
Response error:Pool jnpr-ipb-uk001bnf-001 in pool domain uk001bnf for entity 10.4.4.108 is not eligible for reclaim
When you issue the request after draining is complete—all addresses in the pool are released and no addresses are in use—the command succeeds. IP PoolBot deprovisions the pool and recovers its addresses:
root@jnpr-ipb-cmgd> request ip-poolbot reclaim entity 10.4.4.108 pool-domain uk001bnf pool jnpr-ipb-uk001bnf-001
Pool jnpr-ipb-uk001bnf-001 in pool-domain uk001bnf at entity 10.4.4.108 has been reclaimed.
Pool jnpr-ipb-vs009afd-003 is in the draining state, meaning that IP PoolBot has sent instructions to the BNG to configure an active drain on the pool. You can remove the active drain:
root@jnpr-ipb-cmgd> request ip-poolbot activate entity 10.4.4.108 pool-domain vs009afd pool jnpr-ipb-vs009afd-003
Pool jnpr-ipb-vs009afd-003 in pool domain vs009afd at entity 10.4.4.108 has activated.
The addresses that were already released from subscribers before you removed the drain are now available again for reallocation by the BNG.
You can display critical pools to find pools that are eligible to be drained. For example, pool jnpr-ipb-bet5a-008 has been critical and eligible to drain for over three days.
root@jnpr-ipb-cmgd> show ip-poolbot critical-pools
Entity Pool Domain Pool Age Eligibility 10.4.4.108 vs009afd jnpr-ipb-vs009afd-006 2:33:15 drain 10.2.1.1 bet5a jnpr-ipb-bet5a-008 3 days, 15:20:01 drain
To initiate a drain action on pool jnpr-ipb-bet5a-008:
root@jnpr-ipb-cmgd> request ip-poolbot drain entity 10.2.1.1 pool-domain bet5a pool jnpr-ipb-bet5a-008
Pool jnpr-ipb-bet5a-008 in pool domain bet5a at entity 10.2.1.1 has been set to drain.
Automatic Reclamation of Addresses
Although manual reclamation gives you fine-grained control, it isn’t very practical for large numbers of pools, domains, and entities. Instead, you can configure a few requirements and let IP PoolBot automatically determine and implement an appropriate course of action when it receives a critical alarm for address utilization.
IP PoolBot enforces the following rules for automatic reclamation to minimize issues such as oversubscribing a pool domain or disrupting traffic.
Follow these rules when you perform manual reclamation.
Reclamation must be configured for the pool domain.
The pool domain must not have any ongoing reclamation events (drains or reclamations), because pool domains can have only one reclamation event in effect at a time. If a pool meets reclamation criteria while an event is in process for a domain, IP PoolBot ignores that pool until the domain has no ongoing reclamation events.
This rule helps avoid oversubscribing a domain, because multiple pools in a domain in the drain state increases uncertainty regarding address recovery and provisioning.
Reclamation must be currently allowed. That means that the auto-reclamation configuration specifies always or that the reclamation window is open if it specifies window.
The pool domain must have more than one pool. IP PoolBot will not recover the only pool in a domain.
The pool in critical state must have reached the nominal state at least once. This means that it has used enough addresses to be above the used-address threshold. This rule has two effects:
It protects a provisioned pool from being reclaimed before it’s even been used.
It dampens pool thrashing, where a pool goes through an iterative cycle of provisioning and reclamation.
The pool must be in the critical state longer than the used-address-threshold-age. This buffer period provides an opportunity for the pool to rise above the threshold, returning to nominal state. This requirement can potentially reduce the number and frequency of reclamation operations.
If the critical pool is the pool head, further action depends on the reclaim-pool-domain-head configuration. By default, IP PoolBot never drains the pool head. Other configuration options require IP PoolBot to either always drain a critical pool head or to look for an alternate pool to reclaim.
The number of used addresses in the pool must not exceed the number of free addresses remaining in the pool domain after the pool is drained. Draining a pool in this situation means that the pool domain will be oversubscribed because it will not have enough free addresses to accommodate all the subscribers terminated by the pool drain.
(pool domain free addresses) - (drain pool free addresses) > (drain pool used addresses)
Draining the pool must not drop the number of free addresses across the pool domain below the free-address threshold. Draining a pool in this situation results in the pool being immediately reprovisioned.
(pool domain free addresses) - (drain pool total addresses) > free-address threshold)
You can override this rule in the configuration, which you might do if you want to consolidate addresses and defragment the pools. The newly provisioned replacement pool might start out in the critical state until sufficient subscribers log in to get it over the used-address threshold.
If all these requirements are met, IP PoolBot instructs the router to begin draining the pool.
If any of these requirements are not met, IP PoolBot starts the evaluation for the next pool in the domain.
IP PoolBot has an exception to this list of rules. Before it checks for rule compliance, it evaluates whether a reclaim event was deferred for the critical pool because the pool was already engaged in a drain event. If this is true and the drain event has completed, then IP PoolBot proceeds with pool deprovisioning and recovering the addresses to the source partition.
This behavior enables IP PoolBot to clean up drained pools that are sitting idle. For example, this situation might occur if you configured auto-reclamation for a pool domain to occur only during a maintenance window, but the window expired before reclamation could take place.
Consider Pool Domain B in Figure 7. The pool domain numbers for total, used, and free addresses are the totals across all four pools.
Pools 1, 2, and 3 are all in critical state because their used addresses are below the used-address threshold of 125.
All three pools have been in the critical state longer than the configured used-address-threshold-age.
The order in which the pools went into critical state is Pool 1, then Pool 2, then Pool 3.
The auto-reclamation configuration for the pool head is to use the best alternate pool.
Although Pool 1 has been critical for a longer period, rather than draining Pool 1, IP PoolBot looks for the best alternate. It calculates the reclamation weight for each pool to compare to the pool head’s weight. The weight is the percentage of used addresses in each pool. The Pool 3 weight of .30 is the closest to the Pool 1 weight of .25. Even though Pool 2 has been critical longer than Pool 3 and it has fewer used addresses, its weight of .15 is not as close to Pool 1’s weight. IP PoolBot chooses to drain Pool 3. Before it does, IP PoolBot must check the following:
After Pool 3 is drained, does Pool Domain B have enough free addresses to absorb the drained subscribers? The answer is yes.
1020 - 280 = 740; 740 > 125
After the subscribers are absorbed from the drained Pool 3, does Pool Domain B have enough free addresses to stay above the free-address threshold? The answer is yes.
1020 - 400 = 620; 620 > 50
Figure 8 shows the same pools after the router has drained Pool 3.
Pool Domain B remains above the free-address threshold and in nominal state.
Pool 1 has gained some subscribers and moved above the used-address threshold.
Pool 2 is unchanged and still in critical state. IP PoolBot will consider it for reclamation the next time it evaluates the pool domain’s critical alarms.
Pool 3 is drained of addresses and IP PoolBot will soon schedule it to be deprovisioned by the router. After the pool is deprovisioned, IP PoolBot returns its associated prefix to the source partition for future allocation.
Pool 4 is unchanged and still in nominal state.
In a live system, subscribers will be constantly logging in and out. IP PoolBot receives frequent updates on the current used and free address counts throughout its management environment. There is some delay for HealthBot (the monitoring element) in receiving updates from the routers, and then a further delay before IP PoolBot receives the updates from HealthBot.
This means that in spite of the rules, an automatic reclamation event could still result in a temporary oversubscription of the pool domain. Oversubscription in turn causes a temporary loss of service for some subscribers.
When you monitor IP PoolBot’s reclamation operations, you’ll use the show ip-poolbot entity command. The output shows timestamps for the last discovery, last allocation, and last reclamation events when you display statistics for a router or a specific pool domain. The timestamps are in ISO-8601 format with a 24-hour clock:
T is the delimiter between the date and the time.
Z indicates that the time is in the UTC time zone. If the router time uses a different time zone, the format shows the offset from UTC to identify the time zone.
Time zones west of UTC have a negative offset, designated by -hh:mm.
Time zones east of UTC have a positive offset designated by +hh:mm.
For example, the following timestamps all show the same time, assuming standard time:
2020–03–20T10:10:25-05:00 (New York)
IP PoolBot and Kubernetes
IP PoolBot operates in a Kubernetes cluster environment. IP PoolBot is a containerized application. The containers are packaged by Docker. Kubernetes is the orchestrator for the containers. It groups containers into logical units (pods) that simplify management. The IP PoolBot command and CLI interfaces simplify the interactions with Kubernetes.
A major advantage of Kubernetes as an orchestrator is that it enables the automatic restart of IP PoolBot microservices if they fail. Because it deploys the microservices as replica sets, Kubernetes can ensure that the pod that instantiates a microservice restarts if it fails. Both at initial deployment and upon restart, the service pod checks whether it can connect to the database and message broker and that the configuration has completed loading. When that is confirmed, the IP PoolBot service starts.
Kubernetes provides redundancy for the IP PoolBot functionality, because it distributes and manages the application containers across a cluster that consists of multiple node machines, one master and several workers:
The master node controls the workers. It provides a control plane for decision making and cluster management. You communicate with the master node when you interact with the cluster. A set of processes run on the master node to implement the master node functions.
Worker nodes do the work of the IP PoolBot application. Each worker node must be on a physically separate platform from other worker nodes. If you use VMs to host nodes, the VMs must be on different physical servers.
If a worker node fails, the Kubernetes service that is running on the master node detects the failure. It reschedules the services that were running on the failed node to other worker nodes in the cluster.
The Redis database service that IP PoolBot uses requires at least three physical worker nodes to provide high availability for the application.
Database redundancy is provided by replication. The master database instance is duplicated to one replica database instance. Each instance is a separate pod. An odd number of Redis sentinel instances monitor the master and replica database instance. When a sentinel detects a failure of the master instance, a majority of the sentinels must agree. Then a majority of the sentinels must elect the replica instance to promote to the master role. If the former master instance recovers, it assumes the role of a replica instance.
IP PoolBot-Provisioned Kubernetes Objects
IP PoolBot creates the following setup-layer objects during initial setup. IP PoolBot uses these objects throughout the its life cycle. The objects are deleted when you uninstall IP PoolBot.
Namespace—Virtual cluster of node machines that are running IP PoolBot. All IP PoolBot objects are isolated in the jnpr-ip-poolbot namespace.
External-Services—Objects are created at setup time to obtain the external IP address assigned by the cluster’s load balancer (ingress controller). External services outside the cluster, such as HealthBot, use these external IP addresses to initiate communication with IP PoolBot. If the cluster does not have a backing load balancer, the cluster uses the external IP address of the master node as the external IP address.
cMGD NETCONF—Communicates both internal to the cluster and potentially external to the cluster.
Internal: The configuration manager uses NETCONF over port 22 to connect to cMGD. It loads the initial IP PoolBot configuration this way. It also connects to the MQ Telemetry Transport (MQTT) messaging bus to learn about any configuration changes that cMGD publishes.
External: An external Kubernetes service exposes port 22 outside the cluster as port 8066. This enables the possibility for another, external application manager to manage IP PoolBot through NETCONF messaging.
IPB Message Broker—Provides access to the IP PoolBot message broker and database for HealthBot user-defined actions.
ConfigMap—Stores the configuration file for the database server (redis.conf) and an initial configuration (juniper.conf) for cMGD and the HAProxy service.
PersistentVolumeClaims—For containers that have dynamic data storage requirements. This includes cMGD and the database deployments.