How to Use Command Line Tools to Administer APM
After you've installed the Address Pool Manager (APM) application, you can perform the following administration functions.
Address Pool Manager gives you two command line options for perform administrator tasks. You
can either use the APM utility script (apm) or the Kubernetes Command Line
tool to administer APM.
Access APM Utility Commands
Use the APM utility commands to perform administration functions.
You can use the APM utility script (apm) to administer the application
and to access the CLI that you use to configure the address management functions. The
Juniper APM installation places the utility script in /var/local/apm
and creates a symbolic link to the script in /usr/local/bin/apm.
You can use the apm utility script (which uses the Kubernetes command
line tool and Helm commands ) to do the following:
- Create and delete objects.
- Provide log access.
- Conduct interactive sessions with pod containers.
- Display the status of the APM objects.
Using the apm utility script simplifies many of your administrative
duties. The script performs the tasks you need to manage APM, while masking the complexity
of the kubectl command.
Table 1 lists the commands that you can invoke with the apm utility script and
describes the action that occurs. Many of the individual commands have options that you
can specify.
| Name | Action |
|---|---|
|
|
Clean up unneeded releases and Docker cache. To run this command, you need sudo root privileges. This command offers the following options:
|
|
|
Gives you access to the CLI that you can use to configure APM features. This command offers the following options:
|
|
|
Displays the available contexts for control with APM. This command offers the following options:
|
|
|
Provides access to the Redis database CLI. To run this command, you need sudo root privileges. This command offers the following options:
|
|
|
Displays current state of APM’s database microservice including the current
version, stateful set pods, and their roles. Note: For a
multiple geographically located setup, the
apm db-info displays
the redis roles for workload clusters.This command offers the following options:
|
|
|
Forces the primary database pod to switchover to an eligible backup database
pod. To run this command, you need sudo privileges. This command offers the following options:
Note:
|
|
|
Displays the IP addresses of every service with an external IP address. This command offers the following options:
|
Single geographically located setup
use: Multiple geographically located setup use: sudo -E apm link --context
multi-cluster-context-name
--version software-release --from-running --log
info --no-color --workload-contexts
workload-1-context-name
workload-2-context-name |
Links a cluster to a specific software version. To run this command, you need sudo root privileges. This command offers the following options:
|
apm logs --logset logset-type --services
services-names --context
[context-name
| multi-cluster-context-name]
--workload-context
workload-context-name --log
info --follow --previous --nocolor |
Displays logs for APM microservices. If you run the command without the
|
|
|
Displays the status of the database synchronization between the workload clusters. This command offers the following option:
Note:
This command is only supported in a multiple geographically located setup. |
|
|
Displays the status of the workload clusters from the application’s perspective. This command offers the following option:
Note:
This command is only supported in a multiple geographically located setup. |
|
|
Initiates a switchover to the other workload cluster. The APM microservices are rescheduled on the other workload cluster. To run this command, you need sudo root privileges. This command offers the following option:
This command is only supported in a multiple geography setup. |
|
|
Renames a context. The command does not affect the APM that is currently
running on the cluster. To run this command, you need sudo root privileges. This command offers the following options:
|
|
|
Restart one or more specified services. To run this command, you need sudo
root privileges. This command offers the following options:
|
|
|
Upgrade or start an APM service. To run this command, you need sudo root
privileges. This command offers the following options:
|
sudo -E apm save-config -- context
[context-name
|
multi-cluster-context-name] |
Archives
the current working configuration of APM so that the configuration is preserved
when an This command offers the following options:
|
|
|
Set up the APM application as part of the installation process. To run this command, you need sudo root privileges. This command offers the following options:
|
|
|
Connects you to a running microservice. To run this command, you need sudo
root privileges. This command offers the following options:
|
|
|
Display the current status of the APM services. This command offers the following options:
|
|
|
Stops all APM services. To run this command, you need sudo root
privileges. This command offers the following option:
|
apm storage --context
[context-name
| multi-cluster-context-name]
|
Provides the status of the storage drivers for APM. This command offers the following option:
|
sudo -E apm unlink --context
[context-name
|
multi-cluster-context-name] |
Unlink components associated with the context. To run this command, you need sudo root privileges. This command offers the following option:
|
apm version --context
[context-name
| multi-cluster-context-name]
--detail |
Displays the version of every running microservice in the APM instance as well as the APM utility. It also lists all available APM software releases on the system. This command offers the following option:
|
Use the following general syntax to issue a command:
-
For a short option:
$ apm command-name --option
-
For a long option:
$ apm command-name --option
To target a command at a particular cluster context, use the context option.
$ apm command-name --context context-name
To display a list of available commands with a brief description, use either the
h or help option:
$ apm -h
$ apm --help
To display the options for a specific command:
$ apm command-name -h
To specify the no-color option to disable colored text output (used to
distinguish logs from different microservices):
$ apm command-name --no-color
Upgrade APM to a New Version Using the APM installation Utility
Use this procedure to upgrade to a new version of APM which is installed on a similar cluster deployment (single geography or multiple geography). This procedure assumes APM is running on your system.
Use this procedure to upgrade to a new version of APM which is installed on a cluster that was created by the BBE Cloudsetup utility or by Red Hat OpenShift Container Platform Console. This procedure assumes APM is running on your system.
Start or Stop APM Services Using the APM Utility
Use the apm utility script to start or stop all APM services. The
services start in order of dependency. Essential services (db and mgmt) start first,
followed by the other services. The services stop in reverse order of dependency.
-
To start all APM services:
$ sudo -E apm rollout --context context-name
Note:APM starts from it's initial settings when you execute the
apm setupcommand. Any persistent state is lost when theapm stopcommand is executed. The current configuration can be saved using theapm save-configcommand. The saved configuration is the configuration that is used the next time APM is started. -
To stop all APM services:
$ sudo -E apm stop --context context-name WARNING Shutting down your Address Pool Manager will reset it to factory defaults and you will lose all state on the system. Shutdown will begin in 2 minutes. Please use CTRL+C to cancel.
Setup Secrets Using the APM Utility
You can setup secrets during setup or run the sudo -E apm setup --context
context-name --secrets to setup secrets or update
them.
$ sudo -E apm setup --context context-name --secrets APMi Secret Name (deployed: ) > APMi certificate (default: ) > ./apm.crt Copied /home/user/apm.crt to /var/local/apm/e476597324/secrets/apmi with 600 permissions APMi private key (default: ) > ./apm.key Copied /home/user/apm.keyto /var/local/apm/e476597324/secrets/apmi with 600 permissions APMi root certificate (default: ) > ./rootCA.crt Copied /home/user/rootCA.crt to /var/local/apm/e476597324/secrets/apmi with 600 permissions
If you enter a value for the secret name, you will not be asked for the key or certification files.
Display Database Information Using the APM Utility
The apm db-info command displays current state of APM’s database
microservice including the current version, stateful set pods, and their roles.
Single Geographically Located Setup
$ apm db-info --context context-name [-o|--output json] Version: 6.2.16 Primary: jnpr-apm-redis-0 Backup(s): jnpr-apm-redis-1
Multiple Geographically Located Setup
$ apm db-info --context context-name [--workload-context workload-context-name] [-o|--output json] redis-workload-1: version: 6.2.16-debian-12-r3 Primary: jnpr-apm-redis-workload-1-0 Backup(s): jnpr-apm-redis-workload-1-1 redis-workload-2: version: 6.2.16-debian-12-r3 Primary: jnpr-apm-redis-workload-2-0 Backup(s): jnpr-apm-redis-workload-2-1
apm db-info command, the
workload-context workload-context-name argument is
optional.Check the Status of APM Services Using the APM Utility
Use the apm status utility script to check the status of each APM
service (functional component) listed in Table 2. The status shows whether a service is running, has exited, or has not started. It also
displays the service name on the Kubernetes pod. You can compare uptime for the services
to quickly see whether any service has been restarted.
apm status utility script to check
the status of each APM service for a multiple geography setup.|
Microservice |
Pod Prefix |
|---|---|
|
addrman—Address manager. |
jnpr-apm-addrman |
|
dbSync—Database synchronization. |
jnpr-apm-dbSync |
|
entman—Entity manager. |
jnpr-apm-entman |
|
mgmt—CLI management. |
jnpr-apm-mgmt |
|
apm-operator—Coordinates switchovers and collects system health information. |
jnpr-apm-operator |
|
provman—Provisioning manager. |
jnpr-apm-provman |
|
redis (Remote Dictionary Server)—consists of a set of pods which provide the persistent database. |
jnpr-apm-redis |
To check the status:
For example:
$ apm status --context multi-cluster-context-name --detail MICROSERVICE WORKLOAD CLUSTER POD STATE RESTARTS UPTIME NODE addrman workload-1 jnpr-apm-addrman-workload-1-74d657f6dd-pnj5c Running 0 0:05:52.430246 diatomite-wl1-n-1 addrman workload-2 jnpr-apm-addrman-workload-2-84c58b988-ncdvb Running 0 0:04:09.430327 diatomite-wl2-n-1 dbsync workload-1 jnpr-apm-dbsync-workload-1-84685f955b-rs5ml Running 0 0:05:45.430359 diatomite-wl1-n-1 dbsync workload-2 jnpr-apm-dbsync-workload-2-55965bf696-p2wlq Running 0 0:04:07.430378 diatomite-wl2-n-1 entman workload-1 jnpr-apm-entman-workload-1-6d6b88ccdb-rxnrl Running 0 0:05:50.430397 diatomite-wl1-n-1 entman workload-2 jnpr-apm-entman-workload-2-8644676f96-9tmln Running 0 0:04:08.430412 diatomite-wl2-n-1 mgmt workload-1 jnpr-apm-mgmt-workload-1-5c46f54dc4-lg5j2 Running 0 0:05:51.430427 diatomite-wl1-n-1 mgmt workload-2 jnpr-apm-mgmt-workload-2-56b558875d-5s6ct Running 0 0:04:08.430441 diatomite-wl2-n-1 apm-operator workload-1 jnpr-apm-operator-controller-manager-workload-1-56c55d59df8gchr Running 0 0:05:50.430456 diatomite-wl1-n-1 apm-operator workload-1 jnpr-apm-operator-state-manager-workload-1-66b678469d-l2mfv Running 0 0:05:50.430469 diatomite-wl1-n-1 apm-operator workload-2 jnpr-apm-operator-controller-manager-workload-2-694b6dbc89p5jfm Running 0 0:04:11.430483 diatomite-wl2-n-1 apm-operator workload-2 jnpr-apm-operator-state-manager-workload-2-66c9bbdc6-p2h7j Running 0 0:04:11.430497 diatomite-wl2-n-1 provman workload-1 jnpr-apm-provman-workload-1-6dcc8754bf-fj29b Running 0 0:05:39.430522 diatomite-wl1-n-1 provman workload-2 jnpr-apm-provman-workload-2-779cd68fdc-cmk6m Running 0 0:04:00.430537 diatomite-wl2-n-1 redis-workload-1 workload-1 jnpr-apm-redis-workload-1-0 Running 0 0:06:36.430551 diatomite-wl1-n-1 redis-workload-1 workload-1 jnpr-apm-redis-workload-1-1 Running 0 0:06:19.430570 diatomite-wl1-n-1 redis-workload-1 workload-1 jnpr-apm-redis-workload-1-sentinels-0 Running 0 0:06:04.430590 diatomite-wl1-n-1 redis-workload-1 workload-1 jnpr-apm-redis-workload-1-sentinels-1 Running 0 0:06:04.430610 diatomite-wl1-n-1 redis-workload-1 workload-1 jnpr-apm-redis-workload-1-sentinels-2 Running 0 0:06:04.430628 diatomite-wl1-n-1 redis-workload-2 workload-2 jnpr-apm-redis-workload-2-0 Running 0 0:04:57.430647 diatomite-wl2-n-1 redis-workload-2 workload-2 jnpr-apm-redis-workload-2-1 Running 0 0:04:40.430667 diatomite-wl2-n-1 redis-workload-2 workload-2 jnpr-apm-redis-workload-2-sentinels-0 Running 0 0:04:26.430685 diatomite-wl2-n-1 redis-workload-2 workload-2 jnpr-apm-redis-workload-2-sentinels-1 Running 0 0:04:26.430704 diatomite-wl2-n-1 redis-workload-2 workload-2 jnpr-apm-redis-workload-2-sentinels-2 Running 0 0:04:26.430722 diatomite-wl2-n-1
Display APM IP Addresses Using the APM Utility
Use the apm ip utility script to display the Kubernetes objects that are
necessary for the orchestration of the APM pods. For example:
$ apm ip --context multi-cluster-context-name --detail SERVICE WORKLOAD CLUSTER MICROSERVICE EXTERNAL IP INTERNAL IP PORT(S) apm-apmi-workload-1 workload-1 provman 198.19.0.2 10.43.215.16 20557 apm-apmi-workload-2 workload-2 provman 198.19.0.10 10.47.108.64 20557 apm-mgmt-workload-1 workload-1 mgmt 10.43.244.135 22 jnpr-apm-mgmt-svc workload-1 mgmt 10.43.162.210 8066 jnpr-apm-mgmt-svc workload-2 mgmt 10.47.229.115 8066 apm-mgmt-workload-2 workload-2 mgmt 10.47.244.166 22 apm-operator-workload-1 workload-1 apm-operator 10.43.11.54 20304 apm-operator-workload-2 workload-2 apm-operator 10.47.42.234 20304 apm-ssh-wl1 workload-1 address-pool-manager 10.37.64.128 10.43.212.227 8066 apm-ssh-wl1 workload-2 address-pool-manager 10.37.64.128 10.47.12.177 8066 apm-ssh-wl2 workload-1 address-pool-manager 10.37.64.129 10.43.250.155 8066 apm-ssh-wl2 workload-2 address-pool-manager 10.37.64.129 10.47.237.238 8066 jnpr-apm-redis-sentinels-workload-1-0-svc workload-1 redis-workload-1 10.43.6.167 7381 jnpr-apm-redis-sentinels-workload-1-1-svc workload-1 redis-workload-1 10.43.77.9 7381 jnpr-apm-redis-sentinels-workload-1-2-svc workload-1 redis-workload-1 10.43.139.122 7381 jnpr-apm-redis-workload-1-svc workload-1 redis-workload-1 None 7380 jnpr-dbsync-inter-service-workload-1 workload-1 dbsync 10.43.72.236 61706 jnpr-dbsync-rest-service-workload-1 workload-1 dbsync 10.43.62.234 8080 jnpr-dbsync-inter-service-workload-2 workload-2 dbsync 10.47.25.118 61706 jnpr-dbsync-rest-service-workload-2 workload-2 dbsync 10.47.114.136 8080 jnpr-apm-redis-sentinels-workload-2-0-svc workload-2 redis-workload-2 10.47.249.37 7381 jnpr-apm-redis-sentinels-workload-2-1-svc workload-2 redis-workload-2 10.47.91.138 7381 jnpr-apm-redis-sentinels-workload-2-2-svc workload-2 redis-workload-2 10.47.158.158 7381 jnpr-apm-redis-workload-2-svc workload-2 redis-workload-2 None 7380
Display Logging Using the APM Utility
Use the apm logs utility script to display the logs of events that
occur while APM is running. You can also use the BBE Event Collection and Visualization
utility to display file-based logs collected and stored since the time APM is started. BBE
Event Collection and Visualization is a cloud-based centralized utility that provides a way
to capture APM logs that span the life cycle of APM microservices. You link to the BBE Event
Collection and Visualization logging utility when you set up APM. See the Broadband Edge Event Collection and Visualization
Installation Guide.
Display APM Logging
BBE Event Collection and Visualization is a cloud-based centralized utility that provides a way to capture APM logs that span the life-cycle of APM microservices. If BBE Event Collection and Visualization is running, during the APM setup you can point BBE Event Collection and Visualization to perform the logging. BBE Event Collection and Visualization has a web-based interface to OpenSearch’s capabilities for advanced searching, aggregation, viewing, and data analysis of collected syslog events.
Use the apm logs utility script to display the logs of events
that occur while APM is running. The event logs include events such as those
shown in the following non-exhaustive list:
-
Pool-domain registration events
-
Address allocation failures
-
Network entity connection failures
-
Startup messages
-
Network entity resynchronization events
-
Pool and partition utilization threshold and depletion warnings
By default, APM sends logs to the standard output (stdout) of the service. The output displays the circular buffer of all services or of a specified service. You can also enable logging to follow the log output of the running services. Following the log output creates an open session that continuously streams the logs to stdout.
The APM logging functions mask the underlying complexities of the kubectl
log command that is collecting the log information. You can still
use the kubectl log command, but that is outside the scope of
this documentation.
You can use third-party applications to capture and redirect the stdout stream for the container. Refer to your third-party documentation for assistance. You can also configure Docker with different logging drivers to redirect stdout. Refer to your Docker documentation for assistance.
To display APM logs for all services:
$ apm logs [--context context-name] [--workload-context workload-context-name] [-f] [--previous] [--services SERVICES [SERVICES ...]] [--logset LOGSET] [--nocolor]
Use the apm logs [--context contextName]
[--workload-context workload-context-name]]
--services option only when you are troubleshooting under the
guidance of a Juniper Networks support representative.
To follow the logs for all services, specify the follow option
(-f):
$ apm logs [--context context-name] [--workload-context workload-context-name] -f
To view logs from previous instances of the containers in a Kubernetes pod,
specify the previous container option (-p):
$ apm logs [--context context-name] [--workload-context workload-context-name] -p
You can use standard Ubuntu conventions to redirect the logs to a file or to the terminal and to a file. Refer to the Ubuntu documentation for more information, but you can use the following examples as a starting point:
-
Redirect all logs to only a file.
$ apm logs [--context context-name] [--workload-context workload-context-name] > file-path
-
Redirect all logs to both the screen and to a file.
$ apm logs [--context context-name] [--workload-context workload-context-name] | tee file-path
Use the --services option only for troubleshooting under the
guidance of a Juniper Networks support representative.
You can specify any of the following severity levels, in order of increasing severity:
| Severity Level | Description |
|---|---|
debug |
Detailed information that is typically of interest only when you are trying to diagnose a problem. These logs are often very frequent. |
info |
Events or non-error conditions of interest. Logs at this level provide confirmation that everything is working as expected. These logs are generally not very frequent. |
warning |
Indicates that something unexpected happened or that some problem might occur in the near future. A simple example of the latter is the disk space low warning that indicates that you might run out of disk space soon. In either case, the software is still working as expected, but you might want to monitor it more closely. These logs are generally not very frequent. |
error |
Indicates that a more serious problem has prevented the software from performing some function, but the software has handled the problem as gracefully as possible to continue functioning. |
critical |
A serious error that indicates that the program itself might be unable to continue running. |
You can use the --logset option to display logs either for only
APM services or for only prebuilt services. If you do not use this option, then
only the APM services logs are displayed.
To display logs for only APM services:
$ apm logs [--context context-name] [--workload-context workload-context-name] --logset apm $ apm logs
To display logs for prebuilt services:
$ apm logs [--context context-name] [--workload-context workload-context-name] --logset apm-infra
Prebuilt services are services borrowed from other sources to provide infrastructure functions for APM. These sources include mgmt, redis, and redis-sentinel. For example, redis provides database and messaging services, mgmt provides configuration and CLI services, and so on.
Determine the APM Version Using the APM Utility
Use the apm version [--context context-name] [-o|--output
json] [--detail] [--compare software-version] utility script
to determine the version number of the installed APM release.
To display the release version:
$ apm version --context context-name --detail Address Pool Manager versions: Microservice Release (version) apm: 3.6.0 addrman: 3.6.0 apm-operator: 3.6.0 dbsync: 3.6.0 entman: 3.6.0 mgmt: 3.6.0 provman: 3.6.0 redis: (3.6.0) (6.2.16-debian-12-r3) redis-workload-1: 3.6.0 redis-workload-2: 3.6.0
To compare the specified software release versions against the current deployed release for the specified context:
apm version --compare 3.6.0 –-context context-name components: apm: 3.5.0 -> 3.6.0 addrman: 3.5.0 -> 3.6.0 apm-operator: 3.5.0 -> 3.6.0 entman: 3.5.0 -> 3.6.0 mgmt: 3.5.0 -> 3..0 provman: 3.5.0 -> 3.6.0 redis: 6.2.13 -> 6.2.16-debian-12-r3
Use the -j option to render the version information in JavaScript Object
Notation (JSON) format.
Uninstall and Remove APM Using the APM Utility
Use the apm utility script to uninstall the APM configuration. The
uninstall command reverts the actions you performed when setting up
APM. Use this command to return APM to the state it was in immediately after you installed
the application but before you did any setup configuration.
To uninstall APM:
$ sudo -E apm stop --context context-name sudo -E apm unlink --context context-name
After you uninstall APM, we recommend that you use the Debian uninstall procedure to remove the entire package.
$ sudo sudo -E apm clean [-h] [--log {error | warn | info | debug}] [--no-color] [--docker] [--release release-number] [--dry-run] [--uninstall] [--cluster-repos] ]For the apm clean command options, see the following:
-
-hor--help—Show the help message and exit. -
--logor-l[error | warn | info | debug]—Adjust the log level of the utility scrip. -
--no-color—Print messages without colors. -
--docker—Clean the local docker cache. -
--release release-number—The release to clean (defaults to unused releases). -
--dry-run—List releases or containers that will be removed. -
--uninstall—Uninstall all software releases and remove APM from the system. -
--cluster-repos—Clean the cluster repository for the clusters that have been removed.
How to Access APM Configuration and Operational Commands Using the APM Utility
- Access the APM CLI Using the APM Utility
- Access and Use CLI Configuration Statements Using the APM Utility
- Access and Use CLI Operational Commands Using the APM Utility
Access the APM CLI Using the APM Utility
To access the CLI prompt, enter the following apm utility script
command:
$ apm cli --context context-name [-p|--pipe]
root@jnpr-apm-mgmt
Enter a question mark to see the available top-level CLI commands. This list of commands is a subset of the Junos OS top-level commands.
root@jnpr-apm-mgmt> ? Possible completions: clear Clear information in the system configure Manipulate software configuration information file Perform file operations help Provide help information monitor Show real-time debugging information op Invoke an operation script quit Exit the management session request Make system-level requests restart Restart software process set Set CLI properties, date/time, craft interface message show Show system information ssh Start secure shell on another host start Start shell telnet Telnet to another host test Perform diagnostic debugging traceroute Trace route to remote host
For an overview of Junos OS CLI basics, see Day One: Exploring the Junos CLI. For more detailed information, see the CLI User Guide.
Access and Use CLI Configuration Statements Using the APM Utility
Access and Use CLI Operational Commands Using the APM Utility
To monitor APM, view APM configuration and statistics, or run certain operations manually: