Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Monitoring the System

This section points to parameters that are useful for monitoring the health of the Paragon Active Assurance system.

System Parameters

Standard system parameters:

  • CPU utilization
    • Should stay below 80%
  • Memory utilization, excluding cache and buffers
    • Should stay below 80%
  • Disk utilization
    • Should stay below 80%
  • Network interface load
    • Should stay below 80%

Application Parameters

For Control Center, run the command

The following parameters are of particular interest:

  • test_agent_appliance_online: This indicates how many Test Agent Appliances are currently logged in to the server. The desirable numbers here are of course dependent on how you deploy the Test Agents.
  • scheduled_call_latency: This indicates how far behind schedule the background job processing is. Among other things, this processing collects results from Test Agents and generates periodic reports. The latency should stay below 10 s.

Licenses

The expiry date for the license activated on the server can be inspected with the command

In the output from this command, look for end_date.

Services

The following Paragon Active Assurance services should normally be running:

  • netrounds-callexecuter: Handles background tasks such as
    • Periodically polling monitors: collecting measurement results, updating Test Agent configurations, and evaluating alarms
    • Running scheduled tests
    • Sending periodic reports
  • netrounds-confd: Runs if ConfD has been installed. ConfD is used as an intermediary between the Paragon Active Assurance system and the NETCONF & YANG API. An orchestrator can use this API to configure network equipment and trigger measurements.
  • netrounds-consolidated: Used to proxy internal communication between services.
  • netrounds-license-daemon: From here Control Center learns what licenses to apply to the system. The license daemon uses the Juniper License SDK which is used across all Juniper products.
  • netrounds-metrics: Runs if Streaming API or TimescaleDB is enabled. Receives metrics from Kafka, ingests these metrics into TimescaleDB, and pushes them to the Streaming API.
  • netrounds-plugin: Plugins handles measurement tasks. This service enables plugins to be uploaded to Control Center independently of Control Center upgrades. See the in-app help under "Plugins".
  • netrounds-probe-login: Server which exposes a set of functions to Test Agent Appliances so that they can notify this server about state changes.
  • netrounds-ta3-compat: Needed to achieve full compatibility between Test Agent Applications and Control Center.
  • netrounds-test-agent-gateway: Used to communicate with Test Agent Applications and forward messages between these and Kafka.
  • netrounds-timescaledb: Runs if Streaming API or TimescaleDB is enabled. Time-series database extension for PostgreSQL. Uses a dedicated PostgreSQL database which is distinct from the one used for configuration.
  • openvpn@netrounds: Used to communicate with Test Agent Appliances.

In addition, the following services related to Paragon Active Assurance should be running:

  • apache2: Web server hosting the webapp and the REST API.
  • kafka: Message bus for sharing data asynchronously between services.
  • ntpd: Maintains the system time in synchronization with time servers using the Network Time Protocol (NTP).
  • postgresql: Database for storing configuration and state for the system: basically everything except measurement results and OpenVPN keys.
  • zookeeper: Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Zookeeper is relied on by Kafka.

Finally, it is a good idea to monitor the following system services which are relied on by Paragon Active Assurance:

  • cron: Job scheduler.
  • irqbalance: Distributes hardware interrupts across processors on a multiprocessor system.

Logs

Webapp Logs

The file /var/log/apache/netrounds_access.log contains all HTTP requests made to the Control Center web GUI.

The file /var/log/apache/netrounds_error.log contains all errors reported by Apache for HTTP requests towards the Control Center web GUI. Console output from the Control Center back-end is also in this file; by default, all logging is done by the console.

The configuration of the logging is done using the Python logging module configuration schema (docs.python.org/2/library/logging.config.html#configuration-dictionary-schema).

Call Executer Logs

To see the logs for this service, you can use journalctl, for example to look at the logs from the last hour and then follow all incoming logs:

Plugin Service Logs

For the plugin system you can look at the logs using:

Metrics Service Logs

Note:

These logs are only relevant if you have this feature enabled, which is the case if you are using the Streaming API or TimescaleDB.

TimescaleDB

To view TimescaleDB logs, run:

Kafka

To view Kafka logs, run:

Zookeeper

To view Zookeeper logs, run:

Custom Formatters

Paragon Active Assurance provides two custom formatters that yield additional information about log entries and thus should be favored over Python's default formatters:

  • The formatter netrounds.utils.loggers.ContextFormatter provides record_tags and context_tags that provide additional information about the context of the log entry, for example what Test Agent the log entry is about. Colors are also supported.
  • The formatter netrounds.thirdparty.logstash_formatter.ContextLogstashFormatterV1 outputs a JSON format compatible with Logstash.

The custom formatters are added by default in:

  • LOGGING['formatters']['context']
  • LOGGING['formatters']['context_color']
  • LOGGING['formatters']['logstash']