Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
 

Troubleshooting

Logging

For troubleshooting it is generally helpful to check the following logs:

  • /var/log/apache/netrounds_access.log: All HTTP requests made to the Control Center web interface.
  • /var/log/apache/netrounds_error.log: Errors reported by Apache for HTTP requests towards the Control Center web GUI. Console output from the Control Center back-end is also in this file; by default, all logging is done by the console.
  • /var/log/syslog: General system log.
  • /var/log/mail.log: Email server log.
  • dmesg: Kernel log

It may also help to turn on additional logging in /etc/netrounds/netrounds.conf by changing the line

to

Another way to show logging for the Paragon Active Assurance callexecuter is to use the journalctl utility:

Read more about journalctl here.

Root Shell Option in Test Agent Local Console

You can perform additional troubleshooting under the guidance of Paragon Active Assurance by logging into the root shell of a Test Agent. This is done from the Test Agent local admin console.

How to access the local console is explained in the in-app help under "Test Agents" > "Configuring Test Agents from the local console".

  • Navigate to Utilities and select Root shell.
  • Log in with password onlyfordebugaccess.

The root shell prompt now appears.

The root password can be changed within the root shell using the passwd command. Alternatively, this can be done using the Change root password option in the local console.

Warning:

Any changes made to the Test Agent in the root shell may cause the Test Agent to malfunction and/or void the warranty. When in doubt, always consult Juniper staff before proceeding.

The subsections that follow deal with specific problems.

Problem: Services cannot be accessed

Suggested actions:

  • Check service status with the following commands:

  • Check that the host name resolves to the correct IP, for instance using dig or ping

  • Check listening ports with

  • Check network traffic with

Problem: No data collected

  • Possible cause: NTP time sync problem. This will introduce a time offset in the server-generated result views, so that it may take a while until any results appear in these views.
  • Possible cause: No free disk space.

Problem: ncc migrate command fails

  • Possible cause: Zookeeper has gone down.

If the ncc migrate command fails with an error message saying "Unable to create Kafka topics", this could be due to Control Center being unable to start the Zookeeper server process. Kafka relies on Zookeeper and requires it to be running in order to operate correctly.

There is a known issue where certain Zookeeper files are corrupted, which causes a Java EOF exception. To check the logs for this, run the following command

and look for ERROR entries related to Zookeeper:

To fix this, delete all files of size zero that are located in the /var/lib/zookeeper/version-2 folder. First identify these files:

Then delete the corrupted file or files:

Finally, restart Zookeeper:

Problem: Services fail

  • Possible cause: Kafka has gone down. This in turn may be due to Zookeeper issues, or to the system having run out of memory.

If this happens, services relying on Kafka will also fail and try to reboot until Kafka comes back online.

For example, if the netrounds-ta3-compat service fails, Test Agent Applications will be unable to register to Control Center.

You can check if Kafka is running with the following command:

If Kafka has gone offline, then checking logs with

will return entries like the following:

Problem: A Test Agent has successfully registered but does not come online (status icon remains red)

  • Possible cause: The network does not allow the encrypted VPN connection to be established. Please check the configuration and logs of any firewall between the Test Agent and Control Center.

Detailed description:

The registration is done over HTTPS, while the connection setup after registration is done using OpenVPN on the same port. This can sometimes cause the network to allow the registration, but not the connection attempt.

  • Possible cause: The clocks on the Test Agent and in Control Center are not in sync. Please check that both parties have NTP correctly configured, that their clocks are in sync with the NTP server, and that the NTP server clock in turn is also in sync.

Detailed description:

If the Test Agent clock is behind, then after the Test Agent registers, the TLS certificate signed by Control Center for the Test Agent will be invalid from the Test Agent's point of view until the Test Agent's clock has reached the time of signing according to the Control Center clock. Until that point, the Test Agent will not accept the certificate and will remain offline.

The TLS certificates are generated at the time of installing Control Center. If the clocks were not correct at this point, the certificates must be regenerated. Note that this also requires re-registration of all Test Agents. Follow these steps:

  1. Make sure that the Control Center clock is in sync:

  2. Remove the old certificates:

  3. Generate new certificates:

  4. Register each Test Agent again under a different name.

Contacting Juniper Technical Support

To contact Juniper technical support, file a ticket at https://support.juniper.net/support/requesting-support. In your ticket, please include the output from the command