Troubleshooting
Logging
For troubleshooting it is generally helpful to check the following logs:
/var/log/apache/netrounds_access.log
: All HTTP requests made to the Control Center web interface./var/log/apache/netrounds_error.log
: Errors reported by Apache for HTTP requests towards the Control Center web GUI. Console output from the Control Center back-end is also in this file; by default, all logging is done by the console./var/log/syslog
: General system log./var/log/mail.log
: Email server log.dmesg
: Kernel log
It may also help to turn on additional logging in
/etc/netrounds/netrounds.conf
by changing the line
LOGGING['handlers']['console']['level'] = 'ERROR'
to
LOGGING['handlers']['console']['level'] = 'DEBUG'
Another way to show logging for the Paragon Active Assurance callexecuter is to use the
journalctl
utility:
journalctl -u netrounds-callexecuter
Read more about journalctl
here.
Root Shell Option in Test Agent Local Console
You can perform additional troubleshooting under the guidance of Paragon Active Assurance by logging into the root shell of a Test Agent. This is done from the Test Agent local admin console.
How to access the local console is explained in the in-app help under "Test Agents" > "Configuring Test Agents from the local console".
- Navigate to Utilities and select Root shell.
- Log in with password
onlyfordebugaccess
.
The root shell prompt now appears.
The root password can be changed within the root shell using the passwd
command. Alternatively, this can be done using the Change root
password option in the local console.
Any changes made to the Test Agent in the root shell may cause the Test Agent to malfunction and/or void the warranty. When in doubt, always consult Juniper staff before proceeding.
The subsections that follow deal with specific problems.
Problem: Services cannot be accessed
Suggested actions:
-
Check service status with the following commands:
sudo systemctl status "netrounds-*" openvpn@netrounds sudo systemctl status apache2 openvpn@netrounds
-
Check that the host name resolves to the correct IP, for instance using
dig
orping
-
Check listening ports with
netstat -lan
-
Check network traffic with
tcpdump
Problem: No data collected
- Possible cause: NTP time sync problem. This will introduce a time offset in the server-generated result views, so that it may take a while until any results appear in these views.
- Possible cause: No free disk space.
Problem: Services fail
- Possible cause: Kafka has gone down.
If this happens, services relying on Kafka will also fail and try to reboot until Kafka comes back online.
You can check if Kafka is running with the following command:
sudo systemctl list-units --type=service | grep kafka
If Kafka has gone offline, then checking logs with
journalctl -u netrounds-test-agent-gateway.service
will return entries like the following:
Oct 17 11:17:05 example.com test-agent-gateway-service[23009]: {"level":"info","service":"test-agent-gateway- service","host":"example.com","src":"core","time":"2020-10-17T11:17:05.429473575Z","caller":"/app/cmd/server/ Oct 17 11:17:06 example.com systemd[1]: netrounds-test-agent-gateway.service: Main process exited, code=exited, status=1/FAILURE Oct 17 11:17:06 example.com systemd[1]: netrounds-test-agent-gateway.service: Unit entered failed state. Oct 17 11:17:06 example.com systemd[1]: netrounds-test-agent-gateway.service: Failed with result 'exit-code'. Oct 17 11:17:07 example.com systemd[1]: netrounds-test-agent-gateway.service: Service hold-off time over, scheduling restart. Oct 17 11:17:07 example.com systemd[1]: Stopped Netrounds TA3 Connection Service.
Problem: A Test Agent has successfully registered but does not come online (status icon remains red)
- Possible cause: The network does not allow the encrypted VPN connection to be established. Please check the configuration and logs of any firewall between the Test Agent and Control Center.
Detailed description:
The registration is done over HTTPS, while the connection setup after registration is done using OpenVPN on the same port. This can sometimes cause the network to allow the registration, but not the connection attempt.
- Possible cause: The clocks on the Test Agent and in Control Center are not in sync. Please check that both parties have NTP correctly configured, that their clocks are in sync with the NTP server, and that the NTP server clock in turn is also in sync.
Detailed description:
If the Test Agent clock is behind, then after the Test Agent registers, the TLS certificate signed by Control Center for the Test Agent will be invalid from the Test Agent's point of view until the Test Agent's clock has reached the time of signing according to the Control Center clock. Until that point, the Test Agent will not accept the certificate and will remain offline.
The TLS certificates are generated at the time of installing Control Center. If the clocks were not correct at this point, the certificates must be regenerated. Note that this also requires re-registration of all Test Agents. Follow these steps:
-
Make sure that the Control Center clock is in sync:
ntpq -np
-
Remove the old certificates:
rm /var/lib/netrounds/openvpn/*
-
Generate new certificates:
dpkg-reconfigure netrounds-probe-login
-
Register each Test Agent again under a different name.
Contacting Juniper Technical Support
To contact Juniper technical support, file a ticket at https://support.juniper.net/support/requesting-support. In your ticket, please include the output from the command
ncc generate-troubleshooting-report