Troubleshooting Site Activation Issues
Prerequisites to Activate a Site
Problem
Description: User was unable to activate a site. Specify the prerequisites to activate a site.
Solution
The prerequisites to activate a site are as follows:
- Check the spoke to hub underlay reachability for IPsec/GRE tunnels or the SSH connection and vice versa.
- Check the hub or spoke to CSO (regional MS) reachability or the SSH connection and vice versa.
- Check the hub to CSO (regional MS) reachability or the SSH connection and vice versa.
- Check the firewall policies between the CPE device and the CSO. The hub or spoke must be able to communicate to CSO through ports 443 (activation), 444 (activation for small and medium deployments), 7804 (outbound-ssh), 3514(app-track logs), 514 (syslog), and 2216 (telemetry agent). See Contrail Service Orchestration (CSO) Deployment Guide
Activation Failure for a Hub site
Problem
Description: A failure occurred when activating a hub site.
Solution
- Check the job logs in the Administration Portal for the
activation failure and the reason for the failure.
- Log in to the Administration Portal and select Monitor
> Jobs.
The Jobs page is displayed.
- Select the failed log and click the Detailed View icon
that appears before it.
The Detailed View page appears, showing the details of the job and the number of tasks associated with the job.
- Click View Logs.
The Job status page is displayed.
- Log in to the Administration Portal and select Monitor
> Jobs.
- If ZTP is enabled in the hub device template, then ensure
that the hub device or image supports the phone-home feature. If the
feature is not supported, then upgrade the software image.
If you need to disable ZTP in the hub device template, log in to Administration Portal and select Resources > Device Templates > Template Name > Edit Device Template > Template Settings. Disable ZTP_ENABLED option.
If you need to copy the stage-1 configuration to the hub, then log in to Administration Portal and select Resources > Cloud Hub Devices > Stage 1 Config and copy the configuration.
- Check the outbound SSH connection between the hub and
the regional microservices virtual machine on port 7804.
Log in to the CSO regional microservices virtual machine and execute the following command
root@regionalmsvm:~#netstat -anp | grep 7804tcp 0 0 0.0.0.0:7804 0.0.0.0:* LISTEN 1254/haproxy tcp 0 0 192.0.2.0:7804 192.0.3.0:7310 ESTABLISHED 1254/haproxy >>> Spoke tcp 0 0 192.0.2.0:7804 192.0.4.0:14632 ESTABLISHED 1254/haproxy >>> Hub root@regionalmsvm:~#
If the outbound SSH connection is not established between the hub and the regional microservices virtual machine,
Ensure that TCP port 7804 is not blocked in the path.
Check the reachability between the hub and the regional microservices virtual machine,. Check whether the hub device can establish an SSH connection with the regional microservices virtual machine and vice versa.
View the detailed failure logs in the Kibana dashboard http://regional-infra-IP-Address:5601 or log in to regional microservices virtual machine and execute the following command to view the detailed failure logs.
root@regionalmsvm:~#kubectl get pods –n regional | grep activationcsp.csp-activation-service-1888452022-fv1vt 1/1 Running 1 19h
root@regionalmsvm:~#kubectl logs -f csp.csp-activation-service-1888452022-fv1vt –n regional
If the outbound SSH connection is established, then check if the configurations are pushed to the device.
Log in to the Administration Portal. Select Monitor > Jobs. Click the ZTP failure log and verify the configuration deployment task.
To view the detailed log, use the Kibana dashboard <http://central infra ip:5601> or log in to the central microservices virtual machine and execute the following command.
root@centralmsvm:~# kubectl get pods –n central | grep cmscsp.csp-cms-central-2820689874-gvjbh 1/1 Running 1 19h csp.csp-cms-central-core-2224266535-kmplk 1/1 Running 1 19h root@centralmsvm:~# kubectl logs -f csp.csp-cms-central-core-2224266535-kmplk –n central
Verify that the configurations are pushed successfully to the device.
- For further troubleshooting, collect the logs and output results and contact Juniper Networks Technical Support team.
Activation Failure for a Spoke Site
Problem
Description: Activation failed for a spoke site.
Solution
- Check the job logs in the Administration Portal for the
activation failure and the reason for the failure.
- Log in to the Administration Portal and select Monitor
> Jobs.
The Jobs page is displayed.
- Select the failed log and click the Detailed View icon
that appears before the failed log name.
The Detailed View page appears, showing the details of the job and the number of tasks associated with the job.
- Click View Logs.
The Job status page is displayed
- Log in to the Administration Portal and select Monitor
> Jobs.
- Check the Internet reachability. If Juniper Networks redirect
server is used for CPE ZTP or activation, then ensure that the CPE
device can establish a connection to the Internet. The CSO activation
server IP address (regional microservices virtual machine IP address
for large deployments and central microservices virtual machine IP
address for small and medium deployments), activation server certificate
and the CPE serial numbers are configured in the Juniper Networks
redirect server.
Copy the certificate from the regional microservices virtual machine using the following command,
For large deployments
root@regionalmsvm:~#ls -l /etc/pki/tls/certs/ssl_cert.crt-rw-r--r-- 1 root root 1306 Dec 2 10:08 /etc/pki/tls/certs/ssl_cert.crt root@regionalmsvm:~#
For small and medium deployments
root@centralmsvm:~#ls -l /etc/pki/tls/certs/ssl_cert.crt-rw-r--r-- 1 root root 1306 Dec 2 10:08 /etc/pki/tls/certs/ssl_cert.crt root@centralmsvm:~#
- If Juniper Networks redirect server is not used for CPE
ZTP or activation, then configure the phone-home server in the CPE
device and copy the certificate to the CPE device.
For large deployments
root@cpe-srx #show system phone-homeserver https://regional-ms-ip; ca-certificate-file /root/ssl_cert.crt;
For small and medium deployments
root@cpe-srx #show system phone-homeserver https://central-ms-ip:444; ca-certificate-file /root/ssl_cert.crt
- Check the outbound SSH connection between the spoke and
the microservices virtual machine on port 7804.
For large deployments
root@regionalmsvm:~#netstat -anp | grep 7804tcp 0 0 0.0.0.0:7804 0.0.0.0:* LISTEN 1254/haproxy tcp 0 0 192.2.2.2:7804 192.3.3.3:7310 ESTABLISHED 1254/haproxy >>> Spoke tcp 0 0 192.2.2.2:7804 192.4.4.4:14632 ESTABLISHED 1254/haproxy >>> Hub root@regionalmsvm:~#
For small and medium deployments
root@centralmsvm:~#netstat -anp | grep 7804tcp 0 0 0.0.0.0:7804 0.0.0.0:* LISTEN 1254/haproxy tcp 0 0 192.2.2.2:7805 192.3.3.3:7310 ESTABLISHED 1254/haproxy >>> Spoke tcp 0 0 192.2.2.2:7805 192.4.4.4:14632 ESTABLISHED 1254/haproxy >>> Hub root@centralmsvm:~#
If the outbound SSH connection is not established between the spoke and the regional microservices virtual machine,
Ensure that TCP port 7804 is not blocked in the path.
Check the reachability between the spoke and the regional microservices virtual machine. The spoke device (JDM console) must establish an SSH connection with the regional microservices virtual machine.
View the detailed failure logs in the Kibana dashboard http://regional infra IP-Address:5601 or log in to the regional microservices virtual machine and execute the following command.
root@regionalmsvm:~#kubectl get pods –n regional | grep activationcsp.csp-activation-service-1888452022-fv1vt 1/1 Running 1 19h root@regionalmsvm:~# kubectl logs -f csp.csp-activation-service-1888452022-fv1vt –n regional
For NFX250 device, check the recommended vSRX image uploaded in CSO. Check if the vSRX image is uploaded to the CPE or NFX device. If there is any failure, then check the latency, download or upload speed between CPE device and the regional microservices virtual machine.
Log in to the Administration Portal and select Resources > Images.
If the outbound SSH connection is established, then check that the configurations are pushed to the device.
Log in to the Administration Portal. Select Monitor > Jobs. Click the activation failure log and verify the configuration deployment task.
To view the detailed log, use the Kibana dashboard http://central infra iIP-Address:5601 or log in to central microservices virtual machine and execute the following command.
root@centralmsvm:~# kubectl get pods –n central | grep cmscsp.csp-cms-central-2820689874-gvjbh 1/1 Running 1 19h csp.csp-cms-central-core-2224266535-kmplk 1/1 Running 1 19h root@centralmsvm:~# kubectl logs -f csp.csp-cms-central-core-2224266535-kmplk –n central
Verify the configuration in the log and check that the configurations are pushed successfully to the device.
- For further troubleshooting, collect the logs and output results and contact Juniper Networks Technical Support team.
Certificate File Location and Activation Code for an SRX300 Device
Problem
Description: User was unable to perform ZTP on an SRX300 device that acts as both an SD-WAN and a distributed CPE device. Specify the cert file location (to copy the certificate file from a phone-server) and the activation command.
Solution
You can paste the certificate in any directory on the system but you must reference the same location as shown in the following configuration:
system {
host-name spoke0;
root-authentication {
encrypted-password "$ABC123"; ## SECRET-DATA
}
phone-home {
traceoptions {
file phc.log size 10m;
flag all;
}
server https://192.1.1.9;
ca-certificate-file /var/ssl_cert.crt;
}
}
You can use the test phone-home server-authentication-code 123456 command to enter the activation code on an SRX300 device. Alternatively, you can log in to Customer Portal and enter the activation code from the Sites > Sites Management page.