Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Troubleshoot Deployment

Learn how to troubleshoot your Cloud-Native Router deployment.

Note:

We use the JSON processor jq to format the output of some commands. If jq is not installed on your OS by default, install it using the appropriate package manager for your OS. For example:

Common Problems

Table 1 lists some common deployment problems and remedies.

Table 1: Common Problems
Potential issue What to check Related Commands
Image not found Check if the images are uploaded to the local docker using the command docker images. If not, then the registry configured in values.yaml should be accessible. Ensure image tags are correct. 
kubectl -n jcnr describe pod <crpd-pod-name>
Initialization errors Check if jcnr-secrets is loaded and has a valid license key 
[root@jcnr-01]# kubectl get secrets -n jcnr
NAME                  TYPE                                  DATA   AGE
crpd-token-zp8kc      kubernetes.io/service-account-token   3      29d
default-token-zn6p9   kubernetes.io/service-account-token   3      29d
jcnr-secrets          Opaque                                2      29d
Confirm that root password and license key are present in /var/run/jcnr/juniper.conf
cRPD Pod in CrashLoopBackOff state
  • Check if startup/liveness probe is failing or vrouter pod not running
  • rpd-vrouter-agent gRPC connection not UP
  • Composed configuration is invalid or config template is invalid
 
  • kubectl get pods -A
    kubectl -n jcnr describe pod <crpd-pod-name>
    tail –f /var/log/jcnr/jcnr-cni.log​
    tail –f /var/log/jcnr/jcnr_notifications.json

  • See Access cRPD CLI to enter the cRPD CLI and run the following command:

    show krt state channel vrouter
  • cat /var/run/jcnr/juniper.conf

vRouter Pod in CrashLoopBackOff state

 Check the contail-k8s-deployer logs for errors See Check Deployer Logs.

Check Deployer Logs

The deployer logs should be one of the first places you look when you run into installation problems.
To check the deployer logs:
  1. List the deployer pod.

    Sample output:

  2. View the deployer logs.

Verify vRouter and cRPD Health

  1. Check the vRouter daemonset.
    1. List the daemonsets.

      Sample output:

    2. Get vRouter daemonset details.

      Sample output:

  2. Check the cRPD stateful set.
    1. List the stateful sets.

      Sample output:

    2. Get the cRPD stateful set details.

      Sample output:

Verify cRPD Configuration

The Cloud-Native Router deployment process creates a cRPD configuration file from the parameters in values.yaml for L2 mode and custom configuration via node annotations in L3 mode. This cRPD configuration file is at /var/run/jcnr/juniper.conf on any node running JCNR.

The cRPD configuration can be customized using node annotations. The cRPD pod will stay in pending state if the applied configuration is invalid.

The rendered custom configuration is in /etc/crpd/juniper.conf.master.

In an AWS EKS deployment you can see the rendered custom configuration by accessing the cRPD CLI and navigating to the /config directory.

View Log Files

You can find the Cloud-Native Router log files in the default log_path directory (/var/log/jcnr/) on any node running JCNR. You can change this location by changing the value of the log_path or syslog_notifications parameters in the values.yaml file prior to deployment.

Here's an example of some of the log files that Cloud-Native Router keeps.

Note:

If your deployment fails, check the applier logs in applier/applier.log for more information.

Contrail Tools

Contrail Tools is an optional package that contains troubleshooting utilities that you can use to debug your installation.

To install Contrail Tools, set the contrail-tools install flag to true in the Cloud-Native Router installation Helm chart and then install Cloud-Native Router as you normally do.

Collect Debug and Support Information

To facilitate the gathering of debug information, we provide a single utility that you can run to take a snapshot of the state and condition of all Cloud-Native Router components. This snapshot contains logs, traces, and other useful data that you can analyze yourself or provide to Juniper Networks Support. Collected information is placed in a tarball in /var/log/jcnr and includes:

  • CPU and memory usage

  • node and pod states and events

  • cRPD data and traces

  • vRouter data and traces

  • log files

  • optional crash (core dump) files

  • optional debug information

  • plus much more

Note: This utility is only supported on DPDK installations. It is not supported on eBPF installations.
  1. Ensure contrail-tools is installed in your cluster.

    If you do not have contrail-tools installed in your cluster, then you will need to reinstall Cloud-Native Router with contrail-tools before you can run this collection utility.

  2. Exec into the contrail-tools container on the node where you want to collect the information.
  3. Run the collection utility.

    • To collect essential data: collect_jcnr_support_info

      This command may take several minutes to run.

    • To collect essential and crash data: collect_jcnr_support_info -crash

      This command may take many minutes to run.

    • To collect essential and additional debug data: collect_jcnr_support_info -debug

      This command may affect performance and may take many minutes to run.

    • To collect all data: collect_jcnr_support_info -all

      This command may affect performance and may take many minutes to run.

    You can find the collected data, gzipped and tarred, in the contrail-tools container (and on the underlying host) at:

  4. Repeat on all contrail-tools containers where you want to collect the information.

Collect Interface Throughput and Performance Statistics

Run the dpdkvifstats utility to show interface throughput and performance statistics for the DPDK interfaces.

Available statistics include:

  • transmit and receive packets per second

  • transmit and receive bits per second

  • transmit and receive error counts

By default, this utility refreshes the displayed counts every 3 seconds. As an option, you can specify the duration between refreshes.

Note: This utility is only supported on DPDK installations. It is not supported on eBPF installations.
  1. Ensure contrail-tools is installed in your cluster.

    If you do not have contrail-tools installed, then you will need to reinstall Cloud-Native Router with contrail-tools before you can run this utility.

  2. Exec into the contrail-tools container on the node where you want to show the throughput and performance statistics.
  3. Show the DPDK interface statistics.

    • To show statistics for all DPDK interfaces: dpdkvifstats -all

    • To show statistics for a specific DPDK interface: dpdkvifstats -vif <interface>

      where <interface> is the interface number of the interface you want to show.

    Additionally you can qualify the above with either or both of the -time and -cpu options:

    • To specify a statistics refresh interval, add the -time <seconds> option, where <seconds> is the number of seconds between refreshes.

    • To specify the number of CPU cores you want to show, add the -cpu <number-of-cpus> option, where <number-of-cpus> is a number not exceeding the number of CPUs you've assigned to the vRouter in the Helm chart. The number of CPUs that you assign to the vRouter is either guaranteedVrouterCpus or the number of cores specified in cpu_core_mask.

      Note:

      The CPU number displayed in the output is the DPDK lcore value, which does not necessarily correspond numerically to the actual CPU assigned to run that thread. For the vRouter, lcore numbering for forwarding threads starts at 15.

      The output displays statistics for lcores starting at 15 and continues sequentially for the number of cores specified in <number-of-cpus>.
  4. Repeat on all contrail-tools containers where you want to show these DPDK throughput and performance statistics.