Collect Crash Data for a Routing Engine Kernel

Purpose

When a Routing Engine kernel crashes, the Routing Engine automatically reboots. By default, the Juniper Networks router does not attempt to dump a core if the Routing Engine kernel crashes. As a result, there is no crash data on the router to help investigate the crash. In addition, the system log messages are similar to those generated when the router is powered down and restarted, so you cannot tell if the Routing Engine restart was caused by a kernel crash or a normal power restart.

To collect crash data for a Routing Engine kernel crash, follow these steps:

Check the Routing Engine Core Files

Purpose

If you observe disruption to the Routing Engine kernel, check the /var/crash directory for any core files created around the time of the crash.

Action

To check the /var/crash directory, use the following Junos OS command-line interface (CLI) operational mode command:

user@host> file list detail /var/crash

Sample Output

user@host> file list detail /var/crash

total 1577912
drwxr-x---   2 root  wheel        512 Sep  9 11:59 ./
drwxr-xr-x  22 root  wheel        512 Oct 29  2001 ../
-rw-r--r--   1 root  wheel          2 Jul 20 01:11 bounds
-rw-r--r--   1 root  wheel    2166913 Jul 20 01:11 kernel.0
-rw-r--r--   1 root  wheel          5 Feb 15  2002 minfree
-rw-------   1 root  wheel  805306368 Jul 20 01:11 vmcore.0

Meaning

The sample output lists the contents of the /var/crash/ directory. Check the date and timestamp for any kernel core files created around the time of the crash. In the example above, two core files are listed: kernel.0 and vmcore.0.

List the Core Files

Purpose

To list the core files, follow these steps:

Action

Exit from the CLI environment and create a UNIX-level shell by entering the start shell command:
user@host> start shell
Type su and the root password when prompted. You are now in the shell and the prompt is % instead of >, for example:
% su Password: ****
Change the directory to /var/crash and type ls -l,for example:
root@host% cd /var/crash root@host% ls -l
Look for any core files created around the time of the crash.

Sample Output

user@host> start shell

%  su
Password:  ****
root@host%  cd /var/crash
root@host%  ls -l
total 1577908
-rw-r--r--  1 root  wheel          2 Jul 20 01:11 bounds
-rw-r--r--  1 root  wheel    2166913 Jul 20 01:11 kernel.0
-rw-r--r--  1 root  wheel          5 Feb 15  2002 minfree
-rw-------  1 root  wheel  805306368 Jul 20 01:11 vmcore.0

Meaning

The sample output lists the contents of the /var/crash directory and shows the current core files kernel.0 and vm.core.0.

Compress the vmcore File

Purpose

The gzip compression utility is used to compress files. Compress the vmcore file if it is larger than 50 MB. Files created using the gzip command end with the file extension .gz.

Note: Use lowercase for the gzip command when you are in the shell.

Action

To compress the vmcore file with gzip, use the following command from the shell:

root@host% gzip vmcore .number

To unzip the vmcore file with gzip, use the following command from the shell:

root@host% gzip -d vmcore .number.gz

Meaning

The contents of the vmcore file are compressed into a single compressed file named vmcore.number.gz. The gzip command preserves the mode, ownership, and timestamps of files when compressing or decompressing them.

Log Software Version Information

Purpose

To log the Junos OS version information.

Action

To log the Junos OS version information, use the following Junos OS CLI operational mode command:

user@host> show version

Sample Output

user@host> show version

Hostname: host
Model: m10
JUNOS Base OS boot [5.0R5]
JUNOS Base OS Software Suite [5.0R5]
JUNOS Kernel Software Suite [5.0R5]
JUNOS Routing Software Suite [5.0R5]
JUNOS Packet Forwarding Engine Support [5.0R5]
JUNOS Crypto Software Suite [5.0R5]
JUNOS Online Documentation [5.0R5]
KERNEL 5.0R5 #0 built by builder on 2002-03-02 05:10:28 UTC
MGD release 5.0R5 built by builder on 2002-03-02 04:45:32 UTC
CLI release 5.0R5 built by builder on 2002-03-02 04:44:22 UTC
CHASSISD release 5.0R5 built by builder on 2002-03-02 04:43:37 UTC
DCD release 5.0R5 built by builder on 2002-03-02 04:42:47 UTC
RPD release 5.0R5 built by builder on 2002-03-02 04:46:17 UTC
SNMPD release 5.0R5 built by builder on 2002-03-02 04:52:26 UTC
MIB2D release 5.0R5 built by builder on 2002-03-02 04:45:37 UTC
APSD release 5.0R5 built by builder on 2002-03-02 04:43:31 UTC
VRRPD release 5.0R5 built by builder on 2002-03-02 04:52:34 UTC
ALARMD release 5.0R5 built by builder on 2002-03-02 04:43:24 UTC
PFED release 5.0R5 built by builder on 2002-03-02 04:46:06 UTC
CRAFTD release 5.0R5 built by builder on 2002-03-02 04:44:30 UTC
SAMPLED release 5.0R5 built by builder on 2002-03-02 04:52:20 UTC
ILMID release 5.0R5 built by builder on 2002-03-02 04:45:21 UTC
BPRELAYD release 5.0R5 built by builder on 2002-03-02 04:42:41 UTC
RMOPD release 5.0R5 built by builder on 2002-03-02 04:46:11 UTC
jkernel-dd release 5.0R5 built by builder on 2002-03-02 04:41:07 UTC
jroute-dd release 5.0R5 built by builder on 2002-03-02 04:41:21 UTC
jdocs-dd release 5.0R5 built by builder on 2002-03-02 04:39:11 UTC

Meaning

The sample output shows the hostname, router model, and the different Junos OS packages, processes, and documents.

Open a Case with JTAC

Problem

To open a case with JTAC, follow these steps:

Solution

Send an e-mail to support@juniper.net, and include the information from the show version command.
At the support engineer’s request, ftp the vmcore.number.gz file to a case-number directory at ftp.juniper.net. To ftp the core file to a directory, follow these steps:
1. At the shell prompt, enter ftp ftp.juniper.net.
2. At the name prompt, enter anonymous.
3. At the password prompt, enter your e-mail address as the password.
4. At the ftp prompt, enter the cd pub/ incoming string.
5. Enter the mkdir case-number command, where the case-number is the value of the case you opened with JTAC, for example, 1999-1231-9999. If a directory has already been created, continue with the next step.
6. Enter the cd case-number command.
7. Enter the binary command so that the file transfer is in binary and not ASCII.
8. Enter the put vmcore.0.gz command.

Meaning

The following output is an example of copying a core file from the shell to an ftp directory at ftp.juniper.net:

Sample Output

root@host% ftp ftp.juniper.net
Connected to colo-ftp.juniper.net.
220 colo-ftp.juniper.net FTP server (Version 6.00LS) ready.
Name (ftp.juniper.net: root): anonymous
331 Guest login ok, send your email address as password.
Password: ****
230 Guest login ok, access restrictions apply.
ftp> cd pub/ incoming
250 CWD command successful
ftp> mkdir 1999-1231-9999
257 MKD command successful.
ftp> cd 1999-1231-9999
250 CWD command successful.
ftp> bin
200 Type set to I.
ftp> put vmcore.0.gz

Meaning

The sample output shows that there is a connection to ftp.juniper.net, that the login name and password were entered, and that the core file was successfully copied from the shell to an ftp directory at ftp.juniper.net.