Collect Crash Data for the Packet Forwarding Engine Microkernel
Purpose
Each of the following Packet Forwarding Engine components of a Juniper Networks router runs a microkernel:
- Flexible PIC Concentrator (FPC) on M-series platforms except for the M5 and M10 Internet routers
- Gibson Flexible PIC Concentrator (GFPC) on T640 and T320 Internet routing nodes
- Switched Printed Mezzanine Board (SPMB) on T640 and T320 Internet routing nodes
- Forwarding Engine Board (FEB) on M5 and M10 Internet routers
- System Switching Board (SSB) on an M20 Internet router
- System Control Board (SCB) on an M40 Internet router
- Switching and Forwarding Module (SFM) on M160 and M40e Internet routers
When a crash occurs, crash stack traceback and registration information is placed into nonvolatile random access memory (NVRAM) on the different components. Table 1 shows where the NVRAM is located for the components for each router.
Table 1: NVRAM Location on the Microkernel of the Packet Forwarding Engine Components
Router Type | NVRAM Location |
|---|---|
M5 and M10 | FEB |
M20 | SSB and crash stack traceback and register information for the FPC |
M40 | SCB and crash stack traceback and register information for the FPC |
M40e | FPC |
M160 | FPC |
T320 | GFPC |
T640 | GFPC |
To collect crash data for the Packet Forwarding Engine microkernel, follow these steps:
- Display the Crash Stack Traceback and Registration Information
- Clear the NVRAM Contents
- Check Packet Forwarding Engine Microkernel Core Files
- List the Core Files Generated by the Crash
- Compress the Core Files
- Log Software Version Information
- Open a Case with JTAC
Display the Crash Stack Traceback and Registration Information
Purpose
To display the crash stack traceback and registration information.
Action
To display the crash stack traceback and registration information, follow these steps:
- Exit from the CLI environment and create a UNIX-level
shell by entering the start shell command:user@host> start shell
- Type su and the root password when prompted. You are now in
the shell and prompt is % instead of >, for example:% su Password: ****
- Establish a vty session to the appropriate
component. Use the vty command followed by the executable
name for the component; for example, scb, ssb0, ssb1, fpc0, or fpc1:root@host% vty sfm0
Note: For the M40e and M160 routers, you can also create a cty session to the components if the components are not online.
- Type the show nvram command to view the NVRAM information.
- Type the show syslog messages command to view the system log messages.
Sample Output 1
user@host> start shell
% su Password: **** root@host% vty sfm0 SFM platform (266Mhz PPC 603e processor, 64Mb memory, 512Kb flash) SFM3(host vty)# show nvram System NVRAM : 4080 available bytes, 4080 used, 0 free Contents: mpc106 machine check caused by error on the PCI Bus mpc106 error detect register 1: 0x08, 2: 0x00 mpc106 error ack count = 0 mpc106 error address: 0x0a000000 mpc106 PCI bus error status register: 0x02 mpc106 was the PCI master C/BE bits: I/O read [0b0010] mpc106 error detection reg1: PCI cycle mpc106 PCI status reg: parity error System Exception: Vector/Code 0x00700, Signal 4 Event occurred at: Oct 26 13:32:40.952 Juniper Embedded Microkernel Version 4.2R1 Built by tlim on 2000-09-23 06:11:28 UTC Copyright (C) 1998-2000, Juniper Networks, Inc. All rights reserved. Reason string: "Program Check" Context: Thread (PFE Manager) Registers: R00: 0x06f5f81c R01: 0x06f5f9cc R02: 0x00003344 R03: 0x00000000 R04: 0x00008000 R05: 0x00000000 R06: 0x0010052c R07: 0x06f637e4 R08: 0x06f5f81c R09: 0x00169810 R10: 0x000000e8 R11: 0x00000001 R12: 0x00046cdf R13: 0xffffffff R14: 0xffffffff R15: 0xffffffff R16: 0xffffffff R17: 0xffffffff R18: 0xffffffff R19: 0xffffffff R20: 0xffffffff R21: 0xffffffff R22: 0xffffffff R23: 0xffffffff R24: 0x00000003 R25: 0x00000000 R26: 0x00000001 R27: 0x0000fc78 R28: 0x00150000 R29: 0x0016c4b0 R30: 0x06f5eb7c R31: 0x97cb1d36 MSR: 0x0008b030 CTR: 0x000ac008 Link:0x06f5f81c SP: 0x06f5f9cc CCR: 0x22200024 XER: 0x20000000 PC: 0x06f5f81c DSISR: 0x00000000 DAR: 0xffffffff K_MSR: 0x00001030 Stack Traceback : Frame 01: sp = 0x06f5f9cc, pc = 0x06f5f81c Frame 02: sp = 0x06f5f9e4, pc = 0x000c7e28 Frame 03: sp = 0x06f5fa04, pc = 0x00026620 ROM NVRAM: 0 available bytes, 0 used, 0 free SFM3(host vty)# show syslog messages Oct 26 12:02:05 router tnp_sfm_2 PFEMAN: sent Resync request to Master Oct 26 12:02:07 router tnp_sfm_3 CM(3): Slot 1: On-line Oct 26 12:02:07 router tnp_sfm_3 CM(3): Slot 2: On-line Oct 26 12:02:07 router tnp_sfm_3 CM(3): Slot 6: On-line Oct 26 12:02:07 router tnp_sfm_3 PFEMAN: sent Resync request to Master Oct 26 12:05:58 router tnp_sfm_3 mpc106 machine check caused by error on the PCI Bu s Oct 26 12:05:58 router tnp_sfm_3 mpc106 error detect register 1: 0x08, 2: 0x00 Oct 26 12:05:58 router tnp_sfm_3 mpc106 error ack count = 0 Oct 26 12:05:58 router tnp_sfm_3 mpc106 error address: 0x0a000000 Oct 26 12:05:58 router tnp_sfm_3 mpc106 PCI bus error status register: 0x02 Oct 26 12:05:58 router tnp_sfm_3 mpc106 was the PCI master Oct 26 12:05:58 router tnp_sfm_3 C/BE bits: I/O read [0b0010] Oct 26 12:05:58 router tnp_sfm_3 mpc106 error detection reg1: PCI cycle Oct 26 12:05:58 router tnp_sfm_3 mpc106 PCI status reg: parity error Oct 26 12:05:58 router tnp_sfm_3 ^B Oct 26 12:05:58 router tnp_sfm_3 last message repeated 7 times Oct 26 12:05:58 router tnp_sfm_3 Registers: Oct 26 12:05:58 router tnp_sfm_3 R00: 0x06f5f81c R01: 0x06f5f9cc R02: 0x00003344 R0 3: 0x00000000 Oct 26 12:05:58 router tnp_sfm_3 R04: 0x00008000 R05: 0x00000000 R06: 0x0010052c R0 7: 0x06f637e4 Oct 26 12:05:58 router tnp_sfm_3 R08: 0x06f5f81c R09: 0x00169810 R10: 0x000003b4 R1 1: 0x00000001 Oct 26 12:05:58 router tnp_sfm_3 R12: 0x00017b97 R13: 0xffffffff R14: 0xffffffff R1 5: 0xffffffff Oct 26 12:05:58 router tnp_sfm_3 R16: 0xffffffff R17: 0xffffffff R18: 0xffffffff R1 9: 0xffffffff Oct 26 12:05:58 router tnp_sfm_3 R20: 0xffffffff R21: 0xffffffff R22: 0xffffffff R2 3: 0xffffffff Oct 26 12:05:58 router tnp_sfm_3 R24: 0x00000003 R25: 0x00000000 R26: 0x00000001 R2 7: 0x0000fc78 Oct 26 12:05:58 router tnp_sfm_3 R28: 0x00150000 R29: 0x0016c4b0 R30: 0x06f5eb7c R3 1: 0x97c9c35e Oct 26 12:05:58 router tnp_sfm_3 MSR: 0x0008b030 CTR: 0x000ac008 Link:0x06f5f81c SP : 0x06f5f9cc Oct 26 12:05:58 router tnp_sfm_3 CCR: 0x22200024 XER: 0x20000000 PC: 0x06f5f81c Oct 26 12:05:58 router tnp_sfm_3 DSISR: 0x00000000 DAR: 0xffffffff K_MSR: 0x0000103 0
Sample Output
root@host% vty fpc1 FPC160 platform (PPC 603e processor, 32Mb memory, 512Kb flash) FPC1(host vty)# show nvram System NVRAM : 4080 available bytes, 4080 used, 0 free Contents: 0000000 R06: 0x0000005c R07: 0x850400d0 R08: 0x00000000 R09: 0x00000020 R10: 0x00000000 R11: 0x00000129 R12: 0x00000000 R13: 0x00000000 R14: 0x4005009a R15: 0x20000260 R16: 0xc8828784 R17: 0x84212800 R18: 0xc0004c61 R19: 0x80005900 R20: 0x80206000 R21: 0x84000304 R22: 0xd0410180 R23: 0x8c2005ac R24: 0x00000003 R25: 0x00000000 R26: 0x00000001 R27: 0x0000fc48 R28: 0x001d0000 R29: 0x00000001 R30: 0x00136bb8 R31: 0x00000000 MSR: 0x0000b030 CTR: 0x001331e0 Link:0x000308c8 SP: 0x01baba34 CCR: 0x42200020 XER: 0x00000000 PC: 0x000308cc DSISR: 0x00000000 DAR: 0xffffffff K_MSR: 0x00001030 Stack Traceback: Frame 01: sp = 0x01baba34, pc = 0x000308c8 Frame 02: sp = 0x01babac4, pc = 0x0002647c Frame 03: sp = 0x01babad4, pc = 0x00026590 Frame 04: sp = 0x01babadc, pc = 0x00106fcc Frame 05: sp = 0x01babafc, pc = 0x00026620 ROM NVRAM: 0 available bytes, 0 used, 0 free FPC1(host vty)# show syslog messages [0+00:00:00.780 LOG: Info] Version 4.0R5 by tlim on 2000-08-10 04:45:54 UTC [0+00:00:00.780 LOG: Info] On-board NVRAM contains diagnostic information. [0+00:00:03.175 LOG: Info] PFEMAN: Established connection to Master [Jan 30 21:53:05.804 LOG: Info] SNTPD: Initial time of day set.
Meaning
Sample output 1 and 2 show the stack trace from the microkernel crash. Save the output from the show nvram and show syslog commands so that you can send them to JTAC when you open a case.
Clear the NVRAM Contents
Purpose
Currently the storage area for the logs on the NVRAM is limited to 4 KB. You need to delete old NVRAM logs to make room for new ones.
Action
To clear the content of the NVRAM after you have captured the necessary information, follow these steps:
- Exit from the CLI environment and create a UNIX-level
shell by entering the start shell command:user@host> start shell
- Type su and the root password when prompted. You are now in
the shell and the prompt is % instead of >, for
example: % su Password: ****
- Establish a vty session to the appropriate
component. Use the vty command followed by the abbreviation for the component,
for example:root@host% vty sfm0 SFM3(host vty)# FPC1(host vty)#
- Type the clear nvram command, for example:SFM3(host vty)# clear nvram FPC1(host vty)# clear nvram
Check Packet Forwarding Engine Microkernel Core Files
Purpose
If you observe disruption to the Packet Forwarding Engine microkernel, check the /var/crash directory for any core files created around the time of the crash.
Action
To check the /var/crash directory, use the following Junos OS CLI operational mode command:
Sample Output
user@host> file list detail /var/crash var/crash: total 456630 -rw-r--r-- 1 root wheel 6814720 Dec 18 08:03 core-FPC4.100111808032 -rw-r--r-- 1 root wheel 65613824 Dec 10 04:58 core-SCB.100111004570 -rw-r--r-- 1 root wheel 65613824 Dec 19 00:23 core-SCB.100111900221 -rw-r--r-- 1 root wheel 65545216 Feb 9 20:46 core-SCB.101010920452
Meaning
The sample output lists the contents of the /var/crash/ directory. Check the date and timestamp for any core files created around the time of the crash. In the example above, four core files are listed.
List the Core Files Generated by the Crash
Purpose
To list the core files generated by the crash.
Action
To list the core files, follow these steps:
- Exit from the CLI environment and create a UNIX-level
shell by entering the start shell command:user@host> start shell
- Type su and the root password when prompted. You are now in
the shell and the prompt is % instead of >, for
example:% su Password: ****
- Change the directory to /var/crash and type ls -l, for example:root@host% cd /var/crash root@host% ls -l
- Look for any core files created around the time of the crash.
Sample Output
user@host> start shell
% su Password: **** root@host% cd /var/crash root@host% ls -l total 456630 -rw-r--r-- 1 root wheel 6814720 Dec 18 08:03 core-FPC4.100111808032 -rw-r--r-- 1 root wheel 65613824 Dec 10 04:58 core-SCB.100111004570 -rw-r--r-- 1 root wheel 65613824 Dec 19 00:23 core-SCB.100111900221 -rw-r--r-- 1 root wheel 65545216 Feb 9 20:46 core-SCB.101010920452
Meaning
The sample output shows the current core files for the different components on the router; for example, core-FPC4.100111808032 and core-SCB.100111004570.
Compress the Core Files
Purpose
gzip is a compression utility used to compress the core files. Files created using the gzip command end with the file extension .gz. Compress the core files if they are larger than 50 MB.
Action
To compress the core files with gzip, use the following command from the shell:
Sample Output
root@host% gzip core-SCB.101010920452
Meaning
The contents of the core file are compressed into a single compressed file named core-SCB.10101092045.gz. The gzip command preserves the mode, ownership, and timestamps of files when compressing or decompressing them.
Log Software Version Information
Purpose
To log the Junos OS version information.
Action
To log the Junos OS version information, use the following Junos OS CLI operational mode command:
Sample Output
user@host> show version
Hostname: host Model: m10 JUNOS Base OS boot [5.0R5] JUNOS Base OS Software Suite [5.0R5] JUNOS Kernel Software Suite [5.0R5] JUNOS Routing Software Suite [5.0R5] JUNOS Packet Forwarding Engine Support [5.0R5] JUNOS Crypto Software Suite [5.0R5] JUNOS Online Documentation [5.0R5] KERNEL 5.0R5 #0 built by builder on 2002-03-02 05:10:28 UTC MGD release 5.0R5 built by builder on 2002-03-02 04:45:32 UTC CLI release 5.0R5 built by builder on 2002-03-02 04:44:22 UTC CHASSISD release 5.0R5 built by builder on 2002-03-02 04:43:37 UTC DCD release 5.0R5 built by builder on 2002-03-02 04:42:47 UTC RPD release 5.0R5 built by builder on 2002-03-02 04:46:17 UTC SNMPD release 5.0R5 built by builder on 2002-03-02 04:52:26 UTC MIB2D release 5.0R5 built by builder on 2002-03-02 04:45:37 UTC APSD release 5.0R5 built by builder on 2002-03-02 04:43:31 UTC VRRPD release 5.0R5 built by builder on 2002-03-02 04:52:34 UTC ALARMD release 5.0R5 built by builder on 2002-03-02 04:43:24 UTC PFED release 5.0R5 built by builder on 2002-03-02 04:46:06 UTC CRAFTD release 5.0R5 built by builder on 2002-03-02 04:44:30 UTC SAMPLED release 5.0R5 built by builder on 2002-03-02 04:52:20 UTC ILMID release 5.0R5 built by builder on 2002-03-02 04:45:21 UTC BPRELAYD release 5.0R5 built by builder on 2002-03-02 04:42:41 UTC RMOPD release 5.0R5 built by builder on 2002-03-02 04:46:11 UTC jkernel-dd release 5.0R5 built by builder on 2002-03-02 04:41:07 UTC jroute-dd release 5.0R5 built by builder on 2002-03-02 04:41:21 UTC jdocs-dd release 5.0R5 built by builder on 2002-03-02 04:39:11 UTC
Meaning
The sample output shows the hostname, router model, and the different Junos OS packages, processes, and documents.
Open a Case with JTAC
Problem
To open a case with JTAC, follow these steps:
Solution
- Send an e-mail to support@juniper.net, and include the information from the show version command.
- At the support engineer’s request, ftp the vmcore.number.gz file to a case-number directory at ftp.juniper.net. To ftp the core file to a directory, follow
these steps:
- At the shell prompt, enter ftp ftp.juniper.net.
- At the name prompt, enter anonymous.
- At the password prompt, enter your e-mail address as the password.
- At the ftp prompt, enter the cd pub/ incoming string.
- Enter the mkdir case-number command, where the case-number is the value of the case you opened with JTAC, for example, 1999-1231-9999. If a directory has already been created, continue with the next step.
- Enter the cd case-number command.
- Enter the binary command so that the file transfer is in binary and not ASCII.
- Enter the put vmcore.0.gz command.
Meaning
The following output is an example of copying a core file from the shell to an ftp directory at ftp.juniper.net:
Sample Output
root@host% ftp ftp.juniper.net Connected to colo-ftp.juniper.net. 220 colo-ftp.juniper.net FTP server (Version 6.00LS) ready. Name (ftp.juniper.net: root): anonymous 331 Guest login ok, send your email address as password. Password: **** 230 Guest login ok, access restrictions apply. ftp> cd pub/ incoming 250 CWD command successful ftp> mkdir 1999-1231-9999 257 MKD command successful. ftp> cd 1999-1231-9999 250 CWD command successful. ftp> bin 200 Type set to I. ftp> put vmcore.0.gz
Meaning
The sample output shows that there is a connection to ftp.juniper.net, that the login name and password were entered, and that the core file was successfully copied from the shell to an ftp directory at ftp.juniper.net.
