Analyzing Events

Understanding Root Cause Analysis

Root Cause Analysis (RCA) is a fault management feature located in the Live Event Browser that allows you to diagnose trap events and recommend corrective actions. It is accessed by right-clicking an event and selecting Analyze Event from the menu. This feature references a list of rules defined for a device and event type, performs user-defined actions on the device, searches the output of those actions, and highlights if the expected results of the actions are found. The expected results can be used to diagnose the cause of the event and offer suggestions for further action.

Root Cause Analysis helps you analyze the root cause of the events based on user-defined rules in the /u/wandl/db/config/rca-rules file. You can define various commands such as SNMP and CLI to query event specific details or you can define rules to generate an event. After the rca-rules list is defined, these rules will appear in the Root Cause Analysis window. You can select and execute one or more commands in the RCA Rules pane. Selected commands are executed and the results and status are updated. Figure 2 shows the Root Cause Analysis window and the RCA Rules pane.

Each rule in the rca-rules file should be in a single line and in the following format:

<vendor>, <type>, <action>, <expected-result>, <probable-cause>

RCA Rules Field Explanations:

vendor

—

Name of the device vendor. For example, cisco, juniper, huawei

type

—

Name of the SNMP trap. For example, linkUp, linkDown, jnxVpnPwDown

action

—

Command executed through the device CLI, command executed on the application server, SNMP query, or post an event. Conditional actions can be defined too.

expected-result

—

String that will be searched and highlighted from the output of the defined action. For example, line protocol is down. Supports variables such as (ElementName), simple regular expressions, and logical operators && and ||.

probable-cause

—

Message displayed to offer suggestions for action. For example, check cable connection.

RCA Rules Command Results:

<expected-result> found

—

Command status is updated as Matched and the matching text is highlighted in the command result with yellow color.

<expected-result> not found

—

Command status is updated as Not Matched.

<expected-result> is not defined for the rule

—

After successful completion of the command, the status is updated as Executed.

RCA Rules General Keywords:

ElementName

—

Corresponds to the Element Name variable in the Event Browser.

Device

—

Corresponds to the Device ID variable in the Event Browser.

#

—

Use to comment out a line and it will not be parsed in the file.

RCA Rules Action Commands:

@cli:<command>

—

Specifies the action taken is a command on the device CLI. For example, @cli:show interface.

@sh:<command>

—

Specifies the action taken is a command on the application server. For example, @sh:/u/wandl/bin/status_mplsview

@snmp:<OID>

—

Specifies the action taken is a SNMP query on the OID value. For example, @snmp:1.3.6.1.2.1.1.1.0

RCA Rules Conditional Action

Only the action command @cli: or @sh: or @snmp is required in the action field. The labelname:, @match:, and @notmatch: are optional keywords used for conditional action statements. If an action command is not specified, the root cause analysis parser will attempt to identify the type of command although it is recommended to define the action command type.

Format of conditional action field

—

labelname: [@cli: | @sh: | @snmp:] @match:@notmatch

<labelname:>

—

Tags an action with a label used for conditional actions. For example, mylabel:

@match:

—

<labelname:> skips to the line of the labelname if the expected-result matches.

@notmatch:

—

<labelname:> skips to the line of the labelname if the expected-result does not match.

exit

—

Ignores all the remaining rules and exits the root cause analysis.

Analyzing an Event

1. Select Fault > Live Event Browser.

   The Live Event Browser window is displayed. The following shows the Live Event Browser window with Analyze Event selected.

   Figure 1: Selection for Analyze Event

   
2. Select the event, right-click, and select Analyze Event.

   The Root Cause Analysis window is displayed.

3. Select an event in the top pane.
4. In the RCA Rules pane, select the commands to use to analyze the event, and then click Analyze.

   The commands are executed on the node.

5. Expand the command in the RCA Rules pane to display the results.

   Figure 2 shows the Root Cause Analysis window and RCA rules command results.

   Figure 2: Root Cause Analysis Results