Use Correlated Events to Trigger an Event Policy

Understanding Correlated Events

In addition to configuring an event policy to trigger on a single event, you can also configure an event policy that correlates two or more events. If the events occur as specified, the event policy executes the configured actions. For example, you might want to issue certain operational mode commands when a UI_CONFIGURATION_ERROR event is generated within five minutes after a UI_COMMIT_PROGRESS event. As another example, you might want to upload a particular file if a DCD_INTERFACE_DOWN event is generated two times within a 60-second interval.

To correlate events in an event policy, include the following statements at the [edit event-options] hierarchy level:

In the events statement, you can list multiple trigger events. For information about defining events in event policies, see Event Policies and Event Notifications Overview. To correlate those events with other events, you configure the within and/or the attributes-match statements.

The within statement defines the correlating events that must (or must not) occur within a specified time interval before a trigger event. The attributes-match statement enables the system to take an event's attributes into account. The attributes-match statement can correlate an event's attribute with another event's attribute or with a regular expression. The event policy executes the actions configured in the then statement only if the specified conditions are met. The following sections discuss how to use the statements.

Correlating Events By Time Interval
Correlating Events Based on Event Attributes

Correlating Events By Time Interval

You can configure an event policy to execute only if a trigger event occurs within a specified time interval after another event. To correlate events, configure the within seconds events statement. The policy is executed if any of the correlating events defined in the within seconds events statement occur within the configured number of seconds before any of the trigger events defined in the first events statement. The number of seconds can be from 60 through 604,800. The not statement causes the policy to execute only if the correlating events do not occur within the configured time interval before a trigger event.

For example, the device executes the following policy if one of the trigger events, event3, event4, or event5, occurs within 60 seconds after one of the correlating events, event1 or event2.

To configure an event policy to execute only if a trigger event occurs within a specified time interval after multiple events, you must configure separate within statements. Each within statement must use a different time interval. When you configure multiple within statements, every statement must have one matching event that occurs within its respective time frame for the policy to trigger. Thus, the system evaluates the statements using a logical AND.

For example, the device executes the following policy if one of the trigger events, event4, event5, or event6, occurs within 60 seconds after event1 and also occurs within 50 seconds after either event2 or event3.

To configure an event policy to correlate events by time interval:

Configure one or more trigger events.

Configure the correlating events that must or must not occur before a trigger event and within the specified time interval.
- To specify that a correlating event must occur within the specified time interval before a trigger event, omit the not keyword.
- To specify that a correlating event must not occur within the specified time interval before a trigger event, include the not keyword.
- To specify that multiple correlating events must occur within specified time intervals before a trigger event, include multiple within statements that each define different time intervals.
Configure the actions that the event policy executes if the conditions are met.

The within statement also supports executing an event policy when the triggering event occurs a certain number of times within a given time interval. For more information, see Trigger an Event Policy Based on Event Count.

Correlating Events Based on Event Attributes

Many events have one or more attributes, which you can reference in event policies. For example, consider the following system log message for a UI_COMMIT event:

The generic UI_COMMIT event message is: User 'username' requested 'command' operation (comment: message).

It has the following three attributes: username, command, and message.

The attributes-match statement enables you to construct more precise event policy matching by comparing event attributes against desired values using the equals, matches, and starts-with comparisons. The statement correlates events as follows:

event1.attribute-name equals event2.attribute-name—Execute the policy only if the event1 attibute and the event2 attribute have the same value.
event.attribute-name matches regular-expression—Execute the policy only if the event attribute value matches the given regular expression. For more information, see Use Regular Expressions to Refine the Set of Events That Trigger a Policy.
event1.attribute-name starts-with event2.attribute-name—Execute the policy only if the event1 attribute value starts with the value of the event2 attribute.

The events referenced in the attributes-match statements must be either the trigger event or the correlating events included in the event policy’s within statements. You must define one or more within statements when the attributes-match statement:

Includes an equals or starts-with comparison
Includes a matches comparison that includes a clause for an event that is not a trigger event.

An attributes-match statement can include multiple equals, matches, and starts-with comparison statements. The system evaluates multiple comparison statements using a logical AND operator. Thus, to execute the event policy, all of the attributes-match conditions must evaluate to true.

For example, suppose you configure several real-time performance monitoring (RPM) probes, and one of those probes uses the owner name Connectivity and has a test named Management. If you reference the probe's ping_test_failed event in an event policy, you want to distinguish which RPM probe test generated the event. The following event policy matches on the ping_test_failed event's test-owner and test-name attributes so that the event policy triggers only for the correct probe. The event policy executes only if both match conditions are true. For a detailed explanation of the example, see Example: Correlating Events Based on Time Interval and Event Attributes.

Similarly, the following event policy triggers on the nonstandard SYSTEM event, which has a messages attribute. In the attributes-match statement, all of the matches statements must be true to invoke the event policy:

You can use event policy variables within the attributes-match statement to differentiate between a triggering event attribute and a correlating event attribute. Trigger events are those that you configure at the [edit event-options policy policy-name events] hierarchy level. The double dollar sign ($$) notation represents the event that is triggering a policy, and {$$.attribute-name} resolves to the value of the attribute of the triggering event. For correlating events, the single dollar sign with the event name ($event) notation represents the most recent event that matches the event name, and {$event.attribute-name} resolves to the value of the attribute associated with that event.

For example, the following event policy executes the actions under the then statement if four or more commits are performed within a 5-minute period and the username of one or more of the correlating events is the same as the username of the trigger event.

There are many ways to find the attributes that you can reference for a specific event, for example:

Use the System Log Explorer tool.
Use the help syslog event operational mode command in the CLI.
Use context-sensitive help in configuration mode when you configure the attribute.

The System Log Explorer application enables you to search the standard system log messages for a given operating system and release. The message details include the attributes that you can reference for that event. Additionally, the Attributes field lists all attributes for that event.

In the CLI, the help syslog event operational mode command also displays a list of the attributes that you can reference for a given event. The command output shows the event attributes in angle brackets (<>). The following output shows that the ACCT_ACCOUNTING_SMALL_FILE_SIZE event has three attributes: filename, file-size, and record-size.

Note:

You can filter the output of a search by using the pipe (|) symbol. For more information about using the pipe symbol, see the CLI User Guide.

You can also view event attributes by issuing the set attributes-match event? configuration mode command at the [edit event-options policy policy-name] hierarchy level, as shown in the following example:

Note:

In this set command, there is no space between the event name and the question mark (?).

How to Represent Triggering and Correlating Events in an Event Policy

In event script arguments and supported event policy statements such as the execute-commands statement, you can use event policy variables to differentiate between a triggering event and a correlating event. You configure triggering events and correlating events in the following statements at the [edit event-options policy policy-name] hierarchy level:

Triggering event—Configured in the events statement
Correlating event—Configured in the within seconds events statement

You can use event policy variables of the following forms to represent triggering and correlating events:

{$$.attribute-name}—The double dollar sign ($$) notation represents the event that triggers the policy. When combined with an attribute name, the variable resolves to the value of the attribute associated with the triggering event. For example, {$$.interface-name} resolves to the interface name associated with the triggering event.
{$event.attribute-name}—The single dollar sign with the event name ($event) notation represents the most recent event that matches event. When combined with an attribute name, the variable resolves to the value of the attribute associated with that event. For example, when a policy issues the show interfaces {$COSD_CHAS_SCHED_MAP_INVALID.interface-name} command, the {$COSD_CHAS_SCHED_MAP_INVALID.interface-name} variable resolves to the interface name associated with the most recent COSD_CHAS_SCHED_MAP_INVALID event cached by the event process.
{$*.attribute-name}—The dollar sign with the asterisk ($*) notation represents the most recent event that matches any of the correlating events. The variable resolves to the value of the attribute associated with most recent event that matches any of the correlated events specified in the policy configuration.

In event policies, you can reference specific events by using event policy variables. Consider the following event policy:

In the show interfaces {$$.interface-name} command, the value of the interface-name attribute of event e1, e2, or e3 is substituted for the {$$.interface-name} variable.

In the show interfaces {$e4.interface-name} command, the value of the interface-name attribute of the most recent e4 event is substituted for the {$e4.interface-name} variable.

In the show interfaces {$*.interface-name} command, the value of the interface-name attribute of the most recent e4, e5, or e6 event is substituted for the {$*.interface-name} variable. If one of e1, e2, or e3 occurs within 60 seconds after e4, e5, or e6, the value of the interface-name attribute for that correlating event (e4, e5, or e6) is substituted for the {$*.interface-name} variable. If the correlating event does not have an interface-name attribute, the software does not execute the show interfaces {$*.interface-name} command.

If e1 occurs within 60 seconds of both e4 and e5, then the {$*.interface-name} variable uses the value of the interface-name attribute for e4. The policy uses e4 because the event process (eventd) searches for correlating events in sequential order as configured in the within statement. In this case, the order is e4 > e5 > e6.

Example: Correlating Events Based on Time Interval

The following event policy issues a set of commands and uploads the resulting output file to an archive site. The system executes the event policy if one of the trigger events, event3, event4, or event5, occurs within 60 seconds after one of the correlating events, event1 or event2, occurs. The pseudocode for the policy is as follows:

The event policy destination defines two archive sites. The device attempts to transfer to the first archive site in the list, moving to the next site only if the transfer fails. The event policy configuration is:

Example: Correlating Events Based on Event Attributes

In the following event policy, the two events are correlated if their event attribute values match. Matching on the attributes of both events ensures that the two events are related. In this case, the interface addresses must match and the physical interface (ifd) names must match.

The RPD_KRT_IFDCHANGE error occurs when the routing protocol process (rpd) sends a request to the kernel to change the state of an interface and the request fails. The RPD_RDISC_NOMULTI error occurs when an interface is configured for router discovery but the interface does not support IP multicast operations as required.

Example: Correlating Events Based on Time Interval and Event Attributes

In the following example, an organization wishes to record all failed configurations that resulted in a loss of connectivity to the management network. The organization uses a real-time performance monitoring (RPM) test setup that verifies the reachability of the management network every minute. The operators always commit changes using the commit confirmed CLI command. The commit confirmed command enables the configuration to automatically roll back if the configuration changes impact connectivity.

The event policy executes an event script, save-rollback.slax, to copy the failed configuration overridden by the automatic rollback to a specific location. The event script copies the /config/juniper.conf.1.gz rollback file to the /var/tmp directory for storage and later examination.

The event policy trigger event is the UI_COMMIT_NOT_CONFIRMED event. This event occurs when the device automatically rolls back the commit confirmed. When the system triggers a commit confirmed rollback, the UI_COMMIT_NOT_CONFIRMED event occurs twice. The first event message indicates the rollback is in progress, and the second event message indicates the rollback is complete. Thus, the event policy must correlate the two UI_COMMIT_NOT_CONFIRMED events so that the system doesn't trigger the event policy twice. In this example, the second UI_COMMIT_NOT_CONFIRMED event must occur within 50 seconds after the first event.

To determine if the configuration resulted in a loss of connectivity to the management network, the event policy also correlates the UI_COMMIT_NOT_CONFIRMED event with the PING_TEST_FAILED event from the RPM probe. A PING_TEST_FAILED event occurs when the RPM test to the management network fails. In this example, the UI_COMMIT_NOT_CONFIRMED event must occur within 60 seconds of the PING_TEST_FAILED event. To specify the RPM probe test that generated the event, the event policy includes the attributes-match statement and only triggers when the event's test-owner attribute matches Connectivity and the event's test-name attribute matches Management.

The event policy configures separate within statements for the PING_TEST_FAILED event and the first UI_COMMIT_NOT_CONFIRMED event. By using multiple within statements, both conditions must occur for the event policy to trigger. Each within statement requires a different time interval. Similarly, both attributes-match conditions must be true for the policy to trigger.

The event policy configuration is as follows:

The save-rollback.slax event script is as follows:

When the device running Junos OS triggers the event policy, the device saves the rollback configuration file in the /var/tmp directory with a new filename that includes the timestamp.

ON THIS PAGE