Syslog Ingest

Overview

Starting with Release 2.1.0, HealthBot supports syslog natively as another ingest method, using a data model that aligns with other HealthBot ingest mechanisms to provide all the same feature richness.

• Pattern - A pattern identifies some syslog event; you create a pattern for each event you want to monitor. You can configure patterns for both structured and unstructured events.

• Pattern set - With the patterns configured, you then group them into a pattern set, which you then reference when defining the syslog sensor settings within a rule.

To see how patterns and pattern sets are used, see Example: Creating a Rule Using Syslog Ingest.

System-Generated Fields

Some fields are common in syslog messages. HealthBot extracts these fields and includes them automatically in the raw table, enabling you to make use of them directly when creating a rule, and avoiding the need to configure patterns.

To illustrate use of these values, consider the following example syslog messages:

Structured - <30>1 2019-11-22T03:17:53.605-08:00 R1 mib2d 28633 SNMP_TRAP_LINK_DOWN [junos@2636.1.1.1.2.29 snmp-interface-index="545" admin-status="up(1)" operational-status="down(2)" interface-name="ge-1/0/0.16"] ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16

Equivalent unstructured - <30>Nov 22 03:17:53 R1 mib2d[28633]: SNMP_TRAP_LINK_DOWN: ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16

• "__log_priority__" - Priority of syslog message

  • In the examples, <30> denotes the priority

• “__log_timestamp__" - Timestamp in epcoh in the syslog message

  • In the structured example, 2019-11-22T03:17:53.605-08:00 is converted to epoch with -08:00 indicating the time zone

  • In the unstructured example, the time zone from the configuration will be used to calculate epoch

• "__log_host__" - Host name in the syslog message

  • In the examples, R1 denotes the host name

• "__log_application_name__” - Application name in the syslog message

  • In the examples, mib2d is the application name

• "__log_application_process_id__” - Application process ID in the syslog message

  • In the examples, 28633 is the ID

• "__log_message_payload__" - Payload in the message

  • Structured example - “SNMP_TRAP_LINK_DOWN [junos@2636.1.1.1.2.29 snmp-interface-index="545" admin-status="up(1)" operational-status="down(2)" interface-name="ge-1/0/0.16"] ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16”

  • Unstructured example - “SNMP_TRAP_LINK_DOWN: ifIndex 545, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0.16”

• "Event-id" - Denotes the event ID configured in the pattern

  • In the examples, SNMP_TRAP_LINK_DOWN is the event ID

Usage Notes

• If you add a device in HealthBot using its IP address (and the address is not the actual host name of the device), you must also add the device’s host name in the Syslog Hostnames field.

• Multiple device groups can listen for syslog messages using the same port.

• A configured time zone is not considered when processing structured messages, as the messages themselves include the time zone.

• Daylight savings time is not currently supported.

Optional Configuration Elements

Configure syslog ports

By default, HealthBot listens for syslog messages from all device groups on UDP port 514. You can change the system-level syslog port, and also configure one or more ports per device group. The more specific device group setting takes precedence over the system level setting.

1. Click Settings > Ingest Settings in the left-nav bar.
2. Select the Syslog tab on the left side of the page.
3. On the Syslog Settings page, edit the port number.
4. Click Save & Deploy.

1. Go to the Configuration > Device Group page and click on the name of a device group.
2. Click the Edit (Pencil) icon.
3. In the pop-up window, enter the port(s) in the Syslog Ports field.
4. Click Save & Deploy.

Configure time zone for a device

When a device exports structured syslog messages, time zone information is included within the message. However, unstructured syslog messages do not include time zone information. By default, HealthBot uses GMT as the time zone for a device. In these cases you can assign a time zone to a device or device group within HealthBot.

1. Go to the Configuration > Device Group page and click on the name of a device group.
2. Click the Edit (Pencil) icon.
3. In the pop-up window, enter a value in the Timezone field, for example -05:00.

1. Go to the Configuration > Device page and click on the name of a device.
2. Click the Edit (Pencil) icon.
3. In the pop-up window, enter a value in the Timezone field, for example -05:00.

Configure multiple source IP addresses for a device

In cases where syslog messages arrive from a device using a different source IP address than the one originally configured in the HealthBot GUI, you can add additional source IP addresses.

1. Go to the Configuration > Device page and click on the name of a device.
2. Click the Edit (Pencil) icon.
3. In the pop-up window, enter the IP address(es) in the Syslog Source IPs field.

Configure host name aliases for a device

When a device has more than one host name, such as a device with dual REs, syslog messages can arrive at the HealthBot server with a host name that is not the device’s main host name. In these cases, you can add host name aliases for that device.

1. Go to the Configuration > Device page and click on the name of a device.
2. Click the Edit (Pencil) icon.
3. In the pop-up window, enter the host name(s) in the Syslog Host Names field.

Example: Creating a Rule Using Syslog Ingest

To illustrate how to configure and use a syslog sensor, consider a scenario where you want to:

• Monitor interface operational down status

• Use two syslog events, one structured and one unstructured

  • SNMP_TRAP_LINK_DOWN

  • PSEUDO_FPC_DOWN

• Use a rule with a trigger to indicate when the interface goes down

• CONFIGURE NETWORK DEVICES

• SET UP SYSLOG INGEST

• CREATE RULE, APPLY PLAYBOOK

• MONITOR

The workflow is as follows:

CONFIGURE NETWORK DEVICES

Add syslog configuration to the network device

If not already done, configure your network device(s) to send syslog data to Healthbot. For more details on configuring your Junos device, see the Network Device Requirements section of the HealthBot Installation Guide.

SET UP SYSLOG INGEST

Configure patterns

A pattern is a configuration to monitor some syslog event; you create a pattern for each event you want to monitor. This example uses patterns to monitor four syslog events (two structured and two unstructured).

1. In the HealthBot GUI, click Settings > Ingest in the left-nav bar.
2. Select the Syslog tab on the left of the page.
3. On the Syslog Settings page, click the + Add Pattern button.
4. In the pop-up window that appears, enter the following parameters for the first pattern, named snmp-if-link-down:

   

5. Click OK.
6. Click the add pattern button (+ Add Pattern) once more and enter the following parameters for the second pattern, named fpc-offline:

   

   Note

   The full value entered in the Filter field is fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}

7. Click OK. On the Syslog Settings page you should see the two patterns you just created.
8. At the upper right of the Syslog Settings window, click the save and deploy button (+ Save & Deploy).

Usage notes for the patterns

• The event ID (SNMP_TRAP_LINK_DOWN) references the event name found within the syslog messages.

• Fields are optional for structured syslog messages; if you don’t configure fields, the attribute names from the message will be treated as field names.

  • In this example, however we have user-defined fields:

    • The field names (if-name, snmp-index) are user-defined.

    • The field interface-name value is an attribute from the syslog message, for example, ge-0/3/1.0; this field is renamed as if-name

    • The field snmp-interface-index value is an attribute from the syslog message, for example, ifIndex 539; this field is renamed as snmp-index

    • The field snmp-interface-index here is defined as an integer; by default the fields extracted from a syslog message are of type string, however type integer changes this to treat the value as an integer

• The constant section is optional, in this example, we have user-defined constants.

  • The constant name ifOperStatus is user-defined; in this case it has the integer value of '2'

• Filter configuration is optional for a structured syslog, though you can do so if desired; if used, the filter-generated fields will override the fields included in the syslog message.

• The key fields section is optional; by default the hostname and event ID will be the keys used by HealthBot; add additional key fields here; in this example, we have key-fields, namely interface-name, where the name and value are extracted from the syslog message’s attribute-value pair

• The event ID is user defined, this case PSEUDO_FPC_DOWN

  • For example, neither the unstructured syslog Nov 22 02:27:05 R1 fpc1 Marking ports down nor its structured counterpart <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down includes an event ID

• A filter must be used to derive fields (unlike proper structured syslog); this example uses fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}, where fpc becomes the field name and NUMBER denotes the syntax used to extract the characters out of that particular portion of the message, for example “2”

  • An example of a syslog message that matches the grok filter is “fpc2 Marking ports down”

• constant fpc-status - has a string value of ‘online’

• By default in a pattern, field and constant values are a string; to treat it as an integer or float, define the pattern’s field type as integer or float

• For unstructured patterns, you must configure a filter as the messages are sent essentially as plain text and don’t include field info on their own

• Filters should always be written to match the portion of message after the event ID; this allows the filter to parse a syslog message irrespective of whether it arrives in unstructured or structured format

  • For example, the filter fpc%{NUMBER:fpc} Marking ports %{WORD:port-status} matches both versions of the following syslog message:

    • Structured: <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down

    • Unstructured: Nov 22 02:27:05 R1 fpc1 Marking ports down

Add patterns to a pattern set

With the patterns configured, group them into a pattern set.

1. On the Syslog Settings page, scroll down and click the add pattern set button (+ Add Pattern Set).
2. In the pop-up window that appears, enter the following parameters:

   

3. Click OK. On the Syslog Settings page you should see the pattern set you just created.
4. At the upper right of the window, click the save and deploy button (+ Save & Deploy).

CREATE RULE, APPLY PLAYBOOK

Configure a rule using the syslog sensor

With the syslog ingest settings complete, you can now create a rule using syslog as the sensor.

• A syslog sensor

• Four fields capturing data of interest

• A trigger that indicates when the interface goes down

1. Click Configuration > Rules in the left-nav bar.
2. On the Rules page, click the + Add Rule button.
3. On the page that appears, in the top row of the rule window, set the rule name. In this example, it is check-interface-status.
4. Add a description and synopsis if you wish.
5. Click the + Add sensor button and enter the following parameters in the Sensors tab:

   

6. Now move to the Fields tab, click the + Add field button, and enter the following parameters to configure the first field, named event-id:

   

7. Click the + Add field button again and enter the following parameters to configure the second field, named fpc-slot:

   

8. Click the + Add fieldbutton again and enter the following parameters to configure the third field, named if-name:

   

9. Click the + Add field button once more and enter the following parameters to configure the fourth field, named snmp-index:

   

10. Now move to the Triggers tab, click the + Add trigger button, and enter the following parameters to configure a trigger named link-down:

    

11. At the upper right of the window, click the + Save & Deploybutton.

Usage Notes for the rule

• Sensor tab

  • The sensor name if-status-sensor is user-defined

  • The sensor type is syslog

  • Pattern set check-interface-status - reference to the pattern set configured earlier

  • If not set, the Maximum hold period defaults to 1s

• Fields tab

  • Four fields are defined; although the patterns are capturing more than four fields of data, this example defines four fields of interest here; these fields are used in the trigger settings

  • The field names (event-id, fpc-slot, if-name, snmp-index) are user-defined

  • path event-id - default field created by syslog ingest in the raw table; references the field from the pattern configuration

  • path fpc - references the value from the filter used in the unstructured pattern configuration

  • path if-name - references the field from the pattern configuration

    • Data if missing all interfaces - if the if-name value is not included in the syslog message, use the string value “all interfaces”

  • path snmp-index - references the field from the pattern configuration

• Triggers tab

  • The trigger name link-down is user-defined

  • frequency 2s - HealthBot checks for link-down syslog messages every 2 seconds

  • term is-link-down - when $event-id is equal to SNMP_TRAP_LINK_DOWN, in any syslog message in the last 300 seconds, make red and show the message Link down for $if-name(snmp-id: $snmp-index)

    • $event-id - $ indicates to reference the rule field event-id

    • Link down for $if-name(snmp-id: $snmp-index) - for example, “Link down for ge-2/0/0 of FPC 2”

    • $if-name - references the field value, i.e., the name of the interface in the syslog message

  • term is-fpc-down - when $event-id is equal to PSEUDO_FPC_DOWN, in any syslog message in the last 300 seconds, make red and show the message Link down for $if-name of FPC$fpc-slot

    • $event-id - $ indicates to reference the rule field event-id

    • $if-name - “all interfaces”

    • Link down for $if-name of FPC$fpc-slot - for example, “Link down for all interfaces of FPC 2”

Add the rule to a playbook

With the rule created, you can now add it to a playbook. For this example, create a new playbook to hold the new rule.

1. Click Configuration > Playbooks in the left-nav bar.
2. On the Playbooks page, click the + Create Playbook button.
3. On the page that appears, enter the following parameters:

   

4. Click Save & Deploy.

Apply the playbook to a device group

To make use of the playbook, apply it to a device group.

1. On the Playbooks page, click the Apply (Airplane) icon for the check-interface-status playbook.
2. On the page that appears:

   • Enter a name

   • Select the desired device group

   • Click Run Instance

3. On the Playbooks page, confirm that the playbook instance is running. Note that the playbook may take some time to activate.

MONITOR

Monitor the devices

With the playbook applied, you can begin to monitor the devices.

1. Click Monitor > Device Group Health in the left-nav bar, and select the device group to which you applied the playbook from the Device Group pull-down menu.
2. Select one of more of the devices to monitor.
3. In the Tile View, the tile labeled external contains the parameters from the rule you configured earlier.