Configure System Log Ingest

To be able to apply system log (syslog) ingest in a rule, you must first configure a device to send syslog data, configure syslog ingest by adding events pattern and applying patterns to patter sets, and configure syslog header. You can refer this section to also clone pattern configurations and edit header configurations.

What's Next

When you configure syslog settings in Paragon Insights, you can opt to configure port, time zone for devices, and so on. For more information on optional configurations, see System Log Optional Configurations.

Device Configuration

Configure your network device(s) to send syslog data to Paragon Insights. The example device configuration from the previous section is repeated here:

## 10.10.10.1 = Load balancer IP address

Configure Syslog Events Pattern

An events pattern is a configuration to monitor some syslog event; you create a pattern for each event you want to monitor. This example uses patterns to monitor four syslog events (two structured and two unstructured).

Note:

See the usage notes at the end of this section for more detail on what has been configured.

To configure Syslog pattern in GUI:

Click Configuration > Data Ingest > Settings in the left-nav bar.
Select the Syslog tab on the Ingest Settings page.
On the Syslog Settings page, click the plus (+) icon.
In the pop-up window that appears, enter the following parameters for the first pattern, named snmp-if-link-down:
Click Save and Deploy.
Click the plus (+) once more and enter the following parameters for the second pattern, named fpc-offline:

Note:
The full value entered in the Filter field is fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}
Click Save & Deploy. On the Syslog Settings page you should see the two patterns you just created.

Usage notes for the patterns

For structured syslog:

The event ID (SNMP_TRAP_LINK_DOWN) references the event name found within the syslog messages.
Fields are optional for structured syslog messages; if you don’t configure fields, the attribute names from the message will be treated as field names.
- In this example, however we have user-defined fields:
  - The field names (if-name, snmp-index) are user-defined.
  - The field interface-name value is an attribute from the syslog message, for example, ge-0/3/1.0; this field is renamed as if-name
  - The field snmp-interface-index value is an attribute from the syslog message, for example, ifIndex 539; this field is renamed as snmp-index.
  - The field snmp-interface-index here is defined as an integer; by default the fields extracted from a syslog message are of type string, however type integer changes this to treat the value as an integer
The constant section is optional, in this example, we have user-defined constants.
- The constant name ifOperStatus is user-defined; in this case it has the integer value of '2'
Filter configuration is optional for a structured syslog, though you can do so if desired; if used, the filter-generated fields will override the fields included in the syslog message.
The key fields section is optional; by default the hostname and event ID will be the keys used by Paragon Insights; add additional key fields here; in this example, we have key-fields, namely interface-name, where the name and value are extracted from the syslog message’s attribute-value pair

For unstructured syslog:

The event ID is user defined. In this case, it is PSEUDO_FPC_DOWN
- For example, neither the unstructured syslog Nov 22 02:27:05 R1 fpc1 Marking ports down nor its structured counterpart <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down includes an event ID.
A filter must be used to derive fields (unlike proper structured syslog); this example uses fpc%{NUMBER:fpc} Marking ports %{WORD:port-status}, where fpc becomes the field name and NUMBER denotes the syntax used to extract the characters out of that particular portion of the message, for example “2”.
- An example of a syslog message that matches the grok filter is “fpc2 Marking ports down”.
Constant fpc-status - has a string value of ‘online’.

Regarding filters:

By default in a pattern, field and constant values are a string; to treat it as an integer or float, define the pattern’s field type as integer or float.
For unstructured patterns, you must configure a filter as the messages are sent essentially as plain text and don’t include field info on their own.
Filters should always be written to match the portion of message after the event ID; this allows the filter to parse a syslog message irrespective of whether it arrives in unstructured or structured format.
- For example, the filter fpc%{NUMBER:fpc} Marking ports %{WORD:port-status} matches both versions of the following syslog message:
  - Structured: <166>1 2019-11-22T02:38:23.132-08:00 R1 - - - - fpc1 Marking ports down
  - Unstructured: Nov 22 02:27:05 R1 fpc1 Marking ports down

Add Patterns to a Pattern Set

With the patterns configured, group them into a pattern set.

On the Syslog Settings page, click to expand the Pattern Set section and click the plus (+) icon.
In the pop-up window that appears, enter the following parameters:
Click Save & Deploy. On the Syslog Settings page you should see the pattern set you just created.

Configure Header Pattern

In Paragon Insights, you can configure the pattern for parsing the header portion of a syslog message. With this release, unstructured syslog messages of non-Juniper devices are supported. In earlier releases, you can only parse the payload portion of either a structured syslog message as specified in RFC 5424 standard, or a Juniper device’s unstructured syslog message.

In general, it is assumed that any unstructured syslog message matches the Juniper syslog message pattern. For example, you do not have to configure a Juniper header pattern as this pattern is inbuilt with Paragon Insights. However, in case of a non-Juniper device’s unstructured syslog message that does not match with the inbuilt pattern, a first match is made with one of the user-configured header patterns. Following a successful match, the fields are extracted. When there is no match, the incoming syslog message is dropped.

To configure a header pattern:

Navigate to Configuration > Data Ingest > Settings in the left navigation bar.

The Ingest Settings page is displayed.
Click Syslog to view the Syslog Settings page.
Click to expand the Header Pattern section.

The Header Pattern section of the Syslog Setting page is displayed. You can add a new header pattern and edit or delete an existing header pattern from this page.
Click the plus (+) icon to add a new header.

The Add Header Pattern page is displayed.
Enter the following information in the Add Header Pattern page.
1. Enter a name for the header pattern in the Name field.
2. Enter a description for the header pattern in the Description field.
  For example, you can provide a one-line description of why you are creating this header pattern.
3. Enter the filter or regular expression (regex) for the header patter in the Filter field.
  Note:
  You can use regex101.com to edit, validate, and modify the filter pattern you want to add to the header pattern.
  
  An example of a filter pattern is (.*):([A-Z][a-z]{2} \d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d*)\s:\s([a-z]*)\[(\d*)\]:\s*(.*)\s*.
4. log-host, log-timestamp, and log-message-payload of the Fields section are mandatory fields that determine the position of the header.
  In the Fields section,
  1. Click log-host, and enter the following information.
    - Enter a name for the log host in the Name field.
      
      log-host is the default name.
    - Enter a description for log-host in the Description field.
      
      The default description is Position of host name.
    - Enter the capture group value with prefix $ in the From field.
      
      The capture group determines from which position in the header the log-host starts.
  2. Click log-timestamp, and enter the following information.
    - Enter a name for the log timestamp in the Name field.
      
      log-timestamp is the default name.
    - Enter a description for log-timestamp in the Description field.
      
      The default description is Position of time stamp.
    - Enter the capture group value with prefix $ in the From field.
      
      The capture group determines from which position in the header the log-timestamp starts.
    Note:
    Ensure that timestamp format follows this sample timestamp format: “Jan _2 15:04:05 2006”. Otherwise parsing of syslog messages will lead to an undefined behaviour.
  3. Click log-message-payload, and enter the following information.
    - Enter a name for the log message payload in the Name field.
      
      log-message-payload is the default name.
    - Enter a description for log-message-payload in the Description field.
      
      The default description is Position of payload.
    - Enter the capture group value with prefix $ in the From field.
      
      The capture group determines from which position in the header the log-message-payload starts.
  4. (Optional) Click the plus (+) icon to add a new field.
    - Enter a name for the new field in the Name field.
    - Enter a description for the new field in the Description field.
    - Enter the capture group value with prefix $ in the From field.
      
      The capture group determines from which position in the header the new field starts.
    You can add one or more than one fields by clicking plus (+) icon.
Enter the name(s) of key fields in the Key Fields field.
Click Save to save configuration, or click Save & Deploy to save and immediately deploy the configuration.
Alternatively, to cancel the configuration, click Cancel.

Edit a Header Pattern

To edit an already configured header pattern:

Navigate to Configuration > Data Ingest > Settings in the left-nav bar.

The Ingest Settings page is displayed.
Click Syslog to view the Syslog Settings page.
Click to expand the Header Pattern section.

The Header Pattern section of the Syslog Setting page is displayed.
Select the header pattern you want to edit by selecting the check box next to the name of the header pattern, and click the Edit (pencil) icon.

The Edit Header Pattern<Header Name> page is displayed.
After you have edited the required fields, click Save to save configuration.
You can also click Save & Deploy to save and immediately deploy the edited configuration.

Clone a Syslog Events Pattern

To clone an existing Syslog pattern:

Click Configuration > Data Ingest > Settings in the left-nav bar.

The Ingest Settings page is displayed.
Click the Syslog tab to view the Syslog Settings page.
Click to expand the Events Pattern section in the Syslog Settings page to view existing syslog events patterns.
To clone a pattern, click the Clone.
The Clone Pattern: <name of syslog pattern> page is displayed.

From the Clone Pattern: <name of syslog pattern> page, you can
- Edit existing fields
- Add new fields or constants
- Add or remove key fields
Click Save to save configuration and clone the syslog pattern.
Alternatively, click Save & Deploy to save configuration, clone syslog pattern, and deploy the pattern.

Clone a Pattern Set

To clone an existing Syslog pattern set:

Click Configuration > Data Ingest > Settings in the left-nav bar.

The Ingest Settings page is displayed.
Click the Syslog tab to view the Syslog Settings page.
Click to expand the PatternSet section in the Syslog Settings page to view existing syslog pattern sets.
To clone a pattern set, click the Clone.
The Clone Pattern-set: <name of pattern-set> page is displayed.

From the Clone Pattern-set: <name of pattern-set> page, you can
- Edit the name and description fields
- Add or remove patterns from the Patterns field.
Click Save to save configuration and clone the syslog pattern set.
Alternatively, click Save & Deploy to save configuration, clone syslog pattern set, and deploy the pattern set.

Configure Multiple Source IP Addresses for a Device

You can add additional source IP addresses for devices that send syslog messages using a different source IP address than the one originally configured in the Paragon Insights GUI.

To support additional source IP addresses:

Go to the Configuration > Devices page and click on the name of a device.
Click the Edit (pencil) icon.

The Edit Device-Name page appears.
Click Device ID Details and enter the IP address(es) in the Syslog Source IPs field.

ON THIS PAGE